3. Part 3. Develop a breadth of practice relevant to your role - read and comment on an article - 'Inter-indexer consistency'

Read and comment, in the Professional Development Wiki/Blog, on an article with a potential to change current thinking and perspectives about a professional indexing issue in LIS literature and publicize it amongst team members. Evaluate how this learning has made a difference to my own indexing philosophy.

BOK 2 - Generation, Communication & Use of Information.

Inter-indexer consistency in graphic materials indexing at the National Library of Wales

As part of my role as a senior indexer at the National Library I am part of the team who carries out Quality Reviews of the rest of the Indexing Team's work. This requires me to check selected records for abstract and keyword appropriateness. As I check I often wonder how closely I would have indexed the same article i.e., how much inter-indexer consistency would there be. At other times, on the very odd occassion when a print version of a journal, which has already been indexed, is indexed again when its electronic version arrives, one can compare two indexer's records for the same article and marvel at both the similarities and the differences in focus and choice of keyword. This happened to me recently and got me wondering if any studies had been conducted into the phenomena of inter-indexer consistency.

It turns out there were many such studies but only a couple that were recent enough to select as appropriate for this task. I selected a study by Alan Vaughan Hughes (National Library of Wales, Aberystwyth, UK) and Pauline Rafferty (Department of Information Studies, Aberystwyth University, UK) entitled 'Inter-indexer consistency in graphic materials indexing at the National Library of Wales' and although it is not directly applicable to the work we do here at Index New Zealand (in that it looks at inter-indexer consistency in graphic materials and we index journals and newspapers), it does raise some interesting points and issues worthy of discussion by my colleagues.

The purpose of the paper was to investigate the degree of inter-indexer consistency in the assignment of controlled vocabulary topical subject index terms (taken from the Thesaurus of Graphic Materials) to identical graphical images by different indexers at the National Library of Wales.

In terms of its findings there was alot to be grateful for as inter-indexer consistency was found to be generally low and highly variable (thus bearing out my own anecdotal findings whenever I get to compare two records indexed by two different indexers on the same article).

Throughout the rest of the paper I picked up on some interesting and relevant findings from the literature worthy of discussion:

• there is only a tentative correlation between indexing consistency and retrieval effectiveness
• many people see consistency as a benchmark of quality in service provision to information seekers and potent resource discovery
• user-generated indexing has attacted interest as an information retrieval tool but (thankfully) there will always be a place for expert indexing
• the literature provides an overall indication that indexing consistency varies considerably, absolute consistency is highly unlikely to be achieved and consistency s very difficult to both measure and compare
• the relationship between indexing consistency and 'indexing quality' is also problematic as there is a lack of a concise definition as to what 'quality' actually is - especially as there is no one 'correct' way to index a document
• while it is suggested that consistency indicates the 'reliability' of indexing (and one can perhaps hope for that with Index New Zealand - that a perception by the user of a high degree of consistency between the records will hopefully suggest to them a high degree of reliability at the same time) it is also suggested that consistent indexing, while it can be good, can also be consistently bad too
• in terms of the factors influencing indexing, the categories of 'pragmatic', 'semantic' and 'environmental' (Zunde and Dexter, 1969) included some interesting examples:
• the experience, competencies and dispostion of the indexer
• I know if I am tired or sore that my indexing is affected
• also this is where the subjective judgements of the indexer in the identification of concepts and vocabulary selection come into play and there's not alot theat can be done about this aspect apart from gaining the experience and knowledge that can be used to refine this subjectiveness
• the characterisitcs, topic and specificity of the indexed document (some of those scientific articles have me searching our database for examples of what other indexers have used for the same topic)
• the influence of indexing aids, indexing procedures and exhaustivity (just how good are our manuals and training aids at attaining consistency?)
• the use, size, specificity and semantic aspects of any controlled vocabulary (how long does it take to get your head around the pattern of FAST headings and sub-headings?)
• the climatic conditions, noise, lighting (ever since moving back into our new re-furbished building some of our team are having ongoing issues with glare and reflection)
• taking the controlled vocabulary aspect further, it has been linked to improved consistency, especially when applied by experienced indexers knowledgeable in the subject matter and fully familiar with the terms
• although greater inconsistency was found with the application of pre-coordination and sub-headings
• it seems there is no real correlation between indexing exhaustivity (the level of detail used) and consistency although this is not well represented in the literature
• thankfully there is greater consistency among competent indexers/library professionals than between library patrons at least!

Overall I was left with the following impressions:

• inter-indexing consistency is a difficult target to reach
• our use of the controlled vocabulary FAST is of some benefit
• our use of the abstract is an important way to overcome this lack of keyword consistency in that it is an extra opportunity to more comprehensively cover off the true nature of the article concerned
• anything we can do to suggest consistency and therefore quality will help maintain the need for real, live indexers over user-generated options
• perhaps our indexing training and manuals could more reflect a consistent aim and approach? (although exactly what this means in reality will require further thought)
• that I need not worry if my record was different to my colleagues when I compared the two - as long as they are both in the same 'ballpark' that is all we can hope for
• perhaps it would not hurt us, as a team, to practice occassions where we purposefully index the same articles, then compare our results and discuss what we find -  a great way to discover the different strategies and thought processes different indexers have (especially between experienced and not-so experienced indexers?)

What do you think?

21 November 2012
Publicised this article amongst my fellow Indexing Team members with suggestion for them to read and leave comments. Although there has not been any feedback thus far I intend to use this article as an example of the process and continue to encourage colleagues to add and share any relevant articles we come across in the course of our professional readings.


Reference
Alan Vaughan Hughes, Pauline Rafferty, (2011) "Inter-indexer consistency in graphic materials indexing at the National Library of Wales", Journal of Documentation, Vol. 67 Iss: 1, pp.9 - 32.