[OPEN-ILS-DEV] Questions about keyword search indexing

[Kathy Lussier]

Hi all,

Since implementing Evergreen, the consortia I work with have added 
several custom indexes, mostly for the keyword class, to address the 
following issues:

* Add weights for particular indexes (title, subject, author) for a 
keyword search.
* Add MARC fields that are not covered by MODS or are excluded from the 
default keyword index (aka the blob).
* Add indexes for 880 fields.

Although we love the way the system is flexible enough to allow us to 
make these adjustments, we've come across a few problems that has us 
re-thinking the way we set up our indexes. I wanted to send along some 
of the problems we've encountered and the solutions we're considering to 
see if you all have feedback on our potential solutions, if you see 
drawbacks we hadn't considered, or if you have alternative approaches 
that might work.

Some of the issues we've encountered:

* When adding an index to add a MARC tag that is not covered by the 
keyword blob, we run into the problem where these indexes are not 
"combined." For example, NOBLE added a local field (962) for captions of 
digital images to its records and then added this tag to an entry for 
config.metabib_field.  If you do a keyword search for "ship george 
west," you will successfully find this record - 
http://evergreen.noblenet.org/eg/opac/record/1622058?query=ship%20george%20west;qtype=keyword;locg=1 
- based on records in the 962 field, but if you do a keyword search for 
"ship george west marblehead," you will get no results because 
marblehead is in a different entry of the metabib.keyword_field_entry 
table than the other keywords are.

For a brief period of 2.4, a new combined indexes feature was turned on 
by default to address this problem, but it was subsequently turned off 
due to the bug reported at 
https://bugs.launchpad.net/evergreen/+bug/1169693. Since we use 
additional indexes for the keyword class for the purpose of weighting, 
we don't want to turn them on because we're concerned about how it would 
affect relevance ranking.

* When adding an index to add a MARC tag that is not covered by the 
keyword blob, we also run into a problem of unintentionally adding 
weight to that index even when we don't want to give it additional 
weight. As an example, two of our Evergreen sites added the 260b tag to 
the keyword index so that the publisher could be searchable. We then had 
an issue where a keyword search for "free spirit," pulled up lots of 
results for books by a publisher called Free Spirit ahead of the book 
that was being sought (Free Spirit: Growing Up On the Road and Off the 
Grid). 
http://bark.cwmars.org/eg/opac/results?query=free+spirit&qtype=title&fg%3Aformat_filters=&locg=1&sort=. 
The weight for the publisher field is 1, and the weight for the title 
field is 10. However, those records come up sooner because they score so 
highly when it comes to coverage density.

If there were an easier way to include the publisher field in the blob, 
then we could have avoided this problem. However, there didn't seem to 
be an easy way to include it while continuing to exclude the rest of the 
fields that are contained within the MODS origininfo element.

* We're also aware of the fact that, since we have to add an index every 
time we need to add a MARC tag, we're adding more fields to the 
metabib.keyword_entry table. In addition, one of our sites has added 880 
indexes using the method I described here - 
http://markmail.org/message/dlzvcxezaycychd4 , which indexes all 
records, not just those with an 880 field. We wonder if the addition of 
all these entries ultimately impacts performance. We are also looking at 
ways of indexing just those records with the 880 field when using the 
marc21expand880 format.

* With the addition of our custom indexes, one question that frequently 
arises is if we're hurting overall search performance by adding indexes 
that result in more entries for our metabib.keyword_field_entry tables. 
Since I work with three different Evergreen sites, we're able to do some 
comparison. The site that has added the fewest (if any) custom indexes 
does indeed seem to have faster keyword searches.

If a site is only using the default keyword blob and no other keyword 
indexes, the number of entries in the metabib_keyword_field_entry table 
should be the same as the number of bib records in the system. However, 
when looking at the sites that have added more custom indexes, we see as 
many as 10 times as many entries in the metabib_keyword_field_entry 
table as there are bib records in the database. At what point do the 
number of metabib entries begin to seriously impact search performance?

One of our sites is considering a different approach to configuring its 
search indexes, and I wanted to put our ideas out here to see if you 
have feedback on it.

Instead of using the keyword blob based on mods, they are experimenting 
with setting up all of their keyword indexes to be based on MARC tags. 
We've talked about two ways of making this happen:

* Creating another keyword blob, similar to the current default index, 
that includes all of the MARC tags and subfields that we want included 
in the keyword index. Under this approach, we could easily add tags like 
the 260b or the 962 without creating an additional config.metabib_field 
entry. Those tags would just be incorporated in the main blob and, 
therefore, wouldn't be adding unintended weight to keyword searches or 
running up against the combined indexes problem. Under this scenario, 
there would still be cases where we would need to configure additional 
keyword indexes whenever we wanted to apply additional weight to a 
specific MARC field.

* The other approach is to create multiple config.metabib_field entries 
for individual MARC tags, or perhaps for groups of tags. We would then 
turn the combined indexes on for keyword indexes. Based on Mike's 
comments at 
https://bugs.launchpad.net/evergreen/+bug/1169693/comments/2, since we 
no longer would have the large keyword blob, turning combined indexes on 
shouldn't lead to the relevance-ranking problems that arose when 2.4 was 
first released. We could then adjust the weight for those entries that 
include the more relevant MARC tags. However, I do have concerns that 
we'll continue to come across problems like the "Free Spirit" example I 
mentioned above where an exact match on a 260b tag will continue to 
float above other more relevant results.

While this site continues looking at the indexes, can you tell me if 
there is anything we would be losing by going from indexes based on MODS 
to ones based purely on MARC tags? What were the original reasons for 
basing the default indexes on MODS?

Also, are there any pros or cons to implementing either of the 
approaches I mentioned above? Do you see other ways to address some of 
the issues we've come across in creating custom indexes?

Thanks in advance for your insight!

Kathy


-- 
Kathy Lussier
Project Coordinator
Massachusetts Library Network Cooperative
(508) 343-0128
klussier at masslnc.org
Twitter:http://www.twitter.com/kmlussier
#evergreen IRC: kmlussier