OLE Search Technical Documentation

DocStore Search

1. Indexed Data

1.1 Searchable fields for all document categories, types and formats

1.1.1 Searchable fields for Bibliographic

1.1.2 Searchable fields for Holdings

1.1.4 Searchable fields for License

1.2 Facet fields for all document categories, types and formats

1.3 Field definitions for Work-Bib-MARC documents

Test Template for Work-Bib-Marc document fields

1.4 Format field definitions for Work-Bib-MARC documents

1.5 Field definitions for Work-Bib-DublinCore documents

Test Template for Work-Bib-DublinCore document fields

2. Search

This functionality allows documents to be searched for by giving keywords or phases. Searching can be based on category, type, format, search fields.

2.1  Quick Search

            Select Doc Category : Work

                      Doc Type : Bibliographic

                      Doc Format : ALL

   Searching on default condition(click search button without specifying any conditions) will give all the records in search result page.

            Select Doc Category : Work

                      Doc Type : Bibliographic

                      Doc Format : MARC

   Type one or more keywords in a text box.

   System shows records with any field matching one or more keywords.

2.2  Advanced Search

            Select Doc Category : Work

                      Doc Type : Bibliographic

                      Doc Format : MARC

   The drop down for search fields will be populated based on the category selected above.

   User specifies a search condition:

             Selects a field.

             Enters one or more keywords.

             Specifies whether the keywords should be searched for as "All of these", "Any of these" or "As a phrase".

                            "All of these"   - Any record with the selected field having all the entered keywords is included in the search results.

                           "Any of these" - Any record with the selected field having at least one of the entered keywords is included in the search results.

                           "As a phrase"  - Any record with the selected field having all the entered keywords in same order is included in the search results.

   User adds another condition:

             Chooses whether to apply this condition in addition to the previous one ("AND") or to apply this condition as an alternative to the previous one ("OR")  ("NOT"???),

             "AND"  - the conditions before and after this operator should be satisfied.

             "OR"    - one of the conditions before and after this operator should be satisfied.

             "NOT"  - the condition after this operator should not be satisfied.

    User repeats previous step as many times as needed using the ADD and DELETE links.

          [+]ADD : click on this link to add fields for a new search condition.

           [-]Delete : click on this link to delete the last search condition.

   Search is performed based on the conditions entered by the user.

2.2.1 Using the wildcard char * in searching

    An asterisk (*) can be entered as part of a search term (value to be searched for). But it should not be the first character.

    For example, rec* will match record, recurring etc. j*n will match John, join etc.    

2.3 Solr-specific search rules

Solr allows us to specify how the input data is indexed and searched for.

Data of type String is indexed and stored verbatim.

Data of type Text can be analyzed during indexing time and searching time as follows:

2.3.1 Tokenization

It is the process of splitting the input text into tokens that are indexed and searched for.

White Space Tokenizer is used. It is a simple tokenizer that splits the text stream on whitespace and returns sequences of non-whitespace characters as tokens. Note that any punctuation will be included in the tokenization.

Input: "To be, or what?"

Output: "To", "be,", "or", "what?"

2.3.2 Synonym Filtering

It is the process of synonym mapping. Each token is looked up in the list of synonyms and if a match is found, then the synonym is emitted in place of the token. The position value of the new tokens are set such they all occur at the same position as the original token

It is applied only on search parameters text.

Synonyms are specified in a text file named ‘synonyms.txt’

The following are currently defined in this file.

GB,gib,gigabyte,gigabytes

MB,mib,megabyte,megabytes

Television, Televisions, TV, TVs

# Synonym mappings can be used for spelling correction too

pixima => pixma

2.3.3 Stop word filtering

It is the process of discarding tokens that are on the given stop words list.

The file named stopwords.txt specifies such words. Currently they are:

an and are as at

be but by

for

if in into is it

no not

of on or

s such

t that the their then there these they this to

was will with

2.3.4 Word delimiter (splitting)

It is the process of splitting tokens at word delimiters. The rules for determining delimiters are as follows:

•     A change in case within a word: "CamelCase" -> "Camel", "Case"
•     A transition from alpha to numeric characters or vice versa:"Gonzo5000" -> "Gonzo", "5000"   ;  "4500XL" -> "4500", "XL"
•     Non-alphanumeric characters (discarded): "hot-spot" -> "hot", "spot"
•     A trailing "'s" is removed: "O'Reilly's" -> "O", "Reilly"
•     Any leading or trailing delimiters are discarded: "-hot-spot" -> "hot", "spot"

2.3.5 Lower case conversion

Any uppercase letters in a token are converted to the equivalent lowercase token. All other characters are left unchanged.

2.3.6 Keyword protection

Protecting words from being modified by stemmers.

Protected word list may be specified in a file named protwords.txt

No such words are specified at this time.

2.3.7 Stemming

It is the process of reducing any of the forms of a word such as "walks, walking, walked", to its elemental root e.g., "walk".

Porter Stemming Algorithm is used. It is only appropriate for English language text.

2.3.8 Remove duplicates

Removing duplicate tokens in the stream. Tokens are considered to be duplicates if they have the same text and position values.

2.3.9 Known Behaviors (to be fixed within Search)

• All of these search selection searches for the words in the order they are entered.
• As a phrase search selection does not search for the words in the order they appear.
• Special characters such as ‘&’ and ‘:’ are searchable only if part of a longer phrase and the entire phrase is wrapped in quotation marks.  (If the quotation marks are left off, no hits will be returned.)
• Wildcards (*, ?, #, !, []) do not function. 

3. Display of Search results

    The search results are displayed in pages of 25 records.

    The page size can be changed to 50,100

    User can navigate to different pages of search results by using a pagination control.

    For each record, short display fields (specified in section 1) will be displayed.

    For each record 'View XML' button shows the xml of the document and 'Edit XML' opens the Editor.

   Links are shown if applicable

3.1 Short display

    Each record in the search results shows a subset of the record's fields.

3.2 Detailed display

   A link is (not yet) provided for each record in the search results.

   When this link is clicked, a popup, or a new tab of the browser is opened, with all the fields of the record.

3.3 Highlighting

In each search result, words matching the key words entered by the user are highlighted.

3.4 Facets

Facets are grouping of search results that help to analyze the search results and further filter or narrow down the search results.

These are helpful when the user cannot guess what exact keywords to search for.

Facet fields:

Author,Subject,Format,Language,Publication Date,Genre.

Top 5 facet values in the decreasing order of occurrences are shown for each facet field.

The remaining facets are seen by clicking the “more” link in a popup window.

The facet values in the popup are shown in alphabetical order.

Click on one or more facet values to filter search results and view only those records containing these facet values.

3.5 Sorting

By default, the search results are sorted by Title (A-Z)

The sorting options are:

    Title (A-Z) : sorts the results in the  ascending order (starts with ‘a’) of the title field value. This the default sorting order.
    Title (Z-A) :sorts the results in the descending  order (starts with ‘z’) of the title field value.
    Author (A-Z) :sorts the results in the ascending order (starts with ‘a’) of the author field value.
    Author (Z-A) : sorts the results in the descending order (starts with ‘a’) of the author field value.
    Pub date(new-old) : sorts the results in the descending order of the publication date field value.
    Pub date(old-new) : sorts the results in the ascending order of the publication date field value.  
    Relevance : it is default sort criteria provided by solr.

Records with empty or null values will appear at the top of the search results.

As per JIRA OLE_2194,

2nd indicator in MARC is non-filing character. need to use rules for these in applying sort/display standards. Ex. 245 1 3 $aAn April Shower- 2nd indicator is "3". Ignore first 3 characters, ie "An(space)" in applying sort rule

Test Template for sort

3.5.1 NISO Standard for Sort

The sort rules as recommended in the NISO standard are to be implemented. Please refer to NISO Standard for Sort for details.

NISO Search Results on docstore Search- in progress: https://jira.kuali.org/browse/OLE-2194  

Most of the rules for sorting are done in Solr by specifying filters like LowerCaseFilterFactory in schema.xml

4. Configurability of searchable fields

As seen above, there are different fields in the documents of different category/type/formats, that are to be indexed, searched for and displayed. How a field value should be extracted from a source document is also important.

This information about different document categories, types and formats and their corresponding fields is used in DocStore as well as OLE. It is preferable to store this information in a well encapsulated, reusable and maintainable manner.

This is done in an external xml configuration file (DocumentConfig.xml) which is loaded into a corresponding POJO name DocumentConfig. The configuration file is loaded at the time of startup of application and retained in memory until the application is shutdown. So any changes to this file will become effective only after restarting the application. There are limitations on what can be configured in this file. Changes to this file will need the existing documents to be re-indexed.

4.1 DocumentConfig Info

DocumentConfig.xml

It is available in %ole.docstore.home%/properties folder. (e.g. /opt/docstore/properties)

4.2 Field Definition

Field name convention:

Names of fields used for indexing/searching are suffixed with “_search”.

Names of fields used for display are suffixed with “_display”.

Names of fields used for facets are suffixed with “_facet”.

Names of fields used for sorting are suffixed with “_sort”.

Field Info/Attributes

Field Definition Example:

       <field id="ISBN_search" name="ISBN" type="text">
          <mapping type="custom">
              <include>020-a;z</include>
              <exclude/>                       
          </mapping>
       </field>

4.3 Mapping Definition

Mapping info can be defined for each field which specifies how the value(s) for the field should be extracted from the input file for the corresponding document. Mapping can be specified as XPATH value or a custom value.

Mapping Info/Attributes

xpath Mapping Example:

       <field id="ContractNumber_search" name="Contract Number" type="text">
          <mapping type="xpath">
              <include>/publicationsLicenseExpression/licenseDetail/licenseIdentifier/IDValue/value</include>                       
          </mapping>
       </field>

4.4 Modifying configuration info

1. Open the DocumentConfig.xml file.
2. Add/modify/delete one or more fields of any [document category/type/format].
3. Save the file.
4. Reload the DocStore application. (Restart the Tomcat server.)
5. Re-index the data related to the document category/type/format modified.

Adding a field:

Copy and paste an existing field definition and modify the attributes suitably.

Modifying a field:

Name and mapping info can be modified for any existing field.

Deleting a field:

A field definition can be commented or deleted.

Test Template for Document Config

Transactional Search

OLE coding to-date for Acquisitions functions have utilized KNS Lookups, DocSearch (Detailed Search, Superuser Search), and named or session-based searches......

<insert more info on framework>

1. Doc Search (Button)- takes user to short search, but can jump to Superuser or Detailed Search and specify edocument (PO, etc).
2. OLE Acquisitions Search
   1. OLE created OLE Acquisitions Search to assist Acquisitions and Selectors. We still need to add more transactional search options (such as PO's paid, PO's received on PO searches, etc), but created the following Search fields that combine purchasing and Bib information:
      1. Document Type:
      2. Document Number:
      3. Purchase Order #:
      4. Vendor Name:
      5. Date Created From:
      6. Date Created To:
      7. Initiator:
      8. Requestor:
      9. Account Number:
      10. Organization Code:
      11. Chart Code:
      12. Title:
      13. Author:
      14. Publisher:
      15. ISXN:
      16. Search Type
3. KFS (inherited Custom Doc Searches). 
   1. inherited Custom Doc Searches allow users to save Named Searches, or select from search session history.
   2. OLE will extend or replace existing searches to include docstore or bib data:
   3. Requisition search
   4. Purchase Order search
   5. Receiving search
   6. Payment Request search
4. OLE may also need to extend KFS Account Searches
5. OLE has implemented the following combined search and group-action queues, allowing users to search, sort, filter and apply global actions across multiple documents in current workflow:
   1. OLE Order Holding Queue
      1. Selector:
      2. Document Number:
      3. Requisition Status:
      4. Vendor Name:
      5. Requestor:
      6. Format:
      7. Chart Code:
      8. Account Number:
      9. Object Code:
      10. Workflow Status Change Date From:
      11. Workflow Status Change Date To:
      12. Title:
      13. Author:
      14. Publisher:
      15. (will add ISxN in future)
6. OLE Receiving Queue
   1. Purchase Order Number
   2. ISxN
   3. Title
   4. Journal
   5. Filters (not yet functional, but on UI):
      1. Serials
      2. Standing Orders
      3. Monographs
      4. Status
      5. Vendor
      6. Purchase Order Date
7. Known Issues with KNS searching:
   1. If you know the exact words or phrase, enter the text wrapped in quotes
   2. If you want to return hits for any of the words entered, leave off the quotes
   3. Use "*" only at the end of a word and only when you have entered a single word in the field
   4. Specify dates in the format mm/dd/yyyy.
   5. Boolean operators used within search screens are not currently providing consistent results.

Notes/Not yet implemented:

The following and other features may be added and reviewed with SMEs for future releases.

1. Authority records: linkages, search, NACO standards
2. Call Number Browse (coming in OLE 0.8)
3. Linked PO or Circ record from Item, and Order/Circ status (coming in OLE 0.8)
4. Linked License Agreement (electronic journals etc)- in progress
5. Search filters: Location, Format, TBA
6. External Linked Data: Authority, or other stores
7. Saved DocStore Searches (or user preferences)
8. Wildcard behaviors
9. Positional Operators
10. Truncation
11. Nested Search (more than one operator in same expression)
12. Field/Marc tagging search
13. Checkin, Checkout from Search
14. Rice/KNS upgrades (future): search facets and other enhancements for transactional search
15. Non-Roman Characters (ie, Chinese, Russian, etc)