User Guide - OLE Document Store

Owned by TB (Unlicensed)
Last updated: Oct 31, 2013 by Sambasiva Madalam (Unlicensed)Version comment

1. Introduction

Document Store is a Document storage repository system with features like checkin, checkout, Ingest , Delete etc for library records such as Bibliographic, Instance (Holdings and Items) etc. Most of the records are in XML format but the Document Store is format agnostic in that it stores the content as is without any type conversion. Furthermore indexing of the stored data is also supported for efficient search and retrieval. Although the Document Store is an independent system that comes with basic UI to enable supported operations, majority of interaction happens from within the code of OLE such as ingest of new records, editing of existing records, search and retrievals. 

The functionality of Docstore is mostly used by other processes like OLE.

However, for demonstration and testing docstore, a screen is provided with different tabs for different functions of docstore and information about docstore.

Please refer to the docstore application deployed on Dev server:

http://dev.docstore.ole.kuali.org/

2. Operations

2.1 Summary

 Shows the summary of record count for each category, type and formath

 

2.2 Ingest

 

Allows storing of documents in the document store. The input should be a Request XML with a standard schema and returns the Response XML with a list of ingested UUIDs.

2.2.1 Sending the request

URL: http://dev.docstore.ole.kuali.org/document

Method: POST

Parameters:

                  docAction=ingestContent

                  stringContent=<request.xml as described in the next section>

2.2.2 Request XML

Information about the ingest operation to be performed.

 <request>
    <user>ole-khuntley</user>
    <operation>ingest</operation>
    <requestDocuments>
        <ingestDocument id="1" category="work" type="bibliographic" format="marc">
            <content>
        See section "Sample Input XML for Ingest".
    </content>
        </ingestDocument>
    </requestDocuments>
</request>

2.2.3 Receiving the response

HttpResponse should be read and interpreted as given in the next section below.

2.2.4 Response XML

Information about the response from the service for the given request

 <response>
  <documents>
    	<document id="1" category="work" type="bibliographic" format="marc">
      	    <uuid>10000004</uuid>
	</document>
   </documents>
<user>ole-khuntley</user>
<operation>ingest</operation>
<status>Success</status>
<message>Documents ingested</message>
</response>

Each document that is ingested will be given a UUID as specified in the response.

2.3 Check-in

Check-in functionality allows to modify the content and metadata (additional attributes) of a document identified by its UUID (Universally Unique Identifier).

2.3.1 Sending the request

URL: http://dev.docstore.ole.kuali.org/document

Method: POST

Parameters:

                  docAction=checkIn

                  stringContent=<request.xml as described in the next section>

2.3.2 Request XML

           Information about the check in  operation to be performed     

 <request>
    <user>ole-khuntley</user>
    <operation>checkIn</operation>
    <requestDocuments>
        <ingestDocument id="10000004" category="work" type="bibliographic" format="marc">
            <content>
		See section "Sample Input XML for Check in".
	</content>
        </ingestDocument>
    </requestDocuments>
</request>

2.3.3 Receiving the response

HttpResponse should be read and interpreted as given in the next section below.

2.3.4 Response XML

            Information about the response from the service for the given request 

 <response>
  <documents>
    	<document id="1" category="work" type="bibliographic" format="marc">
      	    <uuid>10000004</uuid>
	</document>
   </documents>
<user>ole-khuntley</user>
<operation>checkIn</operation>
<status>Success</status>
</response>

2.4 Check-out

This operation retrieves the content of a document given its UUID.

2.4.1 Sending the request

URL: http://dev.docstore.ole.kuali.org/document

Method: POST

Parameters:

                  docAction=checkOut

                  uuid=<uuid of the document to be retrieved>

2.4.2 Receiving the response

            HttpResponse should be read and interpreted as given in the next section below.

2.4.3  Response XML

Content of the document with the given UUID.

2.5 Delete

Deletes a record from DocumentStore based on the given UUID.

2.5.1 Sending the request

URL: http://dev.docstore.ole.kuali.org/document

Method: POST

Parameters:

                  docAction=delete or deleteWithLinkedDocs

                  requestContent=<request.xml as described in the next section>

2.5.2  Request XML

         Information about the delete  operation to be performed. The "Id" attribute of <ingestDocument> should be a valid UUID of a previously ingested document.

 <request>
    <user>ole-khuntley</user>
    <operation>delete</operation>
    <requestDocuments>
<ingestDocument id="10000004"></ingestDocument>
 </requestDocuments>
</request>

2.5.3  Receiving the response

             HttpResponse should be read and interpreted as given in the next section below.

2.5.4  Response XML

 Information about the response from the service for the given request.

 <response>
  <documents>
    <document id="10000004"></document>
  </documents>
  <user>ole-khuntley</user>
  <operation>delete</operation>
  <status>Success</status>
</response>

2.6 Bulk Ingest (Admin function)

Bulk ingest process is used for loading the docstore repository with large amounts of document information.

It is usually done by an admin user during off-peak hours to avoid end-users experiencing slow down of docstore.

The input data is copied to a "ole-batchUpload" directory on the server (specified by documentstore.properties)

Go to http://dev.docstore.ole.kuali.org/admin.jsp and 'Bulk Ingest' tab.

Click the button to start the process. Once started, the process runs in background and waits for input files to become available in the "ole-batchUpload" directory.

As soon as a file is available, it is picked up for processing. After a file is ingested it is moved to a ".done" sub-directory and the next available file is picked up.

NOTE: Make sure that input directory ("ole-batchUpload") has the "rwx" permissions for the Tomcat process to create ".done" sub-directory and files. (If unix, run "chmod -R 777 ." in the input directory.)

To verify that the data is stored in DocumentStore, go to URL

           http://dev.docstore.ole.kuali.org

           Click "Refresh Summary" button in the Summary tab.

           Note the count for each DocType.

To verify that the data is indexed in DocStore, go to URL

        http://dev.docstore.ole.kuali.org/discovery

         Click "Refresh" button in the Summary tab.

         Note the count for each DocType.

2.7 Rebuild indexes (Admin function)

Sometimes the indexed data in Docstore may get corrupted. Or the data may need to be reindexed due to changes in indexing criteria and search/sort/facet rules.

In these cases the Docstore data can be re-indexed. This is also done by the Admin user.

Go to http://dev.docstore.ole.kuali.org/admin.jsp and 'Reindex' tab.

Click the 'Start' button to start the process.

Click the 'Status' button to view the status of the process.

Click the 'Stop' button to stop the process. The process is stopped after the current batch of data is reindexed.

To verify that the data is indexed in DocStore, go to URL

        http://dev.docstore.ole.kuali.org/discovery

         Click "Refresh" button in the Summary tab.

         Note the count for each DocType and verify with DocStore count.

2.8 Ingest Binary data (BagIt Requests)

When the document content is of (non-text or binary) format PDF, DOC etc, (as in the case of License Agreement documents) it is difficult to send it to docstore through a web page.

And when the request has to deal with more than one of such documents, it is even more difficult.

So, BagIt packaging standard is used to bundle such requests along with the files of these formats.

The request can be for ingest, checkin, checkout or delete.

This functionality can be demonstrated using the "BagIt Requests" tab in thehttp://dev.docstore.ole.kuali.orgscreen.

To submit a request of this type follow these steps:

1. Create a folder (e.g. /opt/docstore/upload/bagItRequests/ingest) (link to an attachment of zipped bagItRequests folder with ingest, checkin, checkout, delete folders)

     (Make sure this folder has write permission for all users.)

2. Create and copy the request.xml and the corresponding binary files into it.

3. Enter the full path of this folder in the text box for "BagIt Requests Directory".

4. Click the Submit button.

How a bagIt request is processed?

The utility code for handling BagIt request creates a 'Bag' (as per BagIt standard) out of the "BagIt Requests Directory".

The content of the bag is sent to Docstore via an HTTP connection.

Docstore unbags the received content into a temp folder, uses the request.xml to process the files in the temp folder if any.

Docstore creates a respose.xml file to record the outcome of the processed request

and copies it along with any files that are part of response (in case of checkout) to a temp folder.

Then a 'Bag' is created out of the temp folder and sent back to the client.

The utility code receives the content from Docstore, unbags the content to a temp folder (e.g. /opt/docstore/upload/bagItRequests/ingest/response).

The response.xml along with the temp folder name is to the browser.

3.0 Appendix

3.0.1  Sample Input XML for Ingest

 <request>
    <user>ole-khuntley</user>
    <operation>batchIngest</operation>
    <requestDocuments>
        <ingestDocument id="1" category="work" type="bibliographic" format="marc">
            <content><![CDATA[
            <collection xmlns="http://www.loc.gov/MARC21/slim">
    <record>
        <leader>#####nam#a22######a#4500</leader>
    <controlfield tag="001">3</controlfield>
        <controlfield tag="003">OCoLC</controlfield>
        <controlfield tag="005">20090213152530.7</controlfield>
        <controlfield tag="008">131031s########xxu###########000#0#eng#d</controlfield>
        <datafield tag="035" ind1=" " ind2=" ">
            <subfield code="a">(OCoLC)ocm62378465</subfield>
        </datafield>
        <datafield tag="040" ind1=" " ind2=" ">
             <subfield code="a">DLC</subfield>
             <subfield code="c">DLC</subfield>
             <subfield code="d">DLC</subfield>
             <subfield code="d">HLS</subfield>
             <subfield code="d">IUL</subfield>
        </datafield>
         <datafield tag="022" ind1=" " ind2=" ">
            <subfield code="a">1729-1070|20</subfield>
        </datafield>
        <datafield tag="029" ind1="1" ind2=" ">
            <subfield code="a">AU@|b000040176476</subfield>
            <subfield code="b">000040176476</subfield>
        </datafield>
        <datafield tag="037" ind1=" " ind2=" ">
            <subfield code="b">The Managing Editor, BIAC Journal, P.O. Box 10026, Gaborone, Botswana</subfield>
        </datafield>
        <datafield tag="042" ind1=" " ind2=" ">
            <subfield code="a">lc</subfield>
        </datafield>
        <datafield tag="043" ind1="1" ind2="0">
            <subfield code="a">f-bs---</subfield>
        </datafield>
        <datafield tag="050" ind1="0" ind2="0">
            <subfield code="a">HD70.B55|bB53</subfield>
        </datafield>
        <datafield tag="049" ind1=" " ind2=" ">
            <subfield code="a">IULA</subfield>
        </datafield>
         <datafield tag="210" ind1="1" ind2=" ">
            <subfield code="a">BIAC j.</subfield>
        </datafield>
         <datafield tag="222" ind1=" " ind2="0">
            <subfield code="a">BIAC journal</subfield>
        </datafield>
         <datafield tag="245" ind1="0" ind2="0">
            <subfield code="a">The administration of justice. BIAC journal</subfield>
        </datafield>
         <datafield tag="246" ind1="1" ind2="3">
            <subfield code="a">Botswana Institute of Administration and Commerce journal</subfield>
        </datafield>
        <datafield tag="260" ind1=" " ind2=" ">
            <subfield code="a">Gaborone, Botswana :|bBotswana Institute of Administration and Commerce</subfield>
            <subfield code="b">Botswana Institute of Administration and Commerce</subfield>
        </datafield>
        <datafield tag="300" ind1=" " ind2=" ">
            <subfield code="a">v. ;</subfield>
            <subfield code="c">24 cm.</subfield>
        </datafield>
        <datafield tag="300" ind1=" " ind2=" ">
            <subfield code="a">v. ;</subfield>
            <subfield code="c">24 cm.</subfield>
        </datafield>
        <datafield tag="300" ind1=" " ind2=" ">
            <subfield code="a">v. ;</subfield>
            <subfield code="c">24 cm.</subfield>
        </datafield>
        <datafield tag="310" ind1=" " ind2=" ">
            <subfield code="a">Semiannual</subfield>
        </datafield>
        <datafield tag="362" ind1="1" ind2=" ">
            <subfield code="a"> Began in 2004.</subfield>
        </datafield>
        <datafield tag="500" ind1=" " ind2=" ">
            <subfield code="a">Description based on: Vol. 1, no. 1 (May. 2004); title from cover.</subfield>
        </datafield>
        <datafield tag="500" ind1=" " ind2=" ">
            <subfield code="a">Latest issue consulted: Vol. 3, no. 1 (May 2006).</subfield>
        </datafield>
        <datafield tag="650" ind1="0" ind2=" ">
            <subfield code="a">Industrial management</subfield>
            <subfield code="z">Botswana</subfield>
            <subfield code="v">Periodicals.</subfield>
        </datafield>
        <datafield tag="650" ind1="0" ind2=" ">
            <subfield code="a">Occupational training</subfield>
            <subfield code="z">Botswana</subfield>
            <subfield code="v">Periodicals.</subfield>
        </datafield>
        <datafield tag="710" ind1="2" ind2=" ">
            <subfield code="a">Botswana Institute of Administration and Commerce.</subfield>
        </datafield>
        <datafield tag="850" ind1=" " ind2=" ">
            <subfield code="a">DLC</subfield>
        </datafield>
        <datafield tag="891" ind1="2" ind2="0">
            <subfield code="a">9853|81.1</subfield>
            <subfield code="a">v.</subfield>
            <subfield code="b">no</subfield>
            <subfield code="u">2</subfield>
            <subfield code="v">r</subfield>
            <subfield code="i">(year)</subfield>
            <subfield code="j">(month)</subfield>
            <subfield code="w">f</subfield>
            <subfield code="x">05</subfield>
        </datafield>
        <datafield tag="891" ind1="4" ind2="1">
            <subfield code="a">9863|81.1</subfield>
            <subfield code="a">1</subfield>
            <subfield code="b">1</subfield>
            <subfield code="i">2004</subfield>
            <subfield code="j">05</subfield>
        </datafield>
        <datafield tag="596" ind1=" " ind2=" ">
            <subfield code="a">1</subfield>
        </datafield>
    </record>
</collection>
                ]]>
            </content>
            <!--optional -->
            <additionalAttributes>
                <dateEntered></dateEntered>
                <lastUpdated></lastUpdated>
                <fastAddFlag></fastAddFlag>
                <supressFromPublic></supressFromPublic>
                <harvestable></harvestable>
                <status></status>
                <createdBy></createdBy>
                <updatedBy></updatedBy>
                <statusUpdatedOn></statusUpdatedOn>
                <statusUpdatedBy></statusUpdatedBy>
                <staffOnlyFlag>true</staffOnlyFlag>
            </additionalAttributes>
            <linkedIngestDocuments></linkedIngestDocuments>
        </ingestDocument>

    </requestDocuments>
</request>

3.0.2  Sample Input file for Check In

The "Id" attribute of <ingestDocument> should be a valid UUID of a previously ingested document.

 <request>
    <user>ole-khuntley</user>
    <operation>checkIn</operation>
    <requestDocuments>
        <ingestDocument id="1" category="work" type="bibliographic" format="marc">
            <content><![CDATA[
<collection xmlns="http://www.loc.gov/MARC21/slim">
    <record>
        <leader>01142cam 2200301 a 4500</leader>
        <controlfield tag="001">92005291</controlfield>
        <controlfield tag="003">DLC</controlfield>
        <controlfield tag="005">19930521155141.9</controlfield>
        <controlfield tag="008">920219s1993 caua j 000 0 eng</controlfield>
        <datafield tag="010" ind1=" " ind2=" ">
            <subfield code="a">92005291</subfield>
        </datafield>
        <datafield tag="020" ind1=" " ind2=" ">
            <subfield code="a">0152038655 :</subfield>
            <subfield code="c">$15.95</subfield>
        </datafield>
        <datafield tag="040" ind1=" " ind2=" ">
            <subfield code="a">DLC</subfield>
            <subfield code="c">DLC</subfield>
            <subfield code="d">DLC</subfield>
        </datafield>
        <datafield tag="042" ind1=" " ind2=" ">
            <subfield code="a">lcac</subfield>
        </datafield>
        <datafield tag="050" ind1="0" ind2="0">
            <subfield code="a">PS3537.A618</subfield>
            <subfield code="b">A88 1993</subfield>
        </datafield>
        <datafield tag="082" ind1="0" ind2="0">
            <subfield code="a">811/.52</subfield>
            <subfield code="2">20</subfield>
        </datafield>
        <datafield tag="100" ind1="1" ind2=" ">
            <subfield code="a">Sandburg, Carl,</subfield>
            <subfield code="d">1878-1967.</subfield>
        </datafield>
        <datafield tag="245" ind1="1" ind2="0">
            <subfield code="a">Arithmetic /</subfield>
            <subfield code="c">
                Carl Sandburg ; illustrated as an anamorphic adventure by Ted Rand.
            </subfield>
        </datafield>
        <datafield tag="250" ind1=" " ind2=" ">
            <subfield code="a">1st ed.</subfield>
        </datafield>

    </record>
</collection>
                ]]>
            </content>
        </ingestDocument>
    </requestDocuments>
</request>

4. Search

 This functionality allows documents to be searched for by giving keywords or phases. Searching can be based on category, type, format, search fields.

4.1  Quick Search

            Select Doc Category : Work

                      Doc Type : Bibliographic

                      Doc Format : ALL

   Searching on default condition(click search button without specifying any conditions) will give all the records in search result page.

            Select Doc Category : Work

                      Doc Type : Bibliographic

                      Doc Format : MARC

   Type one or more keywords in a text box.

   System shows records with any field matching one or more keywords.

4.2  Advanced Search

            Select Doc Category : Work

                      Doc Type : Bibliographic

                      Doc Format : MARC

   The drop down for search fields will be populated based on the category selected above.

   User specifies a search condition:

             Selects a field.

             Enters one or more keywords.

             Specifies whether the keywords should be searched for as "All of these", "Any of these" or "As a phrase".

                            "All of these"   - Any record with the selected field having all the entered keywords is included in the search results.

                           "Any of these" - Any record with the selected field having at least one of the entered keywords is included in the search results.

                           "As a phrase"  - Any record with the selected field having all the entered keywords in same order is included in the search results.

   User adds another condition:

             Chooses whether to apply this condition in addition to the previous one ("AND") or to apply this condition as an alternative to the previous one ("OR")  ("NOT"???),

             "AND"  - the conditions before and after this operator should be satisfied.

             "OR"    - one of the conditions before and after this operator should be satisfied.

             "NOT"  - the condition after this operator should not be satisfied.

   User repeats previous step as many times as needed using the ADD and DELETE links.

          [+]ADD : click on this link to add fields for a new search condition.

           [-]Delete : click on this link to delete the last search condition.

   Search is performed based on the conditions entered by the user.

For more information refer to the "Search" section in the  document :https://wiki.kuali.org/display/OLE/OLE+Search+Technical+Documentation

Operated as a Community Resource by the Open Library Foundation