OLE Batch Export Architechture

Description

Export files of bibliographic and instance data from OLE for transmission to a variety of destinations, including the public discovery layer, external utilities, or other databases. Exported files may be of any size, ranging from several records to copies of the entire bibliographic and instance portions of the database. Exports will be governed by profiles that support selection and data manipulation according user-defined criteria.

Exports may be ad hoc jobs performed immediately, may be scheduled to run at a particular date and time, or may be setup to run at regular intervals (e.g., nightly export to apply updates to the discovery layer). Output will be in a standard format, such as MARC or MARCXML.

Support of the public discovery layer is a critical export function. OLE does not provide a public discovery layer, and as a result, bibliographic and instance data must be exported from OLE for use by the discovery layer. Existing discovery layer systems expect MARC21 bibliographic records with a subset of associated instance data in local data fields. Institutions have their own particular requirements for type of export (full database, incremental updates) and scheduling to maintain a variety of discovery layer products. OLE needs to be able to export a full copy of the database and to export incremental updates to bib or instance data as required by the library. Updates include additions, changes, and deletions, as well as changes in staff-only status. Data that formerly were staff-only should be added to the export.

Design Technical

• Psuedo Code

1. User creates a batch profile using the options provided in batch profile screen
2. User create a batch process through batch process screen by selecting the profile to execute the output file, batch size and email id for the report
3. User clicks on run now to execute the batch immediately. 
4. OLEBatchProcessDefinitionDocument creates the process definition document and call s the OLEBatchSchedulerService, which starts the job by executing the OLEBatchProcessAdhocStep step 
5. In OLEBatchProcessAdhocStep the BatchProcessFactory is invoked with the process type based on which the appropriate process is created
6. The processBatch(OLEBatchProcessDefinitionDocument processDef,OLEBatchProcessJobDetailsBo jobBo) method is called to perform the batch process as per the profile information provided by the user.
   
   
   1. BatchProcessExportData
      This batchprocess is involved in performing export of bib / intance information based on the profile provided by the user. It has the following methods:
      1. loadProfile(OLEBatchProcessDefinitionDocument processdef)
         1. Performs the profile data load from the database as per the profile id provided in the process definition document
         2. In case of Incremental export, the last export job details is retrieved and that value is used when data is queried in prepareForRead() and getNextBatch() methods
      2. prepareForRead()
         1. Reads the initial set of data for Export using solr query based on the profile information such as Data to export, filters etc.
         2. Calls the ExportDataService.getExportDataBySolr(List<SolrDocument> solrDocumentList, OLEBatchProcessProfileBo profile) method with the solr result and the profile and does the bib checkout for each solr document
         3. If the data to export is Bib only only the checkout bib is processed else if the data to export is BibAndInstance all instance data that belong to the bib is checked out
         4. Instance data checkout is verified whether it is a conventional Instance or an EInstance and processed accordingly
         5. Holding and Item mapping provided are used for mapping the holding and item information into datafileds and subfields
         6. All the data retrieved is added to the BibliographicRecord and is converted to xml format and returned
      3. prepareForWrite()
         1. Identifies the user given path (from process definition document) where the output file shall be written to, if the path is valid and the system can write the file it will write the output to the said location, else it will output the file to a default location and location will be specified in the job report
      4. processBatch()
         1. Data retrived using prepareForRead() and getNextBatch() batch method is written to file as a mrc file or xml depending on the process definition document data
         2. performs the write operation of the batch export and updates the success, failure and totalcount of the job
      5. getNextBatch()
         1. If there are more batches of data to be exported this method retrieves the next set of data
         2. performs all steps that take place in prepareForRead() method
         3. And calls processBatch() method to perform the batch export
      6. updateJobProgress()
         1. Updates the job record with details such as time taken, record count and status

• Flow Charts
  
  

Class Diagram:

BATCH PROCESS EXPORT

 UI Screen

ER Diagram: Batch Process Export

Batch Process

Data Model

Entity : BATCH DEFINITION