PERFORMANCE BENCHMARK FOR INGEST
INGEST BIB DATA
Legacy data ingest:
Ingest 6 million bib records, processing time = 60hrs
Incremental data ingest:
On 2012-04-18 05:16:51,738 on DEV with batch size = 1000 (with about 1.5 million records already ingested)
Bulk Ingest Process for 10,000 bib records:
Ingesting Time: 0:3:41.562(H:M:S.ms)
Indexing Time: 0:1:10.201(H:M:S.ms)
Total Process Time: 0:4:51.854(H:M:S.ms)
INGEST INSTANCE DATA
Legacy data ingest:
Ingest 10 million instance records, processing time may take 42 days!
Detailed time breakdown for ingesting 1000 records:
Time taken for ingest 1000 instance records :
Ingesting Time: 3.33 minutes
Time for Linking to Bib records: 2.51minutes
More detailed please read the xls file.
PERFORMANCE MINIMUM REQUIREMENT
Ingest about 20 million legacy data (including bib, instance..) need to finish in one week!
PERFORMANCE ISSUES
ARCHITECTURE REVIEW
COMMENTS ON DOCSTORE ARCHITECTURE
John Pillans thought we actually don't need the JackRabbit