Provide on-demand diagnostic view of Host LMS ingests
Description
Attachments
Activity

Casey Henderson December 5, 2023 at 4:03 PM
For sure, free any time on Wednesday to discuss.

Jag Goraya December 5, 2023 at 3:53 PM
Something that’s come up from some of the beta testing troubleshooting: additional data would be useful to show for initial full ingest runs per Host LMS:
timestamp of when a full ingest has been started
length of time for full ingest to complete
record count from full ingest
The tricky thing with this is that calculating the harvest duration based on earliest and highest recorded timestamps leads to a potentially misleading ingest run length, because incremental ingests (which can happen at a 2-5 minute interval) will raise the highest recorded timestamp if a new record is added.
So, some unknowns to consider with implementing this for full ingest cycles:
Is there a way to capture start and end times for full harvests independently of incremental updates?
Where is the best place to store this once-per-full-ingest information, eg, on the hostlms ingest status query record?
Do we want to treat this as one-off, or maintain a history of (full) ingest run stats for future diagnostics, performance troubleshooting and downtime planning?
maybe we can pick this up on Wednesday before you get stuck into the detail?

Jag Goraya November 30, 2023 at 1:20 PMEdited
Thanks for recording this requirement . It’s definitely a different thing than adding ingest sources, but let’s keep it here for now while I work out what to do with it.
It’ll definitely be a separate ticket to extend the existing diagnostic view of Host LMS details page:
Editing key-pairs is a different ticket again, which will apply to a less raw, more user-friendly version of the page - we’re definitely aren't ready for that yet (as you say, in the future), and it’ll need 's UX input
It does make me realise that rather than adding a separate Ingest Status page, we should just add an Ingest Status section to the Host LMS details page. I’ll update the description accordingly.

Casey Henderson November 30, 2023 at 11:42 AM
just capturing a conversation regarding the HostLMS details page with from this morning here so that it doesn’t get lost. Not sure if this is the right place for it - apologies if it’s not, wanted to put it somewhere so we don’t lose the info. Might need its own separate ticket, happy to raise if so.
Expanding the HostLMS Details Page to handle different client configs
All HostLMS objects have a clientConfig attribute - this is currently displayed as JSON, and is different / has different attributes for pretty much every HostLMS (see screenshots). These appear to differ based on their lmsConfigClass.
The HostLMS details page needs to be adapted to display these attributes for each HostLMS (and not display non-relevant ones).
In the future, we will want to allow the user to edit these values as key-value pairs - to do this we’d need to add a form on the HostLMS details page.
StLouis is a good example of a HostLMS with a quite complicated clientConfig object. The final screenshot shows how this is currently being displayed in DCB Admin - currently the information isn’t being shown at all.
Details
Assignee
Ian Ibbotson (Use this one)Ian Ibbotson (Use this one)Reporter
Jag GorayaJag GorayaLabels
Components
Parent
Priority
TBD
Details
Details
Assignee
Reporter

Goal
Improve diagnostic tools for DCB Admins when troubleshooting potential ingest and indexing causes of request resolution issues.
In Scope
Existing Host LMS grid
Extend existing Host LMS table to show all configured Host LMS, whether or not enabled for ingest
Add “Ingest Status” column to existing Host LMS datagrid
For each Host LMS, in Ingest Status column
If not enabled for ingest, display “Not enabled”
If enabled for ingest,
display Ingest Status
link ingest Status to Ingest Status section of Host LMS detail page
For Ingest Status:
check for Last Ingest Status Query
If Last Ingest Status Query is not known, display “Enabled”
If Last Ingest Status Query is known, display timestamp
yyyy-MM-dd HH:mm
of highest recorded timestamp from sourceIngest Status Query
When Ingest Status is clicked from Host LMS summary table
run ingest status query
if successful
link through to Ingest Status Details
save timestamp of Last Ingest Status Query
update Ingest Status on Host LMS datagrid
if not successful
update Ingest Status on Host LMS datagrid to “Unknown”
log query status results, including
Last Ingest Status Query (timestamp, default: now)
Highest Recorded Timestamp (default: UNKNOWN / -1)
Record Count (default: UNKNOWN / -1)
Most Recently Ingested Bib Cluster Record ID (default: -1)
link Ingest Status to Ingest Status section for selected Host LMS details page
add tooltip to Ingest Status showing
Last Ingest Status Query: <timestamp>
Record Count: <record count>
Ingest Status Details
Add collapsible Ingest Status section to Host LMS details page (after Client Config section) displaying
ingest enabled (true, false)
ingested record count (integer)
Most Recently Ingested Bib Cluster Record ID
highest timestamp from that source
Last Ingest Status Query
Host LMS Details
Update existing Host LMS details page to include expand/collapse link (in line with Patron Request details page)