Provide on-demand diagnostic view of Host LMS ingests

Description

Goal

Improve diagnostic tools for DCB Admins when troubleshooting potential ingest and indexing causes of request resolution issues.

In Scope

Existing Host LMS grid

  • Extend existing Host LMS table to show all configured Host LMS, whether or not enabled for ingest

  • Add “Ingest Status” column to existing Host LMS datagrid

  • For each Host LMS, in Ingest Status column

    • If not enabled for ingest, display “Not enabled”

    • If enabled for ingest,

      • display Ingest Status

      • link ingest Status to Ingest Status section of Host LMS detail page

  • For Ingest Status:

    • check for Last Ingest Status Query

    • If Last Ingest Status Query is not known, display “Enabled”

    • If Last Ingest Status Query is known, display timestamp yyyy-MM-dd HH:mm of highest recorded timestamp from source

Ingest Status Query

  • When Ingest Status is clicked from Host LMS summary table

    • run ingest status query

      • if successful

        • link through to Ingest Status Details

        • save timestamp of Last Ingest Status Query

        • update Ingest Status on Host LMS datagrid

      • if not successful

        • update Ingest Status on Host LMS datagrid to “Unknown”

    • log query status results, including

      • Last Ingest Status Query (timestamp, default: now)

      • Highest Recorded Timestamp (default: UNKNOWN / -1)

      • Record Count (default: UNKNOWN / -1)

      • Most Recently Ingested Bib Cluster Record ID (default: -1)

    • link Ingest Status to Ingest Status section for selected Host LMS details page

    • add tooltip to Ingest Status showing

      • Last Ingest Status Query: <timestamp>

      • Record Count: <record count>

Ingest Status Details

Add collapsible Ingest Status section to Host LMS details page (after Client Config section) displaying

  • ingest enabled (true, false)

  • ingested record count (integer)

  • Most Recently Ingested Bib Cluster Record ID

  • highest timestamp from that source

  • Last Ingest Status Query

Host LMS Details

  • Update existing Host LMS details page to include expand/collapse link (in line with Patron Request details page)

Attachments

3

Activity

Show:

Casey Henderson December 5, 2023 at 4:03 PM

For sure, free any time on Wednesday to discuss.

Jag Goraya December 5, 2023 at 3:53 PM

Something that’s come up from some of the beta testing troubleshooting: additional data would be useful to show for initial full ingest runs per Host LMS:

  • timestamp of when a full ingest has been started

  • length of time for full ingest to complete

  • record count from full ingest

The tricky thing with this is that calculating the harvest duration based on earliest and highest recorded timestamps leads to a potentially misleading ingest run length, because incremental ingests (which can happen at a 2-5 minute interval) will raise the highest recorded timestamp if a new record is added.

So, some unknowns to consider with implementing this for full ingest cycles:

  • Is there a way to capture start and end times for full harvests independently of incremental updates?

  • Where is the best place to store this once-per-full-ingest information, eg, on the hostlms ingest status query record?

  • Do we want to treat this as one-off, or maintain a history of (full) ingest run stats for future diagnostics, performance troubleshooting and downtime planning?

maybe we can pick this up on Wednesday before you get stuck into the detail?

Jag Goraya November 30, 2023 at 1:20 PM
Edited

Thanks for recording this requirement . It’s definitely a different thing than adding ingest sources, but let’s keep it here for now while I work out what to do with it.

  • It’ll definitely be a separate ticket to extend the existing diagnostic view of Host LMS details page:

  • Editing key-pairs is a different ticket again, which will apply to a less raw, more user-friendly version of the page - we’re definitely aren't ready for that yet (as you say, in the future), and it’ll need 's UX input

It does make me realise that rather than adding a separate Ingest Status page, we should just add an Ingest Status section to the Host LMS details page. I’ll update the description accordingly.

Casey Henderson November 30, 2023 at 11:42 AM

just capturing a conversation regarding the HostLMS details page with from this morning here so that it doesn’t get lost. Not sure if this is the right place for it - apologies if it’s not, wanted to put it somewhere so we don’t lose the info. Might need its own separate ticket, happy to raise if so.

Expanding the HostLMS Details Page to handle different client configs

  • All HostLMS objects have a clientConfig attribute - this is currently displayed as JSON, and is different / has different attributes for pretty much every HostLMS (see screenshots). These appear to differ based on their lmsConfigClass.

  • The HostLMS details page needs to be adapted to display these attributes for each HostLMS (and not display non-relevant ones).

  • In the future, we will want to allow the user to edit these values as key-value pairs - to do this we’d need to add a form on the HostLMS details page.

  • StLouis is a good example of a HostLMS with a quite complicated clientConfig object. The final screenshot shows how this is currently being displayed in DCB Admin - currently the information isn’t being shown at all.

Details

Assignee

Reporter

Components

Priority

Created November 29, 2023 at 9:32 PM
Updated October 17, 2024 at 10:42 PM