Dev Ops Planning

Solution for maintaining various
mappings. No more google
spreadsheets

Post Go-Live

Operations

• agencies.sh (based on google
  spreadsheet)

• Mappings (based on google spreadsheet)

  • Reference value mapping import

  • Numeric range mapping import

in progress

• configuration repository set up to maintain controlled versions

• going forward, these will be managed within DCB Admin

HA Support for dcb-service and dcb-locate

Dress Rehearsal

Operations

Utilize same base URL for both dcb-service and dcb-locate

Dress Rehearsal

Operations

wb: How would path based routing be configured in our load balancer? Given the base URL would each service has a unique path root or would one service be considered the default? Which of these examples do you envision or recommend?

• wb:
  wb-k18.int.folio.ebsco.com/dcb-service,
  wb-k18.int.folioebsco.com/dcb-locate,
  wb-k18.int.folio.ebsco.com/keycloak,
  wb-k18.init.folio.ebsco.com/dcb-admin

• wb: Anything that is not “/dcb-locate”, “/keycloak”, “/dcb-admin” implies dcb-service

• wb: Should we not use path based routing and use unique DNS record for each such as “keycloak-wb-k18…”, “dcb-service-wb-k18…”, “dcb-locate-wb-k18…”, “dcb-admin-wb-k18…”

Admin App AWS deployment

Dress Rehearsal

Operations

ILS Credential Management

Dress Rehearsal

Operations

Preferred to have AWS SSM support instead of DB storage

• wb: How each library controls their sensitive info such as keys, passwords, user accounts and how that information is securely incorporated.
  For example, library/ILS keys, secrets, passwords, user should not need to be in a json payload. dcb-service has the member v5 uuid and could use that to access content directory from a secure vault.

• wb: If we do not have each library responsible for their sensitive content, how do we rotate keys, secrets, passwords, users names?

• wb: Various identifiers are currently provided in the JSON payload. What must be done when a member library wants to “re-brand” themselves? Is this simply updating “ingest” the JSON payload or must a lot of DB and search domain records be modified. Specifically:
  (Vijay: Hoping that this part of the Admin App)

  • wb: Library name

  • wb: Library LMS url

  • wb: Library LMS user name, key, secret

Release notes should have instructions for Upgrade

Dress Rehearsal

Documentation

Completed

Releases

Observability - Synthetic Canaries

Dress Rehearsal

Operations

Not sure what we can monitor

Tests

Dress Rehearsal

Development

• Integration tests

• E2E tests

• Post-deployment automated tests

• Ensure Code Coverage is at 80%

Role of UUIDs and its generation

Dress Rehearsal

Open Question

Issue Triage & Production Support

Dress Rehearsal

Process

• Issues comes via EBSCOConnect

• Issue escalated to K-Int

• This needs to be discussed internally

  • Mark A, Mark V, Christopher, Ian & Tim

  • Bob Cronk - 5 Support specialists

  • Tim to define the SLA

IN PROGRESS

HA Support for Admin App

Go-Live

Operations

ARM64 Support

Go-Live

Operations

Prefer that we deliver this sometime in March so that we can do additional work and testing

COMPLETED

This is natively supported when building pure java version of DCB

Ops Training

Go-Live

Operations

Train operations team on

• Overall Application architecture

• Troubleshooting techniques

• Explanation of logs

• Documented anomalies or key patterns to watch out for in logs

Provide run books for the most common errors that one might encounter

Go-Live

Operations

Observability - Healthcheck

Go-Live

Operations

• Include version in the healthcheck response

• It would be helpful if dcb-service healthcheck listed health and unhealthy LMSs

completed

Versions is in /info response

Keycloak Health check endpoint
returning duplicates

Go-Live

Open Question

Compare https://keycloak-wb-k18.int.folio.ebsco.com/health with https://keycloak.sph.k-int.com/health

On Call Support

Go-Live

Process

Performance Evaluation and Tuning

Go-Live

Development

Document upgrade scenarios for each
service (dcb-service, dcb-locate) and
infrastructure component (keycloak,
OS, Postgres)

Go-Live

Process

1. wb: What are the migration or upgrade steps, if any, for the database and search domain?

2. wb: Is roll-back possible or must the database and search domain be restored from backup?

3. wb: How can we stage an upgrade without impacting the live LMS/ILS ?

   1. wb: dcb-service has DCB_SCHEDULED_TASKS_ENABLED boolean and DCB_INGEST_INTERVAL. What does dcb-locate have to disable populate?

   2. wb:

      1. Does dcb-{locate,service,admin) have any persistent references of each other and external resources it accesses?

      2. Are tasks environment variables ephemeral and can their values change without needing database changes?

      3. We need to be able to quickly (within 60 minutes or less) spin up dcb production in a totally different AWS region and account. What, if any, dcb specific (not infrastructure) details do we need to consider?

Backup recovery procedure

Go-Live

Process

Post deployment verification

Go-Live

Operations

Multi-Tenancy Support

Post Go-Live

Development

Features

Language
Location
Ingestion
FOLIO circulation
Any outstanding issues that requires development

Accessibility - WCAG 2.1 compliance
for Admin app

Development

When/Do we need this ?
Check with Tim regarding compliance

completed

Penetration Testing

• K-Int to address vulnerabilities found

• Admin App in Scope and may be other external endpoints