Dev Ops Planning

Task	Expected Timeframe	Category	Notes	Status
Features			Language Location Ingestion FOLIO circulation Any outstanding issues that requires development
HA Support for dcb-service and dcb-locate	Dress Rehearsal	Operations
HA Support for Admin App	Go-Live	Operations
Utilize same base URL for both dcb-service and dcb-locate	Dress Rehearsal	Operations	wb: How would path based routing be configured in our load balancer? Given the base URL would each service has a unique path root or would one service be considered the default? Which of these examples do you envision or recommend? wb: wb-k18.int.folio.ebsco.com/dcb-service, wb-k18.int.folioebsco.com/dcb-locate, wb-k18.int.folio.ebsco.com/keycloak, wb-k18.init.folio.ebsco.com/dcb-admin wb: Anything that is not “/dcb-locate”, “/keycloak”, “/dcb-admin” implies dcb-service wb: Should we not use path based routing and use unique DNS record for each such as “keycloak-wb-k18…”, “dcb-service-wb-k18…”, “dcb-locate-wb-k18…”, “dcb-admin-wb-k18…”
ARM64 Support	Go-Live	Operations	Prefer that we deliver this sometime in March so that we can do additional work and testing
Admin App AWS deployment	Dress Rehearsal	Operations
ILS Credential Management	Dress Rehearsal	Operations	Preferred to have AWS SSM support instead of DB storage wb: How each library controls their sensitive info such as keys, passwords, user accounts and how that information is securely incorporated. For example, library/ILS keys, secrets, passwords, user should not need to be in a json payload. dcb-service has the member v5 uuid and could use that to access content directory from a secure vault. wb: If we do not have each library responsible for their sensitive content, how do we rotate keys, secrets, passwords, users names? wb: Various identifiers are currently provided in the JSON payload. What must be done when a member library wants to “re-brand” themselves? Is this simply updating “ingest” the JSON payload or must a lot of DB and search domain records be modified. Specifically: (Vijay: Hoping that this part of the Admin App) wb: Library name wb: Library LMS url wb: Library LMS user name, key, secret
Ops Training	Go-Live	Operations	Train operations team on Overall Application architecture Troubleshooting techniques Explanation of logs Documented anomalies or key patterns to watch out for in logs
Release notes should have instructions for Upgrade	Dress Rehearsal	Documentation		COMPLETED Releases
Provide run books for the most common errors that one might encounter	Go-Live	Operations
Observability - Synthetic Canaries	Dress Rehearsal	Operations	Not sure what we can monitor
Multi-Tenancy Support	Post Go-Live	Development
Observability - Healthcheck	Go-Live	Operations	Include version in the healthcheck response It would be helpful if dcb-service healthcheck listed health and unhealthy LMSs	COMPLETED Versions is in `/info` response
Keycloak Health check endpoint returning duplicates	Go-Live	Open Question	Compare https://keycloak-wb-k18.int.folio.ebsco.com/health with https://keycloak.sph.k-int.com/health
Tests	Dress Rehearsal	Development	Integration tests E2E tests Post-deployment automated tests Ensure Code Coverage is at 80%
On Call Support	Go-Live	Process
Solution for maintaining various mappings. No more google spreadsheets	Dress Rehearsal	Operations	agencies.sh (based on google spreadsheet) Mappings (based on google spreadsheet) Reference value mapping import Numeric range mapping import	IN PROGRESS configuration repository set up to maintain controlled versions going forward, these will be managed within DCB Admin
Role of UUIDs and its generation	Dress Rehearsal	Open Question
Performance Evaluation and Tuning	Go-Live	Development
Accessibility - WCAG 2.1 compliance for Admin app		Development	When/Do we need this ? Check with Tim regarding compliance	COMPLETED
Production Support	Dress Rehearsal	Process	Issues comes via EBSCOConnect Issue escalated to K-Int This needs to be discussed internally Mark A, Mark V, Christopher, Ian & Tim Bob Cronk - 5 Support specialists Tim to define the SLA
Document upgrade scenarios for each service (dcb-service, dcb-locate) and infrastructure component (keycloak, OS, Postgres)	Go-Live	Process	wb: What are the migration or upgrade steps, if any, for the database and search domain? wb: Is roll-back possible or must the database and search domain be restored from backup? wb: How can we stage an upgrade without impacting the live LMS/ILS ? wb: dcb-service has `DCB_SCHEDULED_TASKS_ENABLED` boolean and `DCB_INGEST_INTERVAL`. What does dcb-locate have to disable populate? wb: Does dcb-{locate,service,admin) have any persistent references of each other and external resources it accesses? Are tasks environment variables ephemeral and can their values change without needing database changes? We need to be able to quickly (within 60 minutes or less) spin up dcb production in a totally different AWS region and account. What, if any, dcb specific (not infrastructure) details do we need to consider?
Backup recovery procedure	Go-Live	Process
Post deployment verification	Go-Live	Operations
Issue Triage	Dress Rehearsal	Process
Penetration Testing			K-Int to address vulnerabilities found Admin App in Scope and may be other external endpoints