-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-40605: Improve Algolia audit job's reliability #144
Commits on Sep 1, 2023
-
Capture and log exceptions while queuing ingest
There can be all sorts of issues when queueing an ingest, so we'll just log their exceptions and continue so we can get a sense of why these documents are failing.
Configuration menu - View commit details
-
Copy full SHA for 2133095 - Browse repository at this point
Copy the full SHA 2133095View commit details -
Configuration menu - View commit details
-
Copy full SHA for db6da69 - Browse repository at this point
Copy the full SHA db6da69View commit details -
Fix "baseUrl" field in Algolia DocumentRecord
This was baseURL in the Pydantic model, but should be baseUrl to match the existing Algolia schema.
Configuration menu - View commit details
-
Copy full SHA for 1aaec62 - Browse repository at this point
Copy the full SHA 1aaec62View commit details
Commits on Sep 5, 2023
-
Include exception trace in JSON log
Rather than use logging.exception, which places the stack trace outside the structured log, using logging.error instead and log the exception within the JSON structured log message.
Configuration menu - View commit details
-
Copy full SHA for 796499c - Browse repository at this point
Copy the full SHA 796499cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 743ddd7 - Browse repository at this point
Copy the full SHA 743ddd7View commit details -
Add more logging context to ltd ingest handler
- Format the kafka key/value data as a dict for natural JSON representation; the json() method was creating a string. - Bind this context to the logger so that all log messages share it.
Configuration menu - View commit details
-
Copy full SHA for f4b0a68 - Browse repository at this point
Copy the full SHA f4b0a68View commit details -
Include more context about the uploaded records
- Add the number of records and the surrogate key of those records so they can be searched later.
Configuration menu - View commit details
-
Copy full SHA for f20e5a5 - Browse repository at this point
Copy the full SHA f20e5a5View commit details -
Configure the logger when running ook audit and other CLI apps to get structured logging. This is useful since the CLI is primarily used for Kubernetes Jobs.
Configuration menu - View commit details
-
Copy full SHA for 4161eb1 - Browse repository at this point
Copy the full SHA 4161eb1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 190bbb2 - Browse repository at this point
Copy the full SHA 190bbb2View commit details -
Use logger.exception in audit CLI
See if this exception is now nicely formatted by structlog for us.
Configuration menu - View commit details
-
Copy full SHA for 5aa6b82 - Browse repository at this point
Copy the full SHA 5aa6b82View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6c5ba0d - Browse repository at this point
Copy the full SHA 6c5ba0dView commit details -
Use exception logger for Kafka handler errors
Turns out structlog does format the exception trace in the structured log after all, so this is fine.
Configuration menu - View commit details
-
Copy full SHA for a77abf8 - Browse repository at this point
Copy the full SHA a77abf8View commit details -
Configuration menu - View commit details
-
Copy full SHA for b4d3225 - Browse repository at this point
Copy the full SHA b4d3225View commit details -
Configuration menu - View commit details
-
Copy full SHA for c812542 - Browse repository at this point
Copy the full SHA c812542View commit details