Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-40605: Improve Algolia audit job's reliability #144

Merged
merged 14 commits into from
Sep 5, 2023

Commits on Sep 1, 2023

  1. Capture and log exceptions while queuing ingest

    There can be all sorts of issues when queueing an ingest, so we'll just
    log their exceptions and continue so we can get a sense of why these
    documents are failing.
    jonathansick committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    2133095 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    db6da69 View commit details
    Browse the repository at this point in the history
  3. Fix "baseUrl" field in Algolia DocumentRecord

    This was baseURL in the Pydantic model, but should be baseUrl to match
    the existing Algolia schema.
    jonathansick committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    1aaec62 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2023

  1. Include exception trace in JSON log

    Rather than use logging.exception, which places the stack trace outside
    the structured log, using logging.error instead and log the exception
    within the JSON structured log message.
    jonathansick committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    796499c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    743ddd7 View commit details
    Browse the repository at this point in the history
  3. Add more logging context to ltd ingest handler

    - Format the kafka key/value data as a dict for natural JSON
      representation; the json() method was creating a string.
    - Bind this context to the logger so that all log messages share it.
    jonathansick committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    f4b0a68 View commit details
    Browse the repository at this point in the history
  4. Include more context about the uploaded records

    - Add the number of records and the surrogate key of those records so
      they can be searched later.
    jonathansick committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    f20e5a5 View commit details
    Browse the repository at this point in the history
  5. Format logger for ook cli

    Configure the logger when running ook audit and other CLI apps to get
    structured logging. This is useful since the CLI is primarily used for
    Kubernetes Jobs.
    jonathansick committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    4161eb1 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    190bbb2 View commit details
    Browse the repository at this point in the history
  7. Use logger.exception in audit CLI

    See if this exception is now nicely formatted by structlog for us.
    jonathansick committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    5aa6b82 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    6c5ba0d View commit details
    Browse the repository at this point in the history
  9. Use exception logger for Kafka handler errors

    Turns out structlog does format the exception trace in the structured
    log after all, so this is fine.
    jonathansick committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    a77abf8 View commit details
    Browse the repository at this point in the history
  10. Add change log entry

    jonathansick committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    b4d3225 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    c812542 View commit details
    Browse the repository at this point in the history