Objects API registration backend - Improve JSON content field / mapping editor

[alextreme]

Thema / Theme

Plugins

Omschrijving / Description

Taiga DH ZGW 539

Currently the JSON Schema of the 'productaanvraag' isn't very strict. This allows the Form Designer in Open Formulieren to easily make changes to the form, however the consequence is that the data in the 'productaanvraag' changes. Fieldname changes for instance cause the data to be placed at a different location or with a different type, leading to problems in GZAC when processing the 'productaanvraag'

As such, the GZAC team would like to adhere to a stricter JSON schema, this to ensure that the fieldnames stay the same. The JSON schema is to be a more rigid contract between Open Formulieren and GZAC.

In order to assist with this Open Formulieren has a 'JSON content editor' field in the Objects API registration backend. This is documented in:

https://open-forms.readthedocs.io/en/latest/manual/templates.html?highlight=Objects%20api#objecten-api-registratie

The problem is that this 'JSON content editor' is rather basic and misses the ability to validate the form against the 'productaanvraag', an editor doesn't know if changes made in the form or in the editor will lead to a valid 'productaanvraag', the only way to currently check this is by manually sending in a form submission. Adding a validator will ensure that changes in the form and editor can be properly verified.

Analysis

The object type definition and the associated json schema is leading. In this user story, we'll update the UI to configure the data sent to the objects API to remove human errors from the process as much as possible and run validation at form-design time rather than see errors during runtime.

This will prepare us in the direction of the new UI design - while staying backwards compatible.

Some topics have no clear answers yet and we will get to them in due time:

• what does the UI/process look like when updating to a new version of the object type?
• how to deal with the array of uploaded attachment URLs in the documenten API?
• how to (automatically migrate from the legacy system to the new approach?
• what to do with composite data, e.g. field1 concatenated with field2, and registering that somewhere? (user defined variable is now an option)
• include common composite operations -> let's first tackle the basis
• datatype validation - we don't have information about the type inside an object/array -> yolo

Tasks

This rework touches many aspects - please break them up in separate PRs that are easy to review and merge without breaking the current situation.

Phase 1: User interface 1

The changes here are useful for the legacy version too.

• Add a field objecttypes_service to the ObjectsAPIConfig model. We will use this to retrieve a list of available object types and present them to the user.

• Add a data migration to try to create this service from existing objecttype URLs either in ObjectsAPIConfig or form registration objects. Ideally, this should only yield one base URL, but there could be more. If there are more, pick the first. This API service will be incomplete, since we don't know the API token -> add to release notes this needs to be added.

• Extend the registration plugin check to check for access to the objecttypen list.

• Add API endpoints:

  • /api/v2/registration/plugins/objects-api/object-types - give a list of available object types from the configured service
  • /api/v2/registration/plugins/objects-api/object-types/:uuid/versions - give a list of available versions for a given object type

  Possibly this can be a single endpoint if that proves to be more practical for the frontend code - let's figure this out from the frontend code.

• Replace the RJSF form for the Objects API configuration with a React-based form (similar to the ZGW rework done for zaakeigenschappen & the setup for Camunda configuration)

  • Change the objecttype configuration field into a dropdown of available object types. Simple at first, in the future we can upgrade this to react-select with autocomplete/search capabilities
  • When an objecttype is selected, present the available versions.
  • Ensure that a broken/incomplete configuration of the objecttype service does not result in existing objecttype values being cleared - you may need some conditional logic to present either a dropdown or a text field with the raw value in this transitional period.

Phase 2: backend updates 1

The changes here prepare the backend for new/updated UI. Another phase will add additional validation constraints. It
is important we stay backwards compatible to give user time to upgrade to the new mechanisms.

• Add a version field to ObjectsAPIOptionsSerializer - with a default of 1.

  • The default should be set so that legacy exports are interpreted correctly (just a data migration
    is not sufficient)
  • The current run-time behaviour should be applied if the options["version"] == 1

• Extend serializer validation - content_json & similar may only be provided if the
  version is 1

• Ensure that a (resolved) JSON schema for the selected object type is available to the
  frontend. This is the jsonSchema property in the object type version resource.

  This may require an additional API endpoint, this may require some processing on the backend
  to resolve the schema - alternatively if good frontend libraries exist, those could also
  take care of the resolution.

Note

We decided not to implement such validation for now. We expect users to choose a "correct" objecttype version (with a JSON schema other than {} for instance). Even if they do, they will end up seeing that no option is available to map form variables to a json path)

• Add a (JSON) field data_mapping. The data structure must describe how each
  property from the above jsonSchema is filled with a form variable. It's probably
  wise to keep a structure similar to the jsonSchema, but let's see what works. jsonpath
  like expressions are likely going to be problematic with objects/arrays, so ensure we
  can map those kind of variables. It's important that we record the key of the
  variable so we can do a simple lookup in submission.data[variable_key] to get to the
  variable (this should be a FormioData datastructure which uses glom under the hood).

Phase 3: User interface 2

This may be challenging - we want to manage the mapping/registration information from
the variables tab, but the configuration should be persisted in the options of the
registration backend(s).

Also keep in mind that multiple registration backend configuration can be provided!

• Add a column "Registration" to the variables tab. This applies to all types of
  variables: Component, user defined and static.

• For each variable/row, show the summary and controls to modify the configuration

  • Each registration backend (at the form level) needs to be represented
  • For objects API, display a summary like (Objects) record > data > foo.bar
    (property names may contain periods!)
  • Add an 'edit' icon and make either just the icon or the entire summary + icon
    clickable
  • Editing the configuration opens a modal, displaying:
    • text "Registration configuration for objects API ($name)" with the name being
      the name given to the registration option
    • A dropdown selecting the relevant property from the json schema. Only present
      options that are not mapped yet.
    • A read-only "technical path", could be the same as the summary from earlier
  • Upon saving the modal, the mapping is added to the selected registration backend
    options.

Phase 4: backend updates 2, validation

The backend needs to validate the configuration made by the frontend and provide useful
feedback.

• Validate the data mapping configuration against the specified object type version
  schema:

  • Validate that all required properties are mapped/present, note that this may
    be recursive with nested objects!
  • Validate the data types against the property type and format fields.
    • We can probably use openforms.formio.components to infer type and format
      for a given component type.
    • for user defined variables we can probably maintain a mapping between data type
      and json-schema type/format too
    • array/object variable types are going to be harder
    • static variables can also be annotated with type/format fields
    • this will not be water tight, so make sure create follow up backlog issues for
      the blind spots

• Serializer validation errors must find their way back to the UI in a user-friendly
  way -> highlight the variables tab, highlight the tab where a variable is having errors,
  highlight the relevant registration option in the summary and then display the validation
  error(s) in the modal.

• Validation errors warning about unmapped/missing properties: display them in the
  registration tab with the relevant registration option. Possibly in the future we move
  this to the variables tab, TBD.

Phase 5: backend updates 3, runtime behaviour

• Ensure that the configured mapping is applied and results in a valid object being
  created in the Objects API.
• Make sure to use VCR tests and add a docker-compose config for an objecttypes/objects
  API stack.

Phase ??: harder problems & nice to have

To be fleshed out when the MVP work has been done, so we have a better picture of the
state of things.

• Handle non-variable template constructs like documented on https://open-forms.readthedocs.io/en/2.5.1/manual/templates.html#objecten-api-registratie

• Attachments - generic treatment or specific mappings?

• Notifying of a new(er) version being available for the object type

• Payment status update/PATCH call

• Helpful tools to upgrade the configuration to new version

  • Detect what hasn't changed
  • Detect properties that have been added/removed - there should be an option here to
    indicate a property was "renamed" to make migrating easier

• "Try it out" functionality

TODOs found when working on it

• Would be great to go over styling again at the end (I'm currently doing everything with inline CSS)
• target-paths/ endpoint: handle connection errors to the OT API. Currently returns 500.
• Joeri: [#3688] Refactor Objects API plugin to allow for v2 #3954 (comment)
• Display a message when no target path is available? Currently only displays the ------ empty option, which can be confusing.
• To figure out once we get more feedback: the order in which mappings are applied matters (e.g. you can map an object to path > to, and then a string to path > to > otherpath)

[sergei-maertens]

TODO: aanvullen/speccen

[joeribekker]

Refinement: A whiteboard session will be done to see what the exact tasks should be.

[sergei-maertens]

Attached is an example json-schema definition as provided by the GZAC vendors - this is a good minimal case that must be supported.

noise-complaint-example-schema-v2.json

[sergei-maertens]

Closing this as the main feature is done.

Further follow up is coordinated via:

• Improve objects API documents registration robustness #4041
• (wip) mapping: add possibility to select the same path for multiple registrations #4037
• Objects API "contract" feature - improvements #4031
• Better registration plugins configuration options validation #4024