Coping with UPDATE of custom metadata fields with multiple values/entries

[mguidoti]

Hi Alex,

We recently discussed, by email, on how update custom metadata fields and that raised me a couple of question, especially because we're designing this tool to be universally used - not exclusively used by our domain.

Take our universe of custom metadata fields as an example. We have some fields that will always have a single value (most of the DwC based ones) and some other that will have multiple values, like, locations in treatments, or, the OBO ones. For the ones with unique values, the idea of using a relational database (like a spreadsheet to an extant) as input would work perfectly fine. We can take the value on that specific column/row and replace it in the server. But if you start thinking about the custom metadata fields with multiple values, we need to know the value to be changed, not only the new value. Then I have the following questions:

1. how should the user input the data in the incoming spreadsheet if we need the current state and the new desired state?
2. how can we tell Lycophron which fields require the two values, as this will be used universally?
3. are spreadsheets still the best input now that we consider this case?

I've some ideas in mind, but I'll let you start the brainstorm here.

Thanks!

[slint]

1. how should the user input the data in the incoming spreadsheet if we need the current state and the new desired state?

I think the update workflow could go like this:

0. Prerequisites: the user has already described, uploaded, and published a batch of objects to Zenodo via Lycophron (e.g. via filling in a template CSV and importing it). Lycophron keeps the status of these in the local SQLite database.
1. From Lycophron we export a CSV file (edit.csv) with the objects that need editing/updating. This contains the full metadata of the record.
2. We load the edit.csv CSV in Google Sheets
3. The user makes changes to the sheet, aiming to reflect what they want the final state of the metadata to be
4. Once done, the sheet is exported back to CSV format (edit-done.csv) and processed by Lycophron, which then makes the necessary calls to Zenodo in order to update the metadata and publish the changes.

2. how can we tell Lycophron which fields require the two values, as this will be used universally?

So we were discussing having some natural separator in each cell that would split values further... We're kind of free regarding what that separator can be, so even e.g. a newline character could work. See for example this Google sheet:

In cases where we have arrays of objects (e.g. creators) if a value is missing, we just leave an empty line. That would result in the following metadata:

{
    'title': 'Example article',
    'description': 'Example article description',
    'creators': [
        {'name': 'Smith, John', 'orcid': '0000-0002-5082-6404'},
        {'name': 'Doe, Jane', 'affiliation': 'CERN',},
        {'name': 'Agosti, Donat', 'affiliation': 'Plazi', 'orcid': '0000-0001-9286-1200'},
    ],
    'custom': {
        'dwc:kingdom': ['Animalia'],
        'dwc:order': ['Carnivora'],
        'dwc:genus': ['Felidae'],
        'dwc:collectionCode': ['ABCD', 'DEFG', 'HIJK'],
    }
}

3. are spreadsheets still the best input now that we consider this case?

I think spreadsheets have the benefit of being already easy to collaboratively work with (via GDocs), search and filter through, etc, and can be saved as CSV files, which are easily parseable by our application.

An alternative would be to go to a more structured format, like e.g. YAML files or JSONLines, and try to find some sort of online or local application that allows "rich" editing...

Still, this doesn't solve the problem of editing complex values. The only solution I can imagine is developing that kind of rich editor ourselves, which would e.g. bring up a modal when editing a cell that takes multiple/complex values.

[mguidoti]

I'm sure using line breaks won't conform in any possible way with the idea of validating data with frictionlessdata, but I hear your concerns regarding the idea of asking for JSON inputs in the cells of complex fields (such as creators).

Considering that we do want to use frictionlessdata for core validation and extraction of data, what do you think of having an internal, under the hood step that would convert the provided csv with multiple columns for complex fields (creators.name.1, creators.name.2), or with the line breaks, in a ready-to-validate form (JSON), so frictionaless can work smoothly?