Filters applied to variable length data

[DennisHeimbigner]

The discussion started here has been moved to this discussion.

In PR #2231, we explicitly disallowed
defining filters on variable length data.
This was consistent with this comment in the HDF5 code (H5Z.c), which states:

/*-------------------------------------------------------------------------
 * Function: H5Z_ignore_filters
 *
 * Purpose:  Determine whether filters can be ignored.
 *
 * Description:
 *      When the filters are optional (i.e., H5Z_FLAG_OPTIONAL is provided,)
 *      if any of the following conditions is met, the filters will be ignored:
 *          - dataspace is either H5S_NULL or H5S_SCALAR
 *          - datatype is variable-length (string or non-string)
 *      However, if any of these conditions exists and a filter is not
 *      optional, the function will produce an error.
 *
 * Return:   Non-negative(TRUE/FALSE) on success
 *           Negative on failure
 *
 *-------------------------------------------------------------------------
 */

This issue arose because of the addition of filters to NCZarr. We decided to
follow the above comment. However, note that for both netcdf-4(HDF5) and NCZarr,
we currently mark all defined filters as Mandatory (as opposed to Optional).
So as noted in the comment, this causes an error if the type is variable length.

Note that the standard filters in HDF5 -- e.g. deflate and szip -- are marked
as Optional. This means that if one attempts to compress variable-length data
using one of the HDF5 built-in compressors, it will succeed in the sense that the compressor will not be applied to the data. Thus, no data corruption will occur, but neither will the data be compressed.

We have the option of marking all filters as Optional, in which case, we will
suppress them and they will not appear in the meta-data of the created netcdf
dataset. This will occur without any warning to the user.

I need some guidance about what to do. Leave it as is (error) or make it Optional.

This

[sval-dev]

Thanks for organizing this!

My suggestion was make is so there would be a warning to a user by changing libdispatch/dfilter.c#L137 to instead be a fprintf(stderr, "WARNING: Variable length datatypes do not support filters. Ignoring filters.") (or something similar) followed by return ENOERR rather than the current return NC_EFILTER.
I'm happy to put together a PR and update tests to this effect if it helps.

This would allow existing code to continue working with a warning rather than breaking existing code with an error that is a bit cryptic and introducing this as a regression.

If marking the filters optional instead would have the same effect, I think that's reasonable as well, though I think the suggestion above is narrower (i.e. the filters can still be mandatory for fixed length types, but wouldn't be applied for variable-length datatypes where they aren't supported).
I don't know where this mandatory vs optional marking takes place though, so I don't think I'd be able to create a PR (if that was desired) without a bit more information on this.

[DennisHeimbigner]

One we decide on the desired action, the fix is straight forward.
But I feel that it is too easy to overlook a warning, especially in scripts.
So users would be unaware of what is happening.

[sval-dev]

Making users aware by breaking their code, especially for something they can't improve on doesn't seem like the right approach either.

It seems like there might three possible outcomes here.

What happens if a user does not see the warning?
Their code continues to work, but the filters are not applied and compression does not happen.
The user would not know about this. This is the same thing that happened in earlier library versions.

What happens if a user gets an error?
Their code is broken, they now have to figure out why, and they are eventually led to their filter specification.
They remove this filter specification. The filters are therefore not applied and compression does not happen.

What happens if a user does see the warning?
Their code continues to work, and the user is informed.
If user would like to remove the warning, they can remove the filters. (Made easier by the warning being specific to what is happening)
Regardless of what they choose, the filters are not applied and compression does not happen.

If this is being done in scripts, I would say that the script continuing to do what it was doing yesterday is preferable to the breakage, even at the cost of the users not being informed, given that they were already not being informed that this behavior was taking place before, and the underlying behavior (compression not being applied) hasn't changed.

[sval-dev]

I'll also mention that this breaks nccopy for existing files (and likely many other things besides nccopy).

Given the attached (valid) netCDF4 file, nccopy errors out:

$ ./nccopy /tmp/tmp.nc /tmp/tmp1.nc
NetCDF: Filter error: bad id or parameters or duplicate filter
Location: file /tmp/netcdf-c/ncdump/nccopy.c; fcn copy_var_specials line 1152

Using netCDF library version 4.9.2-development of Nov 11 2022 16:38:23 $
tmp.nc.txt(Rename to .nc since .nc upload not allowed)

For users of things like nccopy or other utlities rather than direct users of the library, what are they supposed to do with this?

[czender]

@sval-dev FYI the latest NCO snapshot, eventually to be released as NCO 5.1.4, automatically avoids invoking compression on variable-length types, and thus manages to copy the file above, even with the latest netCDF library. Previously it failed with the same error as nccopy (above). If invoked with a high-enough debugging level (-D 5) it will print and INFO message each time it ignores the compression request for string or vlen variables.

zender@spectral:~/nco$ ncks -O ~/tmp.nc ~/foo.nc
zender@spectral:~/nco$ ncks --hdn -m ~/foo.nc
netcdf foo {
  dimensions:
    Block_Dim = 180 ;

  variables:
    string tmp(Block_Dim) ;
      tmp:_Storage = "contiguous" ;
} // group /
zender@spectral:~/nco$ ncks -O -L 3 ~/tmp.nc ~/foo.nc
zender@spectral:~/nco$ ncks --hdn -m ~/foo.nc
netcdf foo {
  dimensions:
    Block_Dim = 180 ;

  variables:
    string tmp(Block_Dim) ;
      tmp:_Storage = "contiguous" ;
} // group /

[czender]

This is a tough decision. I would prefer if the netCDF layer implements the "Optional" behavior of not calling the filters on variables that do not meet the filter criteria, and does so silently. This includes the Quantize filters. I empathize most with the users who try to compress or copy (as above) data only to be rejected for reasons that may be unclear or confusing to them. I think most such users would prefer that the library compress the variables it can, and not fail on those it can't.

[edwardhartnett]

The normal convention of the netCDF API would be to return an error when you try to do anything that would not work.

So trying to set a filter on a VLEN variable would not work, and the user would get an error if they tried. If this is not the case in the current codebase that is a bug and should be fixed.

However, if there is a significant amount of user code this would break, the rules could be bent. But is there? Most users do not use variable length data types.

Warnings are worthless and should be avoided. Who reads them? Where do they appear on HPC systems, when the code is running on 10000 processors? Just scream your warnings into the void.

There are a few cases where we silently change the settings of a variable, for example if the user sets deflate, we turn on chunking. But I have found those cases always caused confusion. They are documented, but who studies the documentation? So I would avoid any more cases of changing settings silently, such as ignoring the user setting a filter on a var because it will not work.

Let the user decide what they heck they want, and if we can't give it to them, return an error. If they want to compress a VLEN, then return an error and they can decide whether it is worth proceeding without compression, or else changing to a fixed length data structure. (And they might very often choose to do that.)

[sval-dev]

The normal convention of the netCDF API would be to return an error when you try to do anything that would not work.

So trying to set a filter on a VLEN variable would not work, and the user would get an error if they tried. If this is not the case in the current codebase that is a bug and should be fixed.

Just for reference, In previous versions of netCDF, the filters were applied to variable length types and did nothing.

However, if there is a significant amount of user code this would break, the rules could be bent. But is there? Most users do not use variable length data types.

We have code that breaks, and I've already shown above the nccopy code above is broken by this. How many user's code needs to break before we do the right thing for the users who as Charlie mentions just "try to compress or copy (as above) data only to be rejected for reasons that may be unclear or confusing to them".

Warnings are worthless and should be avoided. Who reads them? Where do they appear on HPC systems, when the code is running on 10000 processors? Just scream your warnings into the void.

Not throwing a warning is fine with me. Breaking existing code and uses of existing utilities like nccopy doesn't seem as okay.

There are a few cases where we silently change the settings of a variable, for example if the user sets deflate, we turn on chunking. But I have found those cases always caused confusion. They are documented, but who studies the documentation? So I would avoid any more cases of changing settings silently, such as ignoring the user setting a filter on a var because it will not work.

I'm glad to hear silently changing variable settings has been done before, so at least we'd be going over well-trodden paths.
I disagree that silently dropping the filter is more confusing than the error messages the users would otherwise get if things are left as they are.
Again, the observable behavior, outside of the filters being listed in h5ls (and similar), is identical between silently dropping the filter on newer versions and setting the filter on previous versions and having it not actually apply.

Let the user decide what they heck they want, and if we can't give it to them, return an error. If they want to compress a VLEN, then return an error and they can decide whether it is worth proceeding without compression, or else changing to a fixed length data structure. (And they might very often choose to do that.)

In the nccopy case above, the user don't have a way in which to do what you are suggesting. Should we just tell such users they are out of luck?

As Charlie mentions, I believe the balance should be on the side of not breaking existing user code or the usage of well-known utilities for previously created netCDF data files.

@WardF Any feedback here?

[edwardhartnett]

@sval-dev you make many excellent points. Perhaps at this point it is best to just include this on the list of things we silently do but wish we did not (kind of like paying sales tax or jogging).

[sval-dev]

I ran into this again today:

Code was like the following (from python netCDF4 built on new netCDF4-C library):

product_spec.py", line 154, in _create_datasets
    ds_obj = group_obj.createVariable(ds_name, ds_desc.dtype, dim_names, fill_value=fill_value, zlib=True, complevel=self.complevel, shuffle=True)
  File "src/netCDF4/_netCDF4.pyx", line 2945, in netCDF4._netCDF4.Dataset.createVariable
  File "src/netCDF4/_netCDF4.pyx", line 4184, in netCDF4._netCDF4.Variable.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2014, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: bad id or parameters or duplicate filter
make: *** [Makefile:17: templates] Error 1

Now I can work around by wrapping this in an if statement and not making zlib True when the dtype is a str, but having to fix code that was working isn't all that fun (and it's not even my code) and the "fix" isn't particularly useful.

There is a related issue at Unidata/netcdf4-python#1205 where we can also make the fix (and the maintainer has agreed in principle to accept such a fix), but it would be a lot nicer to fix this here upstream.

Is there any concrete work I can do to move this forward?
I had put forward a proposal for a fix I think I know how to implement (in dfilter.c), but I'm happy to hear anything other suggestions on how to reach a resolution on this.

[sval-dev]

Just thought I'd check back in here. Anything I can do to help close this out?

[DennisHeimbigner]

Let me make sure I understand the proposal. I believe it is this:

If the user attempts to call nc_def_var_filter on a variable length variable, then it returns success, but the filter is suppressed.

So, the questions I have are:

1. Currently, the set of applied filters for a variable is recorded in the meta-data for the variable. Should this include suppressed filters?
2. If netcdf-c encounters attempts to read a variable that has a recorded filter being applied to a variable length type, then should it error out since the data is bogus? Note that this could happen if the data was written using some old ersion of netcdf.

[czender]

My votes would be:

1. No. A filter that was not applied should not be listed in the variable metadata.
2. Yes. I think it's better to error out than silently write likely corrupt data.

[sval-dev]

Just to be clear here, I don't think any part of this proposal is to write likely corrupt data.
The idea is to not apply the filters (silently or with a warning but still proceeding) on variable length data types which don't support them (as was previously done).

As was also shown by my tmp.nc (attached as tmp.nc.txt above), for at least some HDF5 versions, the data read in for a "a variable that has a recorded filter being applied to a variable length type" is very much not bogus but instead is actually the expected data.

I believe @DennisHeimbigner mentions that some previous versions of HDF5 might have written bogus data (rather than the correct data in my example), but that doesn't mean we have to do so here.
If we skip applying the filter in these cases, HDF5 won't do the potentially bad "write compressed pointers".

I'd also be curious about which versions of HDF5 have this bad behavior.
We've used many HDF5 versions, and have this common pattern of compressing all fields.
We haven't run into issues making the round-trip (i.e. we haven't ever seen compressed pointers being written) so it would be surprising if this applied to a wide range of versions at all since it seems like a straightforward regression in HDF5 there (and I argue it's a regression here too).

[sval-dev]

To be clear, the data is not bogus, it's just not compressed. We would never be writing corrupt data, and corrupt data wasn't written by previous versions of netCDF-4, it just failed to compress variable length fields.

The previous behavior was to include the compression filters in the metadata, but applying them to variable length data was a silent no-op.
As I showed above, round-trips already work fine.

The only issue that we are now throwing an error where before we were doing nothing and silently not compressing.

This error breaks previously working code for what seems to be more or less no reason.

Examples include nccopy itself, as shown above.

If you have questions on the previous comments, please do ask them.

[dopplershift]

My $0.02 as a user:

1. If I explicitly ask for zlib and the library can't honor that request it should be an error, not a warning, and definitely not passing silently.
2. nccopy should just "do the right thing", at least if it's working on a whole file or set of variables
3. If netcdf-c previously managed to write the file without error, it needs to read the file now with no error. No exceptions.

[sval-dev]

@dopplershift, I definitely agree with 1 in principle, but if this was previously allowed, and existing code has made use of these semantics, should we now break that existing code that has been working as before (e.g. my code shown above, nccopy).

On 2, it isn't doing the right thing, it's failing because a file written with a previous version of netcdf-4 has metadata on a variable length field that says it should be compressed, but it isn't supported so it errors out.
Similarly, code that was previously written to "unconditionally" compress all fields, as was done above in my python code, and which previously compressed all fields except variable length fields (silently), would also now fail and need to be re-written so that compression is only enabled for those fields which aren't variable length.
What do those developers modifying their existing code get for this trouble now causing errors and code re-writes? Exactly the same behavior they had before, variable length fields still aren't compressed, anything else they wanted to compress still is.
How is this a beneficial situation?

On 3, reading the files works in all cases both before PR #2231 and now.

In case it's not made explicit above, my $0.02 are that as a user I'd like my previously written code to not throw errors and cause me to re-write based on new versions of the library.
If it doesn't cost us much to support backwards compatibility for our users, I'm not sure why we would to knowingly break previously working user code.

[dopplershift]

@dopplershift, I definitely agree with 1 in principle, but if this was previously allowed, and existing code has made use of these semantics, should we now break that existing code that has been working as before (e.g. my code shown above, nccopy).

If it reduces the burden on future users, removes a case where users think the library is doing one thing but it's actually doing another? Absolutely I want that breakage. I should have to update my code because it was BROKEN before. It's a bug fix, and just because it requires changes to user code doesn't make it not a bug fix. I do this all the time for so many other libraries.

On 2, it isn't doing the right thing, it's failing because a file written with a previous version of netcdf-4 has metadata on a variable length field that says it should be compressed, but it isn't supported so it errors out.

My assertion was that nccopy should be made to do the right thing, not that it's behaving correctly now.

On 3, reading the files works in all cases both before PR #2231 and now.

And on (3), I was responding to the question from @DennisHeimbigner: "If netcdf-c encounters attempts to read a variable that has a recorded filter being applied to a variable length type, then should it error out since the data is bogus?" My assertion is that if it was able to write it, it should never error out, even if the data is "bogus".

Libraries should be strict on write, permissive on read.

[sval-dev]

I'm not sure I agree it was broken before. The HDF5 and netCDF-4 libraries don't support compression of variable length fields, but they permitted users to attempt to compress them and did nothing. Perhaps at some future date they might actually support this and the code will achieve better compression.
From the point of view of a developer, their code will do exactly the same thing both before with earlier versions of netCDF and now after they "fix" it for newer versions of netCDF4.
How was this an improvement for our users?
What do we or they get for this idealistic purity?
Should developers really be forced to know about the difference (only now coming to light) between fixed length fields and variable length fields not supporting the same filters?
As @edwardhartnett says above, I think we should "include this on the list of things we silently do but wish we did not (kind of like paying sales tax or jogging)".

Beyond developers, non-developers (e.g. scientists) make use of code that is delivered to them by developers.
These people may receive code (e.g. the NOAA repo that I linked to, internal PGE code for NASA projects), that was previously working fine, but on updates to NetCDF4 they can no longer compile this delivered code and it begins to rot.
Again, where is the tangible benefit for this real non-theoretical cost to these non-developers who can't fix the code anyway and who would get the same behavior with "fixed" code?

On nccopy, what is the right thing here given the command I gave above?
Is throwing an error and making the file unusable the right thing as it's doing now?
I think you are agreeing it is not.
Okay, so maybe it should just drop the filter and still allow you to work with these "valid" netCDF4 files (at least they were "valid" at the time they were created).
Well, it turns out, this is the approach taken by NCO and exactly what I'm proposing we do upstream of nccopy, so nccopy doesn't need code modifications, and neither does the NOAA code, and neither would have the NCO code, and neither does any of the other code that is currently working and which will derive no benefit from getting modified to the new paradigm.
Just on the above uses and my own time, that's quite a lot of human and financial capital invested for this change, and this code isn't even in wide usage yet.

On 3, I agree we should strive to read as much as possible.
I disagree that we should be "strict" on writers (including normal users just trying to compile existing code) such that previously working code is broken without a tangible benefit.
If existing code usages lead to some horrible breakage, or incorrect results, or the requested code change in some way improves something in exchange perhaps there is a trade off to discuss.
What would that tradeoff be here?

[DennisHeimbigner]

To be clear, the data is not bogus, it's just not compressed. We would never be writing corrupt data, and corrupt data wasn't written by previous versions of netCDF-4, it just failed to compress variable length fields.

This is not correct. The old netcdf-c code was compressing the set of pointers to the variable length data.
So it is bogus.

[sval-dev]

It was not doing that, and the round trip was shown to work perfectly fine.
I can show that again here if it's relevant and would change any opinions.

[sval-dev]

I have actually shown that above in tmp.nc (attached at tmp.nc.txt).
Here, the correct output (x repeated a number of times) is shown for the "tmp" field:

h5ls -dv tmp.nc/tmp
Opened "tmp.nc" with sec2 driver.
tmp                      Dataset {180/180}
    Attribute: DIMENSION_LIST {1}
        Type:      variable length of
                   object reference
        Data:
               (DATASET-1:331 "")
    Location:  1:685
    Links:     1
    Chunks:    {180} 1440 bytes
    Storage:   1440 logical bytes, 264 allocated bytes, 545.45% utilization
    Filter-0:  shuffle-2 OPT {16}
    Filter-1:  deflate-1 OPT {5}
    Type:      variable-length null-terminated UTF-8 string
    Data:
         "", "x", "xx", "xxx", "xxxx", "xxxxx", "xxxxxx", "xxxxxxx", "xxxxxxxx",
         'x' repeats 8 times, 'x' repeats 9 times, 'x' repeats 10 times,

It says it has deflate applied to this variable length field, but it's application both on reading and writing is silently dropped.

[DennisHeimbigner]

OK, Please publish here the exact proposal you are making. Indicate what the behavior should be in the cases
I have identified.

[sval-dev]

On whether the metadata should be written or not, the old behavior is to write the compression filter metadata though it was silently not applied.
It would be more "correct", to write appropriate applicable filter metadata to the file that reflects that actual filters that were applied and would therefore leave out the compression filters that were silently dropped.

I would propose we adopt that behavior, though it might break existing code which checks for filters application in for example unit tests.
Personally, that seems strictly better than breaking all client code that writes these fields.
If we'd like to avoid that potential breakage of unit tests or other expectations of continuing filter metadata as well, we can preserve the old approach.

On the second point, the code should continue to do reading correctly as it has always done.

Concretely, my proposal, given that warning appear to be frowned upon, is to continue silently dropping the filter application for variable length fields on write as was done before.

Over the longer term, I think the hope would be to get compression working for variable length fields, in which case future versions of the code can actually do the thing the metadata says they are doing.

I'm open to revisions, but I would like to avoid breaking client code as much as possible especially given that some people are simple users of data products (e.g. in the nccopy case above) or receivers of code who might just 'pip install netCDF4' and get a version that breaks their code.

Regardless of whether this proposal is adopted, some work needs to be done either way.
The work will either be to my code, the code in nccopy, and all of the other client codes that might potentially make these now invalid writes, or it can be as proposed here to this single place.

[sval-dev]

At the most basic level, I'm proposing we just remove the line at

netcdf-c/libdispatch/dfilter.c

Line 137 in 43abd69

if(!fixedsize) return NC_EFILTER;

(and the line above and the declaration of fixed_size since they would become unused).
If we want to leave out the filter metadata as well, we can instead modify line 137 such that if (!fixed-size) return stat;.

[DennisHeimbigner]

See the discussion for #2189.
Old HDF5 behavior applied the filter. They changed it, but any HDF5 library before
that change would have compressed the pointers.

[sval-dev]

Thanks for the link! I had read that, but I disagree with the conclusion.
As you mentioned, I think the question was "whether we want to remove the filter definition when the combination doesn't work and silently proceed, or whether we want to fail with an informative message".

I believe making it an error and causing client code to need modification (as we already see NOAA code doing, and I had to do, on NASA work) is a waste of resources and gets you exactly the same externally observable behavior (i.e. that variable length fields are not compressed).

[sval-dev]

To be clear on the NOAA work, I'm referring to NOAA-CSL/MELODIES-MONET#162

What did they do? They added an extra if to gate whether to enable compression where before it was always applied. (https://github.com/zmoon/MELODIES-MONET/blob/7f5f1212e82c09fd9e6a912600bbcd83e3b37956/melodies_monet/util/write_util.py#L49)

For that change, which took dev time to both understand the new error and code a fix, packaging and everything that might accompany that like doc updates and new pipeline runs, they got exactly the same externally observable behavior.

[DennisHeimbigner]

Lets finish this. I talked with Ward. The proposal on the table is this:

1. Assume we have a variable V whose type is variable length
2. When creating a file, attempts to define a filter on V (using nc_def_var_filter) will return NC_NOERR, but the filter will be ignored and logged.
3. When reading a file, if V has one or more associated filters, then they will be ignored and logged.

Anyone have violent objection?

[sval-dev]

Thanks for the discussion! This sounds great to me!

I'm happy to help with a PR if it's useful.

Would (2) be modifying

netcdf-c/libdispatch/dfilter.c

Line 137 in b30b4e8

if(!fixedsize) return NC_EFILTER;

to return NC_NOERR instead of NC_EFILTER and adding a log line?

For (3), would that be modifying nc_inq_var_filter() so that if it isn't fixed size it logs a message (near

netcdf-c/libdispatch/dfilter.c

Line 188 in 43abd69

/* Get the filter ids */

) and sets *idp = 0 as if no filters were defined?

[sval-dev]

Just thought I'd check back in on this.

Anything I can do to help move this forward?

[DennisHeimbigner]

See #2716

[sval-dev]

Thanks for the link to the PR!

I think the eventual implementation differed a bit from what was proposed above (or my understanding of it), in that I understood that if a previous file has a fixed length variable defined with filters on it, the filters would be ignored, not the variable itself.

This made some sense since we'd shown above that it can make the round-trip (at least in some cases), but I understand that there was some concern that we might be reading in garbage (though I don't think we ever found a combo of library versions that would actually corrupt the data).

The change as-is at least prevents breakage with existing code that was previously working to write out files, but it does also mean that existing files which have successfully been written with filters for variable length fields (such as the tmp.nc above) are now not fully supported (e.g. nccopy command above would fail to copy the tmp variable but wouldn't error out).

If it's true that we might be reading in corrupted data, I agree this is preferable to returning garbage (though as shown above, the garbage assertion isn't true for all versions of the libraries and hasn't been conclusively shown here).

I appreciate the work and hope the community appreciates the refactoring savings they might benefit from as well!

[DennisHeimbigner]

Over the longer term, I think the hope would be to get compression working for variable length fields, in which case future versions of the code can actually do the thing the metadata says they are doing.
This requires changes in HDF5, which not under our control. For new user-defined filters, it is possible to handle variable length data, though it is potentially complex. But all the existing filters defined on the HDF5 custom filters page would need to be rewritten and I have not idea who is going to carry out that herculean task.

[DennisHeimbigner]

So I went searching in the HDF5 code (H5Z.c specifically).
There is the following comment:

/*-------------------------------------------------------------------------
 * Function: H5Z_ignore_filters
 *
 * Purpose:  Determine whether filters can be ignored.
 *
 * Description:
 *      When the filters are optional (i.e., H5Z_FLAG_OPTIONAL is provided,)
 *      if any of the following conditions is met, the filters will be ignored:
 *          - dataspace is either H5S_NULL or H5S_SCALAR
 *          - datatype is variable-length (string or non-string)
 *      However, if any of these conditions exists and a filter is not
 *      optional, the function will produce an error.
 *
 * Return:   Non-negative(TRUE/FALSE) on success
 *           Negative on failure
 *
 *-------------------------------------------------------------------------
 */

[DennisHeimbigner]

So, we can implement that exact same behavior by making all nc_def_var_filter calls create optional filters.

[sval-dev]

I like this idea!

[DennisHeimbigner]

But note that behavior (#2554 (comment)) is equivalent to the PR I put up originally.

So we have two choices:

1. include the filter in the filter chain and suppress when invoked
2. suppress the filter at the point when nc_def_var_filter is called.

The remaining problem is when we are given a dataset that has a filter on a var-len variable and it is not optional.
HDF5 will produce an error while what we have implemented is to just suppress the variable to avoid the error.

[sval-dev]

Looks like this might just be a case of setting the 1 at line

netcdf-c/libhdf5/nc4hdf.c

Line 917 in 2cd9d26

#if 1

to a 0 (so that we take the #else).
That would essentially revert a change made in 3ffe7be

On the existing dataset with mandatory filters, are we still expecting these might be corrupted reads?
If so, then suppressing the variable seems better than returning potentially corrupt data or an error.

As a particular example for the optional case, the tmp.nc above has the filter defined as OPTIONAL:

>>> f["tmp"].id.get_create_plist().get_filter(1)
(1, 1, (5,), b'deflate')

Where the second item in the tuple (i.e. 1) is the uint flags return, and where H5Zpublic.h defines H5Z_FLAG_OPTIONAL == 0x0001

That means the sample tmp.nc above could be used as a test case to make sure things like nccopy don't suppress access to this variable or cause errors as nccopy was doing before the newest PR was merged.

[DennisHeimbigner]

So the proposal is as follows:

• If the variable is variable length, then look at all the associated filters and make sure they are optional.
• If they are all optional, then go ahead an let the variable be read or written.
• If one or more filters is not optional, then suppress the variable because it is unreadable without generating an error.

Correct?

[sval-dev]

That sounds good to me!
Pairing this with "nc_def_var_filter calls create optional filters" ensures that even when code like the above referenced NOAA code or other code that tries to compress everything by default runs, then bullet one in the proposal will be fulfilled such that no errors are generated on reads or writes.

[DennisHeimbigner]

One additional comment. If a vlen variable has a non-optional filter, then it would be desirable to see the metadata
for the variable. But unfortunately, that would cause e.g. nccopy to try to read the variable and that would have
to produce an error. So we really have to make such variables completely invisible.