Fix random "only endian='native' allowed for NETCDF3 files" error

[msakai]

When I open NetCDF3 files using netCDF4.Dataset(filename, "r"), I sometimes get the following errors and sometimes not.

  File "src/netCDF4/_netCDF4.pyx", line 2489, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2092, in netCDF4._netCDF4._get_vars
  File "src/netCDF4/_netCDF4.pyx", line 4062, in netCDF4._netCDF4.Variable.__init__
RuntimeError: only endian='native' allowed for NETCDF3 files, got 'big' (variable 'FOO', group '/')

It seems that nc_inq_var_endian() returns NC_ENOTNC4 and iendian is left uninitialized in that case.

[CLAassistant]

All committers have signed the CLA.

[jswhit]

could a test for this be added to tests/test_endian.py?

[jswhit]

@msakai if you can provide me sample code for generating a dataset that triggers this error, I can add a test.

[msakai]

@jswhit I'm sorry for not getting back to you sooner.

The file I have encountered the problem contains non-public data and therefore cannot be sent, but is a normal file created with netcdf-c v4.9.2 on macOS 13.6.6 (arm64). I would like to create a file that can be sent.

But from what I understand, the problem is not in the file itself, but rather in how the C stack was used before reaching this code position. The problem happened in the real code but was not reproduced when the same file was simply read in the test code. Thus it may be difficult to catch the problem in the test.

[msakai]

I reduced the dataset that causes the problem with my script to test.nc in test.nc.zip. The file is generated by:

from netCDF4 import Dataset
dataset = Dataset("test.nc", "w", format="NETCDF3_CLASSIC")
dataset.createDimension("A", 1)
dataset.createDimension("B", 52)
dataset.createVariable("X", "f4")

But I'm still struggling with making the script itself smaller.

[msakai]

Also, I can reproduce this problem if I apply the following patch

diff --git a/pr1355.c b/pr1355.c
new file mode 100644
index 00000000..a0e6515a
--- /dev/null
+++ b/pr1355.c
@@ -0,0 +1,10 @@
+#include <stdio.h>
+
+extern void pr1355_prepare_stack() {
+    volatile int a[4096];
+    for (int i = 0; i < 4096; i++) {
+        a[i] = 2;
+    }
+    printf("%d\n", a[0]);
+    // volatile and printf() are to prevent C compiler from optimizing away the array.
+}
diff --git a/setup.py b/setup.py
index c23eb034..3835f57c 100644
--- a/setup.py
+++ b/setup.py
@@ -420,6 +420,7 @@ if 'sdist' not in sys.argv[1:] and 'clean' not in sys.argv[1:] and '--version' n
     source_files = [
         netcdf4_src_pyx,
         str(nc_complex_dir / "src/nc_complex.c"),
+        "pr1355.c",
     ]
     include_dirs = inc_dirs + [
         "include",
diff --git a/src/netCDF4/_netCDF4.pyx b/src/netCDF4/_netCDF4.pyx
index 6023406c..06ab1335 100644
--- a/src/netCDF4/_netCDF4.pyx
+++ b/src/netCDF4/_netCDF4.pyx
@@ -2206,6 +2206,9 @@ cdef _inq_vardimid(int ncid, int varid, bint auto_complex):
     return result
 
 
+cdef extern void pr1355_prepare_stack()
+
+
 # these are class attributes that
 # only exist at the python level (not in the netCDF file).
 
@@ -2536,6 +2539,7 @@ strings.
         # get dimensions in the root group.
         self.dimensions = _get_dims(self)
         # get variables in the root Group.
+        pr1355_prepare_stack()
         self.variables = _get_vars(self, self.auto_complex)
         # get groups in the root Group.
         if self.data_model == 'NETCDF4':

and then running the following code

import netCDF4
netCDF4.Dataset("test.nc", "r")

Even though pr1355_prepare_stack() is a harmless function call, it changes the behavior of the subsequent _get_vars call.

I have been able to reproduce the problem in the following environments using this method:

• Python 3.11.3 on macOS 13.6.6 (ARM64)
• Python 3.10.12 on Linux (x86_64)