Using API via matlab_bridge

[martinholub]

Hello,

Despite several trials, my installation of matlab-bridge-api wasn't successful. Main points:

• Python API seems to be running smoothly ("./test_all.sh" and for example f = nwb_file.open(**settings) run)

• The issue appears to be in h5py installation. I have tried both, installing h5py beforehand (via conda from conda-forge and anaconda and via pip, I have also tried multiple versions) and let the nwb setup.py installer do the job. The later fails, whereas the the former passes successfully but results in failure when calling the module from matlab. (I have done setenv('HDF5_DISABLE_VERSION_CHECK', '1'))

Some reproducible error messages:

1. >> nwbModule = py.importlib.import_module('nwb.h5gate')
   Error using _objects>init h5py._objects
   (D:\Build\h5py\h5py-2.7.0\h5py_objects.c:7748) (line 1)
   Python Error: ImportError: DLL load failed: The specified procedure
   could not be found.
   Error in _objects>init h5py.h5r
   (D:\Build\h5py\h5py-2.7.0\h5py\h5r.c:3222) (line 12)
   Error in h5r>init h5py._conv
   (D:\Build\h5py\h5py-2.7.0\h5py_conv.c:7536) (line 21)
   Error in init> (line 34)
   from ._conv import register_converters as _register_converters
   Error in find_links> (line 6)
   Error in h5gate> (line 17)
   Error in init>import_module (line 37)
   import(name)

2. >> run_all_tests
   pyversion is;
   version: '2.7'
   executable: 'C:\Users\Martin\Anaconda3\envs\nwb-edit\python.exe'
   library: 'C:\Users\Martin\Anaconda3\envs\nwb-edit\python27.dll'
   home: 'C:\Users\Martin\Anaconda3\envs\nwb-edit'
   isloaded: 1
   Undefined variable "py" or class "py.nwb.h5gate.File".
   Error in h5g8.file (line 23)
   obj.fg_file = py.nwb.h5gate.File(file_name, kwargs);
   Error in nwb_file (line 52)
   f = h5g8.file(file_name, spec_files, arg_vals.default_ns, options);
   Error in t_annotation/create_annotation_series (line 36)
   f = nwb_file(settings{:});
   Error in t_annotation/test_annotation_series (line 19)
   create_annotation_series(fname, name, 'acquisition/timeseries');
   Error in t_annotation (line 78)
   test_annotation_series()
   Error in run_all_tests (line 37)
   feval(name, verbosity);

3. Previous two cases were with python 2.7, whereas on python 3.6 the matlab crashes with system error:
   Abnormal termination:
   Access violation
   Register State (from fault):
   [...]
   Stack Trace (from fault):
   [ 0] 0x0000000000000000 +00000000
   [ 1] 0x00007ffd51cd52a9 C:\Users\Martin\Anaconda3\lib\site-packages\h5py_conv.cp36-win_amd64.pyd+00021161 PyInit__conv+00006969
   [ 2] 0x00007ffd51cd5219 C:\Users\Martin\Anaconda3\lib\site-packages\h5py_conv.cp36-win_amd64.pyd+00021017 PyInit__conv+00006825
   [ 3-126] .....
   [127] 0x000000000844a4cc C:\Program Files\MATLAB\R2016b\bin\win64\m_parser.dll+00042188 mps_count_formal_outputs+00002196

4. And when trying to install fresh h5py with python setup.py install from the nwb_api-python folder I get:
   c:\users\martin\appdata\local\temp\easy_install-g72ay1\h5py-2.7.0\h5py\api_compa
   t.h(27) : fatal error C1083: Cannot open include file: 'hdf5.h': No such file or directory
   error: Setup script exited with error: command 'C:\Users\Martin\AppData\Loca
   l\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.e
   xe' failed with exit status 2

I will be grateful for tips how to solve the problem. Thank you!

[jeffteeters]

I've reproduced this problem, and unfortunately, don't have a ready solution. I posted an issue on the matworks website:

https://www.mathworks.com/matlabcentral/answers/345709-how-to-call-python-h5py-from-windows-matlab-works-in-os-x-and-linux

And filed a ticket with Mathworks. I'll update this thread if / when I get any information.

BTW, the matlab_bridge works in OSX and Linux, but at least on Ubuntu Linux some configuration is needed first. These are described in the above post.

[jonc125]

I'm facing the same problem. Hopefully Mathworks will sort out it.

Another approach, which I may try next week, could be to build a conda package of h5py & hdf5 using the same HDF5 version as Matlab uses. But it'd need one per Matlab release, so not ideal.

[jonc125]

Matlab is using HDF5 1.8.12 on R2016b through R2017b. I've just successfully tested using my own conda package for this with Python 2.7, and would appreciate feedback as to whether it works for others. Install with

conda install hdf5=1.8.12 h5py=2.7.0 -c jonc

It's only built for Windows at present, either Python 2.7 or 3.5 (though 3.5 is still crashing for me).

[martinholub]

Hello @jonc125 ,
I've tried using your package but obtained the same results as previously, i.e. crash of Matlab. Any idea why it would work for you whereas crash in my case? (I have Matlab 2016b and run python 2.7.)

[jonc125]

Hi @martinholub, thanks for giving it a try. I was working in a completely clean anaconda environment on a virtual machine, so there's plenty of scope for something different about your machine tripping it up! It may be worth you trying just installing hdf5 & h5py in a new conda env, then restarting Matlab and using that env with pyversion before testing. If that works, you can try adding in further packages and see if it starts failing. I've certainly noticed in my testing that it's sometimes the second failure that crashes Matlab, not the first, so have taken to restarting it every time I try something new.

[martinholub]

Hello,

I tested my environment as suggested. h5py and hdf5 work from Matlab when installed from your channel, even after nwb module was installed. Nonetheless, when I execute script abstract_feature.m, Matlab will crash, as previously, at the line f = nwb_file(settings{:});. More specifically, the line py.nwb.nwb_init.nwb_init(f.fg_file, mode, arg_vals.start_time, identifier, description, creating_file) in nwb_file.m causes the problem. Note that if I call f.fg_file.set_dataset(...) from Matlab, it succeeds. Moreover, I tried commenting out whole content of nwb.nwb_init.nwb_init function, which didn't prevent Matlab from crashing with 'Access violation'.

Additionaly, if I run:
f.fg_file.file_changed = 0
py.nwb.nwb_init.nwb_close_callback(f.fg_file)
action proceeds successfully.

Note that at this point I can already successfully execute:
nwbModule = py.importlib.import_module('nwb.h5gate')
or f = py.h5py.File('mytestFile3','w') which is already an improvement compared to the situation when the issue for initially reported.

[jonc125]

Ah, so you're writing NWB from Matlab? I've only tried reading, as that's all I need so far!

What fun 😬

[martinholub]

Hi @jonc125 , yes, I am trying to write to NWB via the API. I don't see many other options how to go from our current custom format to NWB other than this API + bridge. I was considering reading .mat file directly from python but the file structure once loaded someplace else than Matlab becomes terribly messy (Which is annoying as .mat should be de facto hdf5 file)

[jonc125]

Hmm, isn't Windows annoying! I'm afraid I've been away and not had time to look at this further. With the new NWB2 developments, I'm trying to find out what impact those will have before devoting more effort on this, since it seems api-python is being deprecated?

[jeffteeters]

The plan is to depreciate the api-python version (NWB 1.x) and replace it with NWB2.

[jonc125]

Fair enough. I won't do any further work on this at present in that case.

[jonc125]

From discussion on Slack, there will be a pure Matlab API for NWB2, which should avoid these issues entirely. No repository for it as yet, however.