[WIP] Add changes to facilitate lookup of SpikeEventSeries from Units table #819

bendichter · 2019-02-17T02:37:52Z

add link from SpikeEventSeries to UnitSeries
add DEPRECATED to doc for ClusterWaveforms
add link from ElectrodeGroup to SpikeEventSeries
add UnitSeries object to misc, which includes optional link to Units table
Remove DecompositionSeries from LFP (It never worked anyway)
Allow for 3D data in ElectricalSeries to handle SpikeEventSeries
add get_spike_waveforms as method of Units table

Motivation

Now we can pull waveforms for specific spike events in the units table.

How to test the behavior?

import numpy as np

import pynwb
from datetime import datetime
from dateutil import tz

from pynwb.ecephys import SpikeEventSeries
from pynwb.misc import UnitSeries

dataset_zero_time = datetime(2006, 1, 2, 12, 0, 0, tzinfo=tz.gettz('US/Pacific'))
file_create_date = datetime.now(tz.tzlocal())
data_dir = 'path/to/nwb/files'
nwb_filename = 'test.nwb'
nwb = pynwb.NWBFile(session_description='no session description',
                    identifier='no identifier',
                    session_start_time=dataset_zero_time)

ecephys_mod = nwb.create_processing_module('ecephys', 'stores ecephys data')

unit_series = UnitSeries('UnitSeries0', data=np.array([0, 0, 1, 0, 1], dtype=int),
                         timestamps=np.linspace(10, 50, 5))

implant = nwb.create_device('implant')
shank0 = nwb.create_electrode_group('shank0', 'shank 0 (0-indexed)', location='unknown', device=implant)

for i in range(5):
    nwb.add_electrode(np.nan, np.nan, np.nan, np.nan, 'unknown', 'unknown', shank0)

electrode_table_region = nwb.create_electrode_table_region(np.arange(5), 'shank0 electrodes')

spike_event_series0 = SpikeEventSeries('SpikeEventSeries0', np.arange(5*5*30).reshape(5, 5, 30),
                                       np.linspace(10, 50, 5), unit_series=unit_series,
                                       electrodes=electrode_table_region)

ecephys_mod.add_data_interface(spike_event_series0)
shank0.spike_event_series = spike_event_series0


nwb.add_unit(spike_times=[1., 2., 3.], electrode_group=shank0)
nwb.add_unit(spike_times=[1.5, 2.5], electrode_group=shank0)

###

def get_spike_waveforms(units, row, spike_number=None):
    ses = units['electrode_group'].data[row].spike_event_series
    if spike_number is None:
        return ses.data[ses.unit_series.data == row, ...]
    else:
        inds = np.where(ses.unit_series.data == row)[0]
        return ses.data[inds[spike_number], ...]

Checklist

Have you checked our Contributing document?
Have you ensured the PR description clearly describes problem and the solution?
Is your contribution compliant with our coding style ? This can be checked running flake8 from the source directory.
Have you checked to ensure that there aren't other open Pull Requests for the same change?
Have you included the relevant issue number using #XXX notation where XXX is the issue number ?

* add link from SpikeEventSeries to UnitSeries * add DEPRECATED to doc for ClusterWaveforms * add link from ElectrodeGroup to SpikeEventSeries * add UnitSeries object to misc, which includes optional link to Units table * Remove DecompositionSeries from LFP (It never worked anyway) * Allow for 3D data in ElectricalSeries to handle SpikeEventSeries * add get_spike_waveforms as method of Units table

bendichter · 2019-02-17T04:33:24Z

fix NeurodataWithoutBorders/nwb-schema#248

bendichter · 2019-02-17T04:34:48Z

@ajtritt can you help? I am getting this error and I don't know what's going wrong

import numpy as np

import pynwb
from datetime import datetime
from dateutil import tz

from pynwb.ecephys import SpikeEventSeries
from pynwb.misc import UnitSeries

dataset_zero_time = datetime(2006, 1, 2, 12, 0, 0, tzinfo=tz.gettz('US/Pacific'))
file_create_date = datetime.now(tz.tzlocal())
data_dir = 'path/to/nwb/files'
nwb_filename = 'test.nwb'
nwb = pynwb.NWBFile(session_description='no session description',
                    identifier='no identifier',
                    session_start_time=dataset_zero_time)


unit_series = UnitSeries('UnitSeries0', data=np.array([0, 0, 1, 0, 1], dtype=int),
                         timestamps=np.linspace(10, 50, 5))

implant = nwb.create_device('implant')
shank0 = nwb.create_electrode_group('shank0', 'shank 0 (0-indexed)', location='unknown', device=implant)

for i in range(5):
    nwb.add_electrode(np.nan, np.nan, np.nan, np.nan, 'unknown', 'unknown', shank0)

electrode_table_region = nwb.create_electrode_table_region(np.arange(5), 'shank0 electrodes')

spike_event_series0 = SpikeEventSeries('SpikeEventSeries0', np.arange(5*5*30).reshape(5, 5, 30),
                                       np.linspace(10, 50, 5), unit_series=unit_series,
                                       electrodes=electrode_table_region)
ecephys_mod = nwb.create_processing_module('ecephys', 'stores ecephys data')

ecephys_mod.add_data_interface(spike_event_series0)
shank0.spike_event_series = spike_event_series0


nwb.add_unit(spike_times=[1., 2., 3.], electrode_group=shank0)
nwb.add_unit(spike_times=[1.5, 2.5], electrode_group=shank0)


with pynwb.NWBHDF5IO('test_ses.nwb', 'w') as io:
    io.write(nwb)

Traceback (most recent call last):
  File "/Users/bendichter/dev/pynwb/test_spike_event_times.py", line 44, in <module>
    io.write(nwb)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 195, in write
    call_docval_func(super(HDF5IO, self).write, kwargs)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 274, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/backends/io.py", line 39, in write
    f_builder = self.__manager.build(container, source=self.__source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 159, in build
    result = self.__type_map.build(container, self, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 1536, in build
    builder = attr_map.build(container, manager, builder=builder, source=getargs('source', kwargs))
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 751, in build
    self.__add_groups(builder, self.__spec.groups, container, manager, source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 956, in __add_groups
    self.__add_groups(sub_builder, spec.groups, container, build_manager, source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 972, in __add_groups
    self.__add_containers(builder, spec, attr_value, build_manager, source, container)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 1015, in __add_containers
    self.__add_containers(builder, spec, container, build_manager, source, parent_container)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 982, in __add_containers
    rendered_obj = build_manager.build(value, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 159, in build
    result = self.__type_map.build(container, self, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 1536, in build
    builder = attr_map.build(container, manager, builder=builder, source=getargs('source', kwargs))
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 751, in build
    self.__add_groups(builder, self.__spec.groups, container, manager, source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 972, in __add_groups
    self.__add_containers(builder, spec, attr_value, build_manager, source, container)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 1015, in __add_containers
    self.__add_containers(builder, spec, container, build_manager, source, parent_container)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 982, in __add_containers
    rendered_obj = build_manager.build(value, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 159, in build
    result = self.__type_map.build(container, self, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 1536, in build
    builder = attr_map.build(container, manager, builder=builder, source=getargs('source', kwargs))
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 750, in build
    self.__add_datasets(builder, self.__spec.datasets, container, manager, source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 931, in __add_datasets
    self.__add_containers(builder, spec, attr_value, build_manager, source, container)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 982, in __add_containers
    rendered_obj = build_manager.build(value, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 159, in build
    result = self.__type_map.build(container, self, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 1536, in build
    builder = attr_map.build(container, manager, builder=builder, source=getargs('source', kwargs))
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 794, in build
    self.__add_attributes(builder, self.__spec.attributes, container, manager, source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 866, in __add_attributes
    target_builder = build_manager.build(attr_value, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 159, in build
    result = self.__type_map.build(container, self, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 1536, in build
    builder = attr_map.build(container, manager, builder=builder, source=getargs('source', kwargs))
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 750, in build
    self.__add_datasets(builder, self.__spec.datasets, container, manager, source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 931, in __add_datasets
    self.__add_containers(builder, spec, attr_value, build_manager, source, container)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 1015, in __add_containers
    self.__add_containers(builder, spec, container, build_manager, source, parent_container)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 982, in __add_containers
    rendered_obj = build_manager.build(value, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 159, in build
    result = self.__type_map.build(container, self, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 1536, in build
    builder = attr_map.build(container, manager, builder=builder, source=getargs('source', kwargs))
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 784, in build
    bldr_data.append(ReferenceBuilder(manager.build(d)))
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 159, in build
    result = self.__type_map.build(container, self, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 1536, in build
    builder = attr_map.build(container, manager, builder=builder, source=getargs('source', kwargs))
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 752, in build
    self.__add_links(builder, self.__spec.links, container, manager, source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 891, in __add_links
    self.__add_containers(builder, spec, attr_value, build_manager, source, container)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 982, in __add_containers
    rendered_obj = build_manager.build(value, source=source)
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/dev/pynwb/src/pynwb/form/build/map.py", line 158, in build
    raise ValueError("Can't change container_source once set")
ValueError: Can't change container_source once set

bendichter · 2019-02-18T21:36:49Z

I'm breaking this into smaller commits

ajtritt · 2019-02-19T17:30:38Z

src/pynwb/ecephys.py

@@ -19,20 +18,25 @@ class ElectrodeGroup(NWBContainer):
    __nwbfields__ = ('name',
                     'description',
                     'location',
-                     'device')
+                     'device',
+                     {'name': 'spike_event_series', 'child': False, 'doc': 'doc'})


This should be an ElectricalSeries, and doc should be something that makes sense. The docstring will get rendered on the API docs, so this is what users will see if they use the API docs for reference.

ajtritt · 2019-02-19T17:33:24Z

src/pynwb/ecephys.py

@@ -107,7 +113,7 @@ class SpikeEventSeries(ElectricalSeries):
    electrode).
    """

-    __nwbfields__ = ()
+    __nwbfields__ = ({'name': 'unit_series', 'child': False, 'doc': 'doc'}, )


doc needs to be something meaningful... see above for explanation.

yarikoptic · 2023-02-24T19:19:10Z

is it really still WiP @bendichter or might want to be closed (or converted to draft at least)?

CodyCBakerPhD · 2023-02-24T22:38:26Z

The approach we were moving towards (and I believe is even currently possible though not ideal) is to add the waveforms for each spike of a unit as a doubly indexed (index=2 syntax) column to the Units table, which facilitates that lookup

bendichter changed the title ~~Add changes to facilitate lookup of SpikeEventSeries from Units table~~ [WIP] Add changes to facilitate lookup of SpikeEventSeries from Units table Feb 17, 2019

add spike_event_series to __nwbfields__ of ElectrodeGroup

460d7b0

bendichter requested a review from ajtritt February 17, 2019 04:33

bendichter removed the request for review from ajtritt February 18, 2019 21:59

ajtritt requested changes Feb 19, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add changes to facilitate lookup of SpikeEventSeries from Units table #819

[WIP] Add changes to facilitate lookup of SpikeEventSeries from Units table #819

bendichter commented Feb 17, 2019

bendichter commented Feb 17, 2019

bendichter commented Feb 17, 2019

bendichter commented Feb 18, 2019

ajtritt Feb 19, 2019

ajtritt Feb 19, 2019

yarikoptic commented Feb 24, 2023

CodyCBakerPhD commented Feb 24, 2023

[WIP] Add changes to facilitate lookup of SpikeEventSeries from Units table #819

Are you sure you want to change the base?

[WIP] Add changes to facilitate lookup of SpikeEventSeries from Units table #819

Conversation

bendichter commented Feb 17, 2019

Motivation

How to test the behavior?

Checklist

bendichter commented Feb 17, 2019

bendichter commented Feb 17, 2019

bendichter commented Feb 18, 2019

ajtritt Feb 19, 2019

Choose a reason for hiding this comment

ajtritt Feb 19, 2019

Choose a reason for hiding this comment

yarikoptic commented Feb 24, 2023

CodyCBakerPhD commented Feb 24, 2023