[bug] "Large" Datasets can't be queried

[TeoZosa]

Note

I saw #2470 + a few other PRs and it looks like @tssweeney & co are already working on resolving this, but adding the issue as a +1 vote to that ongoing work.

This seems to be due to data size; creating Datasets using toy data seems to work fine even at larger cardinalities.

Steps to reproduce

With Datasets created by duplicating a single example with some pretty verbose string fields and base64 image data, Datasets greater than 33 examples fail to query.

>>> import weave
>>> weave.init(project_name="my-project")
>>> real_dataset = weave.ref("my-real-dataset").get()
>>> dataset = weave.Dataset(name="dataset_size_33", rows = [real_dataset.rows[0]]*33)
>>> weave.publish(dataset)
📦 Published to ...
ObjectRef(entity='...', project='...', name='dataset_size_33', digest='...', extra=())
>>> dataset = weave.Dataset(name="dataset_size_34", rows = [real_dataset.rows[0]]*34)
>>> weave.publish(dataset)
📦 Published to ...
ObjectRef(entity='...', project='...', name='dataset_size_34', digest='...', extra=())

Examples

Real data

>>> import weave
>>> weave.init(project_name="my-project")
>>> len(weave.ref('dataset_size_33').get().rows)
33
>>> len(weave.ref('dataset_size_34').get().rows)
Traceback (most recent call last):
...
    raise requests.HTTPError(
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: /table/query. Reason: 

Click here for screenshots

Toy data

Confirmed working even at 100 examples

>>> import weave
>>> weave.init(project_name="my-project")
>>> len(weave.ref('dataset_size_100:v0').get().rows)
100

Click here for screenshots

[jamie-rasmussen]

Tracking internally as https://wandb.atlassian.net/browse/WB-21344

[tssweeney]

Thanks @TeoZosa - you are indeed correct that we are investing in this path - next up is to convert our response to a streaming response which will avoid this bug. Will keep you posted on priority.