Fix Brotli compression #144

wilwade · 2024-09-10T15:22:01Z

Note: The reference Brotli file doesn't work because it is too large. It decompresses to an extremely large size (1Gb+)

Closes #140
Closes #125 (Likely?)

A new brotli sample file was generated using python code seen below.

What Changed?

Moved the esbuild work to mjs. This was required to get some of the builds working correctly
Fixed bug in the esbuild when it tried to build the browser code in parallel with the test code causing a race condition
Split the compression.ts file into a browser version and a node version
Swapped over to use esbuild-plugin-wat which worked better than the copy-pasted one from esbuild
Integrated the brotli-wasm correctly for browser, but used brotli natively in nodejs

Testing!

There is a nodejs test for the node version
There is a browser test for it as well:
1. npm i
2. npm run build:browser
3. npx serve .
4. open http://localhost:3000/test/browser/ in your preferred browser
The example server also has it:
- cd examples/server/
- npm i
- node app.js
- cd ../../ && npm run serve
- open http://localhost:3000/ in your preferred browser

Brotli Sample File generation script

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

# Create a small sample DataFrame
data = {
    'id': [1, 2, 3, 4, 5],
    'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'age': [25, 30, 35, 40, 45]
}

df = pd.DataFrame(data)

# Convert DataFrame to PyArrow Table
table = pa.Table.from_pandas(df)

# Define output Parquet file path
output_file = "sample_brotli_compressed.parquet"

# Write to Parquet file with Brotli compression
pq.write_table(table, output_file, compression='BROTLI')

print(f"File {output_file} created successfully!")
``

# Conflicts: # package-lock.json # package.json

wilwade · 2024-09-10T15:22:48Z

esbuild-plugins.mjs

mjs version swap and removing now unused wasm plugin

wilwade · 2024-09-10T15:23:43Z

esbuild-serve.mjs

esbuild-plugin-wat does what we did, but better!

wilwade · 2024-09-10T15:23:56Z

esbuild-shims.mjs

mjs conversion

wilwade · 2024-09-10T15:24:40Z

esbuild.mjs

+    }
+  })
+  .then(() => {
+    return Promise.all(testTargets.map(esbuild.build));


This has to be after the other targets are built as it is dependent on it.

wilwade · 2024-09-10T15:25:01Z

examples/server/views/parquetFiles.ejs

Visible output!

wilwade · 2024-09-10T15:25:34Z

lib/browser/brotli.ts

This code is mostly copied from https://github.com/httptoolkit/brotli-wasm?tab=readme-ov-file#in-browser-with-streams

wilwade · 2024-09-10T15:26:50Z

lib/browser/compression.js

Replaced with the typescript version!

wilwade · 2024-09-10T15:28:39Z

test/reference-test/read-all.test.ts

@@ -27,7 +27,7 @@ export const unsupported = [
  'delta_encoding_optional_column.parquet', // DELTA_BINARY_PACKED unsupported
  'delta_encoding_required_column.parquet', // DELTA_BINARY_PACKED unsupported
  'delta_length_byte_array.parquet', // ZSTD unsupported, DELTA_BINARY_PACKED unsupported
-  'large_string_map.brotli.parquet', // BUG?
+  'large_string_map.brotli.parquet', // Fails as the large string is > 1 GB


This took a while to figure out. It is a small file, but the large string test is very large. Too much for js for the most part.

wilwade · 2024-09-10T15:28:58Z

test/test-files.js

@@ -216,6 +216,15 @@ describe('test-files', function () {
    }
  });

+  describe('Brotli compression', function () {


esbuild.mjs

wilwade · 2024-09-10T21:07:28Z

esbuild-serve.mjs

    sourcemap: 'external',
    bundle: true,
+    minify: false,


Keeping minify off here for the local serve.

shannonwells

I ran the examples and reviewed the code. Looks good to go, glad you found it was mainly a poor example file.

wilwade added 12 commits September 9, 2024 09:20

For node, use zlib for brotli

6303cc9

Adding an html output to the example

4080b3f

Add additional examples

ae55300

Updating compression for node and web

252a403

# Conflicts: # package-lock.json # package.json

Add additional browser test and docs on how to run it in a browser

d90bdcc

Attempting to add Brotli to browser

6e755dd

Browser version of compress.ts

699f879

Fix esbuild files to use modules

c95b8bc

Add note to failing reference test

380fd7c

Brotli working in the browser and node

fee18a2

Add example file to example server

6c5d024

Fix serve issue

870e78d

wilwade commented Sep 10, 2024

View reviewed changes

esbuild-plugins.mjs Outdated

Copy link

Member Author

wilwade Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mjs version swap and removing now unused wasm plugin

Docs fix

8b0ba53

wilwade commented Sep 10, 2024

View reviewed changes

esbuild-serve.mjs Outdated

Copy link

Member Author

wilwade Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

esbuild-plugin-wat does what we did, but better!

wilwade commented Sep 10, 2024

View reviewed changes

esbuild-shims.mjs Outdated

Copy link

Member Author

wilwade Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mjs conversion

wilwade commented Sep 10, 2024

View reviewed changes

examples/server/views/parquetFiles.ejs Outdated

Copy link

Member Author

wilwade Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Visible output!

wilwade commented Sep 10, 2024

View reviewed changes

Remove unneeded logs

1a8026c

wilwade commented Sep 10, 2024

View reviewed changes

lib/browser/compression.js Outdated

Copy link

Member Author

wilwade Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced with the typescript version!

wilwade added 2 commits September 10, 2024 11:27

Remove comment

9ddc9fb

Remove bad test

5ec71df

wilwade commented Sep 10, 2024

View reviewed changes

wilwade added 3 commits September 10, 2024 11:50

Fix lint issues

c6c1900

Update gitignore

23e281c

Update test for merge

45be2f3

shannonwells reviewed Sep 10, 2024

View reviewed changes

esbuild.mjs Outdated Show resolved Hide resolved

Update esbuild.mjs

3d4146b

wilwade commented Sep 10, 2024

View reviewed changes

esbuild-serve.mjs

sourcemap: 'external',

bundle: true,

minify: false,

Copy link

Member Author

wilwade Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping minify off here for the local serve.

wilwade requested a review from shannonwells September 10, 2024 21:08

shannonwells approved these changes Sep 10, 2024

View reviewed changes

wilwade merged commit 6797c99 into main Sep 11, 2024
1 check passed

wilwade deleted the bug/fix-brotli-compression-140 branch September 11, 2024 19:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Brotli compression #144

Fix Brotli compression #144

wilwade commented Sep 10, 2024

wilwade Sep 10, 2024

wilwade Sep 10, 2024

wilwade Sep 10, 2024

wilwade Sep 10, 2024

wilwade Sep 10, 2024

wilwade Sep 10, 2024

wilwade Sep 10, 2024

wilwade Sep 10, 2024

wilwade Sep 10, 2024

wilwade Sep 10, 2024

shannonwells left a comment

Fix Brotli compression #144

Fix Brotli compression #144

Conversation

wilwade commented Sep 10, 2024

What Changed?

Testing!

Brotli Sample File generation script

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shannonwells left a comment

Choose a reason for hiding this comment