Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error on import inside worker module #22

Open
fmerg opened this issue Apr 7, 2023 · 1 comment
Open

Error on import inside worker module #22

fmerg opened this issue Apr 7, 2023 · 1 comment

Comments

@fmerg
Copy link

fmerg commented Apr 7, 2023

I have a CPU intensive operation over JubJub, which I want to split into parallel tasks and offload to different worker threads. Each worker needs a eddsa instance to work with. My context does not allow for shared memory among workers and transferring eddsa to the worker threads is also not possible in any feasible way. I would like each worker to create its own eddsa instance:

const circomlibjs = require('circomlibjs');
...
const eddsa = await circomlibjs.buildEddsa();

However, importing circomlibjs inside the worker module raises an error when the worker starts running:

TypeError: Cannot destructure property 'mod' of 'threads.workerData' as it is undefined.

  at workerThread (node_modules/web-worker/cjs/node.js:151:5)
  at Object.<anonymous> (node_modules/web-worker/cjs/node.js:79:56)
  at Object.<anonymous> (node_modules/circomlibjs/node_modules/ffjavascript/build/main.cjs:8:14)

The exact error depends on the multithreading library in use. The above one is raised with workerpool. If I use the native node:worker_threads or web workers for the browser, I get something like

TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
    at new NodeError (node:internal/errors:399:5)
    at validateString (node:internal/validators:163:11)
    at Module.require (node:internal/modules/cjs/loader:1120:3)
    at require (node:internal/modules/helpers:112:18)
    ...
    at Module.load (node:internal/modules/cjs/loader:1103:32) {
  code: 'ERR_INVALID_ARG_TYPE'

I do not face import issues with other libraries. I wonder if this is my fault (that is, if circomlibjs should be imported in a different way inside js threads) or a bug.

Any idea or hint is welcome. Thanks in advance.

@tornadocontrib
Copy link

tornadocontrib commented Apr 21, 2024

@fmerg Solution would be requiring web-worker dependency. If you are bundling for the browser, you need to exclude it.

Something like

{
    input: 'src/merkleTreeWorker.ts',
    output: [
      {
        file: 'static/merkleTreeWorker.umd.js',
        format: "umd",
        esModule: false
      },
    ],
    treeshake: 'smallest',
    external: ['web-worker'],
    plugins: [
      esbuild({
        include: /\.[jt]sx?$/,
        minify: false,
        sourceMap: true,
        target: 'es2016',
      }),
      nodeResolve(),
      commonjs(),
      json(),
      replace({
        'process.browser': 'true'
      })
    ],
  }

Also look at the process.browser replace as the process object doesn't exist for browsers ( I think it is intended to support bundlers like webpack, but for rollup.js you need to manually replace it to true statement )

So either circomlibjs or ffjavascript would work inside Web Workers and node:worker_threads. Except that you need to bundle the worker file to use only either node.js or browser module.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants