Duplicated modules under optimizeDeps #13538
Comments
|
possible duplicate of #9171 ? |
|
I can't say this with absolute certainty, but I think that this is a different problem (with similar error symptoms, though). #9171 describes duplicated modules within the chunks generated by the optimizer ( Essentially, there are two "import paths" that are not merged into the same module copy, hence the module existing twice. Edit: I tried running the plugin from the linked issue's description. It does not detect any duplicated modules. But it does demonstrate the issue nicely. The following is a list of all modules rendered during the build ( vite v4.3.9 building for production...
✓ 11 modules transformed.
[
'\x00vite/modulepreload-polyfill',
'/home/michael/projects/temp/vite-duplicated-modules-repro/external/node_modules/dep1/init1.js',
/** This is the normal copy of the module */
'/home/michael/projects/temp/vite-duplicated-modules-repro/external/node_modules/shared/inner.js',
'/home/michael/projects/temp/vite-duplicated-modules-repro/external/node_modules/dep2/init2.js',
/** This module contains a bundled copy of `inner.js` as well! */
'/home/michael/projects/temp/vite-duplicated-modules-repro/node_modules/.vite/deps_build-dist/shared.js',
'/home/michael/projects/temp/vite-duplicated-modules-repro/src/app/index.js',
'/home/michael/projects/temp/vite-duplicated-modules-repro/src/index.html'
]
dist/index.html 0.32 kB │ gzip: 0.23 kB
dist/assets/index-4ec38ee8.js 1.14 kB │ gzip: 0.57 kB
✓ built in 114msThe contents of // external/node_modules/shared/inner.js
window.SHARED_INNER_EXECUTED ?? (window.SHARED_INNER_EXECUTED = 0);
window.SHARED_INNER_EXECUTED += 1;
//# sourceMappingURL=shared.js.mapVite config (also see reproduction repo)export default defineConfig({
root: resolve(__dirname, "src"),
optimizeDeps: {
disabled: false,
},
build: {
commonjsOptions: { include: [] },
},
plugins: [
autoInitPlugin(),
{
name: "Test",
apply: "build",
enforce: "pre",
renderChunk(code, chunk) {
console.log(Object.keys(chunk.modules));
return null;
},
},
],
}); |
because the rewritten-app is inside node_modules, dep Optimizer refused the scan the deps and thus does not check if there are already optmized dependencies. We can fake it though and make the importer lookup from root `/` since the module-resolver will handle that. dep optimizer then cannot see that its inside node_modules are does its job. similar issue here: vitejs/vite#13538
because the rewritten-app is inside node_modules, dep Optimizer refused the scan the deps and thus does not check if there are already optmized dependencies. We can fake it though and make the importer lookup from root `/` since the module-resolver will handle that. dep optimizer then cannot see that its inside node_modules are does its job. similar issue here: vitejs/vite#13538
|
I don't know much about Vite, so I'm not sure this is related, but I am getting duplicate modules breaking my app, and using With this config, I get errors like these: I have verified that paths like Without the I do have an |
|
Maybe my issue is not optimizeDeps-related. Changing my alias: {
'solid-js': path.resolve('./node_modules/solid-js'),
}, |
|
Dunno if this is related, but I just ran into an issue where running I narrowed it down to the use of For context... This happened for me in a monorepo using "npm workspaces". We mainly use This preparation looks a little like: # Generate PNPM Workspace and Settings
cat 'package.json' | yq eval '{"packages": .workspaces}' -P - > 'pnpm-workspace.yaml'
echo 'link-workspace-packages = true' > '.npmrc'
echo 'shared-workspace-lockfile = false' >> '.npmrc' # <---- this was causing the duplication
echo 'shamefully-hoist = true' >> '.npmrc'
# Generate PNPM Lockfile
pnpm import
# Install Dependencies
pnpm install --frozen-lockfileThe deployment/ejection looks like: # Create isolated/deployable application
pnpm deploy --filter="$project_name" --prod /usr/src/deployIn between those stages we would build and test etc... Using either This is where we noticed that the resulting assets for one of our very-small frontend packages was slightly larger than expected. Running Annoyingly, that option is kind of important for backend applications (as it seems to ensure deps with binaries are correctly exported when deploying), but for frontend applications/assets having it is causing this bundling bloat. I was hoping I could create an Anyways.... in my case at least, this issue seems to be related more to |
Describe the bug
Hi everyone,
as this is my first issue here i'd like to thank you for this great tool ❤.
I am having an issue with vite's optimization of dependencies during development: under some (admittingly very specific, but in my opinion legitimate) circumstances, vite will instantiate a module from
node_modulestwice.This in turn leads to side effects happening multiple times, confused object identities and
instanceofchecks failing, which can be very hard to debug. When the depsOptimizer is not present (e.g. duringvite build), everything works as expected.I'll do my best describing the issue. I have also attached a repository with as minimal of a reproduction as i could manage.
The problem
At my company, we are using a vite/rollup plugin to add some reflection capabilities to an application.
Essentially, from a single import in the application's code to a virtual module, the apps entire dependency graph may be scanned. All transitive node dependencies are visited and for certain packages code is generated, including imports to modules or objects within those packages.
The reproduction repository uses a simplified system that produces the same problem. A single
import "virtual:init-all-packages";triggers analysis of the dependency graph, visits all referenced packages (just viadependencies), and if theirpackage.jsoncontains a"customAutoInit": "..."declaration, that init module will be imported to trigger the initialization code within that module.The
import "virtual:init-all-packages";is thus turned into:Note that this virtual module cannot use bare imports (e.g.
import "dep1/init1.js") because transitive dependencies may not be reachable from the virtual modules "location" (nestednode_modulesetc. should be taken into account). This is why the plugin resolves the module ids passing someimporterfrom where the import is possible. This results in an absolute path like/@fs/.../dep1/init1.js?v=d7007810, which will then be ignored by the depsOptimizer.The problem here arises when the unoptimized modules mentioned above use node modules that are also used by the application directly (and are thus optimized in
node_modules/.vite/...). There will be two copies of any modules used in this way: raw files in node_modules and a copy bundled by esbuild in.vite, with different parts of the application observing one or the other. In the reproduction repository, the moduleshared/inneris duplicated.I don't think excluding certain packages from optimization would really help here, since only a static set of packages can be configured and dependencies may add additional imports whenever they feel like it, making any such solution very hard to maintain. Also optimization is a great feature in principle and i would like to make it work correctly in this case.
Additional observations
I have dug around vite's code a bit.
All problems would be avoided if the
resolve(...)call in our plugin would receive an optimized module id instead of the raw location.The implementation of
resolveeven has this capability, adding a missing import and then re-triggering optimization, but that branch is skipped in these specific circumstances because the (virtual) importer is innode_modules.However, i cannot use a location outside of
node_modulesbecause the module id in question might not be reachable from elsewhere.I have verified locally that removing the
(importer && isInNodeModules(importer))check from the linked condition would fix the issue in this case. However that line is probably there for a good reason (?). The check was introduced by Evan in this commit, but I couldn't find any additional context.An additional option for resolve, or an improved
skipcondition (although i can't think of one) or any other kind of workaround with would be greatly appreciated. However, because the optimizer can alter the observable behaviour of the app in a significant matter, I believe a fix should be internal.Reproduction
https://github.com/mbeckem/vite-duplicated-modules-repro
Steps to reproduce
Run
pnpm installfollowed bypnpm vite devand then open the browser, for more details see the linked repository's README.System Info
System: OS: Linux 5.15 Ubuntu 22.04.2 LTS 22.04.2 LTS (Jammy Jellyfish) CPU: (16) x64 AMD Ryzen 7 PRO 4750U with Radeon Graphics Memory: 17.12 GB / 30.59 GB Container: Yes Shell: 5.1.16 - /bin/bash Binaries: Node: 18.13.0 - ~/.nvm/versions/node/v18.13.0/bin/node npm: 8.19.3 - ~/.nvm/versions/node/v18.13.0/bin/npm Browsers: Chrome: 114.0.5735.133 Firefox: 114.0.1 npmPackages: vite: ^4.3.9 => 4.3.9Used Package Manager
pnpm
Logs
Click to expand!
Validations
The text was updated successfully, but these errors were encountered: