Commit Graph

35 Commits

Author SHA1 Message Date
Jarred Sumner
528620e9ae Add postinstall optimizer with native binlink support and script skipping (#24283)
## Summary

This PR introduces a new postinstall optimization system that
significantly reduces the need to run lifecycle scripts for certain
packages by intelligently handling their requirements at install time.

## Key Features

### 1. Native Binlink Optimization

When packages like `esbuild` ship platform-specific binaries as optional
dependencies, we now:
- Detect the native binlink pattern (enabled by default for `esbuild`)
- Find the matching platform-specific dependency based on target CPU/OS
- Link binaries directly from the platform-specific package (e.g.,
`@esbuild/darwin-arm64`)
- Fall back gracefully if the platform-specific package isn't found

**Result**: No postinstall scripts needed for esbuild and similar
packages.

### 2. Lifecycle Script Skipping

For packages like `sharp` that run heavy postinstall scripts:
- Skip lifecycle scripts entirely (enabled by default for `sharp`)
- Prevents downloading large binaries or compiling native code
unnecessarily
- Reduces install time and potential failures in restricted environments

## Configuration

Both features can be configured via `package.json`:

```json
{
  "nativeDependencies": ["esbuild", "my-custom-package"],
  "ignoreScripts": ["sharp", "another-package"]
}
```

Set to empty arrays to disable defaults:
```json
{
  "nativeDependencies": [],
  "ignoreScripts": []
}
```

Environment variable overrides:
- `BUN_FEATURE_FLAG_DISABLE_NATIVE_DEPENDENCY_LINKER=1` - disable native
binlink
- `BUN_FEATURE_FLAG_DISABLE_IGNORE_SCRIPTS=1` - disable script ignoring

## Implementation Details

### Core Components

- **`postinstall_optimizer.zig`**: New file containing the optimizer
logic
- `PostinstallOptimizer` enum with `native_binlink` and `ignore`
variants
  - `List` type to track optimization strategies per package hash
  - Defaults for `esbuild` (native binlink) and `sharp` (ignore)
  
- **`Bin.Linker` changes**: Extended to support separate target paths
  - `target_node_modules_path`: Where to find the actual binary
  - `target_package_name`: Name of the package containing the binary
  - Fallback logic when native binlink optimization fails

### Modified Components

- **PackageInstaller.zig**: Checks optimizer before:
  - Enqueueing lifecycle scripts
  - Linking binaries (with platform-specific package resolution)
  
- **isolated_install/Installer.zig**: Similar checks for isolated linker
mode
  - `maybeReplaceNodeModulesPath()` resolves platform-specific packages
  - Retry logic without optimization on failure

- **Lockfile**: Added `postinstall_optimizer` field to persist
configuration

## Changes Included

- Updated `esbuild` from 0.21.5 to 0.25.11 (testing with latest)
- VS Code launch config updates for debugging install with new flags
- New feature flags in `env_var.zig`

## Test Plan

- [x] Existing install tests pass
- [ ] Test esbuild install without postinstall scripts running
- [ ] Test sharp install with scripts skipped
- [ ] Test custom package.json configuration
- [ ] Test fallback when platform-specific package not found
- [ ] Test feature flag overrides

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Native binlink optimization: installs platform-specific binaries when
available, with a safe retry fallback and verbose logging option.
* Per-package postinstall controls to optionally skip lifecycle scripts.
* New feature flags to disable native binlink optimization and to
disable lifecycle-script ignoring.

* **Tests**
* End-to-end tests and test packages added to validate native binlink
behavior across install scenarios and linker modes.

* **Documentation**
  * Bench README and sample app migrated to a Next.js-based setup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Dylan Conway <dylan.conway567@gmail.com>
2025-11-03 20:36:22 -08:00
Dylan Conway
f770b1b1c7 fix(install): fix optional peer resolving (#24272)
### What does this PR do?
Allows optional peers to resolve to package if possible.

Optional peers aren't auto-installed, but they should still be given a
chance to resolve. If they're always left unresolved it's possible for
multiple dependencies on the same package to result in different peer
resolutions when they should be the same. For example, this bug this
could cause monorepos using elysia to have corrupt node_modules because
there might be more than one copy of elysia in `node_modules/.bun` (or
more than the expected number of copies).

fixes #23725
most likely fixes #23895

fixes ENG-21411

### How did you verify your code works?
Added a test for optional peers and non-optional peers that would
previously trigger this bug.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Improved resolution of optional peer dependencies during isolated
installations, with better propagation across package hierarchies.

* **Tests**
* Added comprehensive test suite covering optional peer dependency
scenarios in isolated workspaces.
* Added test fixtures for packages with peer and optional peer
dependencies.
* Enhanced lockfile migration test verification using snapshot-based
assertions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2025-11-01 22:38:36 -07:00
Dylan Conway
6c8635da63 fix(install): isolated installs with transitive self dependencies (#23222)
### What does this PR do?
Packages with self dependencies at a different version were colliding
with the current version in the store node_modules. This pr nests them
in another node_modules

Example:
self-dep@1.0.2 has a dependency on self-dep@1.0.1.

self-dep@1.0.2 is placed here in:
`./node_modules/.bun/self-dep@1.0.2/node_modules/self-dep`

and it's self-dep dependency symlink is now placed in:

`./node_modules/.bun/self-dep@1.0.2/node_modules/self-dep/node_modules/self-dep`

fixes #22681
### How did you verify your code works?
Manually tested the linked issue is working, and added a test

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2025-10-04 02:59:47 -07:00
Dylan Conway
4580e11fc3 [PKG-517] fix(install): --linker=isolated should spawn scripts on the main thread (#21425)
### What does this PR do?
Fixes thread safety issues due to file poll code being not thread safe.
<!-- **Please explain what your changes do**, example: -->

<!--

This adds a new flag --bail to bun test. When set, it will stop running
tests after the first failure. This is useful for CI environments where
you want to fail fast.

-->

### How did you verify your code works?
Added tests for lifecycle scripts. The tests are unlikely to reproduce
the bug, but we'll know if it actually fixes the issue if
`test/package.json` doesn't show in flaky tests anymore.
<!-- **For code changes, please include automated tests**. Feel free to
uncomment the line below -->

<!-- I wrote automated tests -->

<!-- If JavaScript/TypeScript modules or builtins changed:

- [ ] I included a test for the new code, or existing tests cover it
- [ ] I ran my tests locally and they pass (`bun-debug test
test-file-name.test`)

-->

<!-- If Zig files changed:

- [ ] I checked the lifetime of memory allocated to verify it's (1)
freed and (2) only freed when it should be
- [ ] I included a test for the new code, or an existing test covers it
- [ ] JSValue used outside of the stack is either wrapped in a
JSC.Strong or is JSValueProtect'ed
- [ ] I wrote TypeScript/JavaScript tests and they pass locally
(`bun-debug test test-file-name.test`)
-->

<!-- If new methods, getters, or setters were added to a publicly
exposed class:

- [ ] I added TypeScript types for the new methods, getters, or setters
-->

<!-- If dependencies in tests changed:

- [ ] I made sure that specific versions of dependencies are used
instead of ranged or tagged versions
-->

<!-- If a new builtin ESM/CJS module was added:

- [ ] I updated Aliases in `module_loader.zig` to include the new module
- [ ] I added a test that imports the module
- [ ] I added a test that require() the module
-->

---------

Co-authored-by: taylor.fish <contact@taylor.fish>
2025-07-28 12:29:47 -07:00
Dylan Conway
f24e8cb98a implement "nodeLinker": "isolated" in bun install (#20440)
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
2025-07-09 00:19:57 -07:00
Dylan Conway
73bcff9d01 fix 16842 (#16952) 2025-01-31 22:39:30 -08:00
Dylan Conway
5058bd3913 handle bundle(d)Dependencies in bun install (#16055) 2024-12-31 13:40:55 -08:00
Dylan Conway
d5f1f2f8ad Use the same hoisting logic for text lockfile (#15778) 2024-12-16 16:37:46 -08:00
Dylan Conway
247456b675 fix(install): continue install if optional postinstall fails (#14783) 2024-10-23 21:58:53 -07:00
Dylan Conway
4a58a97fa0 fix sleep tests in bun-install-registry.test.ts (#13874) 2024-09-10 22:28:57 -07:00
Dylan Conway
5f7b96b58f fix(install): optional peer dependency bugfix (#12485)
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
Co-authored-by: dylan-conway <dylan-conway@users.noreply.github.com>
2024-07-10 20:04:32 -07:00
Zack Radisic
cdc68a2237 .npmrc follow up (#12390)
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
Co-authored-by: Dylan Conway <35280289+dylan-conway@users.noreply.github.com>
2024-07-10 19:35:21 -07:00
Dylan Conway
da27f22622 fix(install): install binaries for packages installed multiple times (#11886)
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
Co-authored-by: dylan-conway <dylan-conway@users.noreply.github.com>
2024-06-28 03:24:39 -07:00
Dylan Conway
b23ba1fe18 fix(install): allow unresolvable optionalDependencies (#11977) 2024-06-19 15:23:51 -07:00
Dylan Conway
fa952b163c fix(install): tarball extracting bugfix (#11864)
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
2024-06-15 00:22:16 -07:00
Dylan Conway
eedb3e530c fix(install): handle transitive folder dependencies (#10445)
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
Co-authored-by: Jarred-Sumner <Jarred-Sumner@users.noreply.github.com>
2024-06-15 00:08:40 -07:00
Dylan Conway
e9d1e7ac5e fix(install): return non-zero exit code when tarballs fail to download (#11828) 2024-06-14 01:15:53 -07:00
Dylan Conway
5e619ee337 fix(install): manifest parsing and peer dependency bugfix (#11763)
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
2024-06-10 20:29:16 -07:00
Dylan Conway
c614d5b1da fix(install): aliased workspace without version in package.json (#11630) 2024-06-05 22:04:57 -07:00
Dylan Conway
c2eef9eded fix(install): manifest package name mismatch (#11549) 2024-06-02 22:56:57 -07:00
Dylan Conway
c65a911a57 fix(install): lockfile printing bugfix (#11545) 2024-06-02 18:49:07 -07:00
Jarred Sumner
cabfca4039 Fix adding packages in workspaces (#11177)
Co-authored-by: Dylan Conway <dylan.conway567@gmail.com>
2024-05-20 21:24:37 -07:00
Dylan Conway
d966fe6afd fix flaky and broken test (#10500)
* add package

* fix test
2024-04-25 01:14:11 -07:00
dave caruso
5296c26dab fix bunx-bins verdaccio package (#9697)
* fix bunx-bins verdaccio package

* env suck

* [autofix.ci] apply automated fixes

* ugh

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-03-29 17:55:06 -07:00
dave caruso
e808cdb725 windows: fix argument handling in bin shim (#9622)
* awa

* update v5

* a

* [autofix.ci] apply automated fixes

* tarball

* a

* a

* blah

* [autofix.ci] apply automated fixes

* bump

* wah

* [autofix.ci] apply automated fixes

* farther

* a

* [autofix.ci] apply automated fixes

* im confused

* [autofix.ci] apply automated fixes

* Fix typo (#9623)

* Fix crash in Bun.escapeHTML (#9619)

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>

* Fixes #9610

* Implement `fs.openAsBlob` (#9628)

* Implement `fs.openAsBlob`

* Use a function

---------

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>

* Fix `bun install -g` not working on Docker
Closes #8753

* Revert "Fix `bun install -g` not working on Docker"

This reverts commit 20fce1a1be.

* what happens if nonblocking tty (#9608)

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>

* merge

* a

* blah

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
Co-authored-by: sequencerr <45060278+sequencerr@users.noreply.github.com>
Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>
Co-authored-by: Ashcon Partovi <ashcon@partovi.net>
2024-03-26 22:38:01 -07:00
Dylan Conway
e39a7851c8 fix(install): package binary map bugfix (#9621) 2024-03-25 12:49:52 -07:00
Dylan Conway
931e04b019 fix(install): hoisting bug with devDependencies (#9519)
* fix with tests

* move to hoisting describe scope

* Update bun-install-registry.test.ts
2024-03-19 21:46:54 -07:00
Dylan Conway
d37fbbd4e0 fix(install): lifecycle script changes (#8943)
* empty trustedDependencies

* tests

* handle edgecases with default trusted dependencies

* could be zero length

* --trusted and skipped scripts

* resolver too

* second run --trusted

* --trust, better formatting

* more tests

* --trusted applies to dep deps, more tests

* progress

* fix build

* fix crash, make it look good, comments

* alphabetize, verbose log

* feature flag

* update lockfile

* update skipped text

* check update requests first

* be more careful with inline strings

* only with scripts

* fix tests, todo tests

* fix another test

* fix merge

* fix fix merge

* check binding.gyp for tarball and git resolutions

* remove dead code

* debug assert

* move newline printing

* use enum for `__has_install_script`

* oops

* clone packages

* Update src/install/install.zig

Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>

---------

Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
2024-03-07 19:22:21 -08:00
Dylan Conway
76ed8d8086 fix(install): remove invalid bin symlinks (#8293)
* handle invalid bin symlink

* test
2024-01-19 18:37:03 -08:00
Dylan Conway
351e47355a add default trustedDependencies and run lifecycle scripts during installation (#7132)
* [install] allow parallel execution of `postinstall` scripts

- fixes spurious exit code 42 from `spawn()`

* postinstall to a pipe

* feat(install): include top 500 packages as defaults for postinstall

* windows newline handling*

*i did not test it

* stuff

* cool

* a

* fix merge

* set `has_trusted_dependencies`

* fix a bunch of tests

* fix merge

* remove `PackageManager`

* remove commented code

* change to function

* Update lockfile.zig

* run scripts if added to `trustedDependencies` after install

* packages without `resolved` properties

* node-gyp scripts

* node-gyp script in the root

* another test

* git deps run prepare scripts

* fix merge

* run lifecycle scripts during installation

* Update lockfile.zig

* always increment

* 🏗️

* update tests

* tickWIthoutIdle

* const uws

* loop forwards through trees

* single buffer bitset list

* tag.isGit

* windows path separators

* `bun.sys.read` and enable/disable buffering

* fix test and waiter thread

* waiter thread and tests

* Update bun-install-registry.test.ts

* workspace exclude `preprepare` and `postprepare`

* Create esbuild.test.ts

* make sure length is the same

* remove deferred binlinks, add estrella test

* test with another version

* address some comments

* remove .verdaccio-db.json

* ooops

* fix build

* use `pid` to wait

* dont register pid_poll when using waiter thread

* stress test

* free

* fix failing tests

* fix linux crash, snapshot stress test

* oops

* concurrent scripts

* activate as soon as possible

* test

* delete stress test packages

* remove unused packages

* comment stress test and maybe fix segfault

* delete snapshot

* fix assertion

* use cpu_count * 2 for default concurrent scripts

* gear emoji

* add --concurrent-scripts to docs

* more docs

---------

Co-authored-by: alexlamsl <alexlamsl@gmail.com>
Co-authored-by: dave caruso <me@paperdave.net>
Co-authored-by: Dylan Conway <33744874+MilesWright7@users.noreply.github.com>
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
2023-12-11 22:08:25 -08:00
Dylan Conway
8db407e9a7 make tmpdir more random (#7243)
* make tmpdir more random

* 100
2023-11-21 14:29:35 -08:00
Dylan Conway
85c997513d node modules bin directory fix (#7151)
* place binaries in `.bin` of package node_modules

* test

* fix tests
2023-11-15 22:44:44 -08:00
Dylan Conway
f7f6233ea8 fix semver edge cases and buffer used for prerelease comparisons (#6981)
* use different buf for group and manifest versions

* tests

* tests for installs that should fail

* allow `<=<prerelease-version`

* `Bun.semver.satisfies`

* one

* stringify

* symbol tests

* deinit

* arena

* more tests

* fix test

* version always on the left

* `Bun.semver.order`

* handle more edge cases, more tests

* whitespace

* more
2023-11-08 03:36:17 -08:00
Dylan Conway
2f301e8404 skip tags in version ranges 2023-11-07 07:55:01 +01:00
Dylan Conway
142cdf1714 more install tests (#6867)
* some tests

* Apply suggestions from code review

* Update src/install/install.zig

* `zig fmt`

* Update install.zig

---------

Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>
2023-11-03 02:37:03 -07:00