Files
bun.sh/docs/project/developing.md
Jarred Sumner 6362414d65 Bun gets a new bundler (#2312)
* alright now just gotta try running it

* fix a gajillion compiler errors

* even more code

* okay i fixed more errors

* wip

* Update launch.json

* Update string_builder.zig

* `fast_debug_build_mode` makes debug build 2x faster

* Update bundle_v2.zig

* more code!

* It bundles!

* Rename `Bun.Transpiler` to `Bun.Bundler`

* `import()` expressions almost work

* wip attempt to get import() expr to work

* Bundle namespace imports

* Attempt to fix the issue with import() unsuccessfully

* consider current working directory when resolving relative paths (#2313)

* consider current working directory when resolving relative paths

fixes #2298

* comment test

---------

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>

* support `expect().toThrow(/pattern/)` (#2314)

- fix time-zone-dependent test failure

* fix missing `Blob` error messages on Linux (#2315)

* fix & clean up tests (#2318)

- skip flaky tests when running as `root`
- use `expect().toThrow()`
- clean up temporary files after tests

* feat(tty): add some `tty.WriteStream` methods to `process.{stdout, stderr}` (#2320)

* feat(stdio): add some `tty.WriteStream` methods

* chore(builtins): add process builtin gen'd code

* Fix docker install command

* `bun test` on macOS in GitHub Actions (#2322)

* Fixes #2323

* throw invalid parameter errors in `crypto.scryptSync` (#2331)

* throw invalid parameter errors

* remove comptime, add empty buffer function

* remove error_name comptime

* Add reference documentation for bun:test (#2327)

* Reorganize tests (#2332)

* Fix html-rewriter.test.js

* fix the wrong thing being incremented in hmr example (#2334)

* Add more test harness

* Improve Benchmarking page, small fixes (#2339)

* Improve benchmarking page

* WIP

* Add typescript instructions to hot

* Document preload in Plugins. Fix loader in plugin types.

* Fix typo

* Fix links

* run prettier

* Document openInEditor

* improve `Buffer` compatibility with Node.js (#2341)

* improve `Buffer` compatibility with Node.js

* use `memmove()`
allow `encoding` to be `undefined`

* run `bun test` after macOS builds (#2343)

* "binary" is an alias of "latin1"

Fixes https://github.com/oven-sh/bun/issues/2110

* More spec compliant `Blob.prototype.type` (#2340)

* Make `Blob.prototype. type` more spec compliant

* Add a few more checks for isNumber()

* Fix `make headers`

* Safer JSValue.isString()

* More tests for blob.slice

* Make `Blob.prototype.type` more spec compliant

* Add isASCII check

* Fix types

* Fix failing type test

* Update blob.zig

* Update blob.zig

* Fix .eql check on empty values

---------

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>

* Fix bug in test runner

* Support `import()` expressions

* Implement `require()`

* clean up bit_set.zig slightly

* Move some things around

* misc cleanup

* Cleanup some things

* Fix a lot of stuff

* Fix `module.exports.fn = fn;` in ESM entry point

* Fix crash due when printing file

* Fix issue with class names

* Fix issue with `export default identifier`

* Update js_parser.zig

* optimization: inline single-property object acceses and arrays

* Fix undefined memory in renamed symbols list

* Handle call target

* wip

* Inline it

* Fix undefined memory issue when reclaiming blocks in ast

* Halt linking on any parse errors

* alias

* Rename `enable_bundling` to `enable_legacy_bundling`

* Workaround anonymous struct literal zig bug

* Use slower approach (without bitset) because it doesn't break after 8 symbols

* Fix incorrectly-renaming statically defined symbols

* Handle more edgecases in our bit_set fork

* Reduce number of allocations for `define`

* Do not rename unbound symbols

* Clean up dot defines a little more

* Make the generated names prettier

* Workaround runtime symbol missing issue

* Fail the build on errors

* Support export * from

* Support `--outfile`

* partially fix renaming

* fanicer symbol renaming impl

* misc, extremely revertible cleanup

* Fix up some bugs with symbol renaming

* formatting

* Update launch.json

* Parse `__PURE__` comments

* clean up simd code for pure comments

* changes to merge

* workaround runtime issue

* Fix issue with `export * as` not propagating correctly

* Make all top-level declarations `var` when bundling

* Fix missing prefix

* Fix assigning to stack copy

* Fix missing runtime symbol

* Fix bug with namespace exports

* Dramatically reduce allocations

* Update launch.json

* Add missing flags

* Update js_parser.zig

* small cleanup

* Make the export name better

* Fix unnecessary `var foo = foo`

* Implement CommonJS -> ESM conversion

* Implement module redirects

* Port esbuild bundler tests for new bundler (#2380)

* started porting esbuild tests

* clean up test names and api before moving on

* port tests using a program i wrote

* replace todo generated comment

* fix generated tests not including some files

* work on tests

* [github web editor] add define, external, inject, minifySyntax, minifyWhitespace options.

* get most of the todo comments out of the way, but expectBundled does not handle most of the cases

* continue working on esbuild tests

* use test.skip for unsupported tests

* Fixups for test runner

* Hoist imports & exports

* Fix test

* Hoist classes

* bundler test refining, 51/835

* Fix runtime require

* bundler test refining, 81/835

* bundler test refining, 93/835

* Make the test work in any timezone

* feat(expect): update toBeInstanceOf (#2396)

* feat: update instanceof binding

* fix: according to PR comments

* Rename `expectObjectTypeCount` to `expectMaxObjectTypeCount`

* Fix socket tests with connection errors (#2403)

* release pending activity with connection error handler

* unref poll_ref

* remove trailing comma

* Organize Dockerfiles for official status

* Remove test Dockerfile

* Remove old Docker workflow

* Feat(test): add toMatch (#2404)

* Fix various fetch/response/request tests (#2416)

* fix most fetch tests, skip a few

* fastGet, toValueGC, and invalid init

* bigint unreachable, range error, log process as process

* remove extra fetch_headers

* remove js_type parameter, check isObject()

* throw invalid mime type error, use enum literal

* switch back to promise rejection

* RangeError pascal case

* Fix several bugs (#2418)

* utf16 codepoint with replacement character

* Fix test failure with `TextEncoder("ascii')`

* Add missing type

* Fix Response.prototype.bodyUsed and Request.prototype.bodyUsed

* Fix bug with scrypt error not clearing

* Update server.zig

* oopsie

* 💅

* docs: Use correct url in the 'Issues' link in README header (#2420)

* Fix crash when rendering error page and the server or network is slow

* [fetch] Make the default body value `null` when unspecified

This is better aligned with the fetch spec

* Make node-net tests less flaky

* [node:net] Fix issue with `listen` callback firing before it's listening

* Always clear timers in node test harness

* Fix out of bounds access

Repro'd in Buffer tests

* Update UWS

cc @cirospaciari

* Make this test more thorough

* Hanging abort test

* 0 length body is a null stream

* Several bug fixes (#2427)

* Fix test

* Fix segfault when unexpected type is passed in `expect().toThrow`

* Fix issues with request constructor

* Don't bother cloning headers when its empty

* woops

* more tests

* fix incorrect test

* Make the fetch error messages better

* Update response.zig

* Fix test that failed on macOS

* Fix test

* Remove extra hash table lookups

* Support running dummy registry directly

cc @alexlamsl

* Update test

* Update test

* fixup

* Workaround crash in test runner

* Fixup test

* Fixup test

* Update os.test.js

---------

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>

* Remove usages of port numbers in tests

* Set -O2 and -fno-rtti

* Remove -g

* Prevent undefined memory access

* [bun test] Implement `--rerun-each` flag to run each test N times

* Reduce number of module scopes created

* add some extra abort checks into streams (#2430)

* add some checks to avoid UAF

* avoid multiple calls to finalize if endFromJS is called more than once

* fix no-op comment

* mark as requested_end on abort

* remove requested_end from abort

* remove unnecessary check (#2432)

* Fix bug with scoped aliased dependencies in bun install on macOS

* remove `addLog`, remove `--prominent-compile-errors`

* Finish the upgrade

* Optional chaining flag

* Implement same_target_becomes_destructuring optimization

* bundler test refining, 109/835

* Reset bindings

* Support multiple entry points

* Implement `--entry-names` flag

* Use a tempdir with a better name

* prettier

* Log file name

* Update js_parser.zig

* Mark all bun builtins as external

* Make resolve errors actually errors

* Update bundler_default.test.ts

* Fix `await import(foo)`

* WIP react server components

* Do more stuff at runtime

* ✂️

* Support automatic JSX imports

* Use a module cache for now

* Update tsconfig.base.json

* Fix ThisOutsideFunctionNotRenamed

* woopsie

* moar cpu

* clamp it

* fixup

* Add a bunch of assertions

* Bun uses automatic runtime by default

* Parse Import Attributes

* Add a note about Valgrind

* Update developing.md

* Fix up code splitting for React Server Components

* Implement client component manifest

* Fix crash with --react-server-components and no client components

* Backport 4d31e3c917

* Update launch.json

* Fix for latest zig

* Workaround bug with ?[]const string

Occasionally saw alignment errors in this code

Workaround https://github.com/ziglang/zig/issues/15085

related: https://github.com/ziglang/zig/pull/15089

* switch to regular slice

* Avoid initializing named_imports and named_exports as undefined

* Reduce usages of `undefined`

* Add more assertions

* --watch wip

* Update javascript.zig

* Possibly fix the race condition

* Faster `do`

* bump allocator

* Reduce the size of `Symbol` slightly

* Alphabetically sort runtime import symbols, for determinism

* Prepare for code splitting

* handle overlapping stdout

* pure

* clean up some things

* Fix bug with `$$typeof`

* Address CommonJS -> ESM hoisting bug

* Support `"use server"` in manifest

* Implement `"use server"`

* Fix importing bun builtins when bundling

* Make `commonjs_to_esm` a feature flag, fix some splitting bugs

* ✂️

* fixme remove this

* Fix crash in longestCommonPath

* Chunking! Just need to do import paths now.

* Import paths work...now trying to figure out how to make runtime symbols work

* add workaround

* Replace `bun bun` with `bun build`

* Fix crash with dual package hazard

* Fix many CommonJS <> ESM interop bugs

* Support package.json `"sideEffects"`

also skip loading unnecessary package.json data in `bun run`

* add a not good --watch implementation

* bundler test refining, 140/831

* remove accidentally committed file

* do not return status code 1 on successful bundles

* bundler test refining, 159/830

* pass exit code to exitOrWatch

* clean up help menu

-remove two spaces to line up bun build
-moved all <r> tags to the end of the text they are colorizing
-moved other colors to the start of the text they colorize
-removed unneeded <r> tags, keeping only one at the start of the block

* importstar is fully ported

* wip

* you can run code in this branch now

* Disable this transform

* organize and document bundler tests

* Fix double import

* Fix sloppy mode function declarations

* Disable our CommonJS transform for now

* add `assertNotPresent` to make splitting cases easier

* Bump!

* Update bun.d.ts

* use import.meta.require in runtime code

* Disable this again

* Fix dirname

* Fix ESM -> CJS wrapper

* 💅

---------

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>
Co-authored-by: Alex Lam S.L <alexlamsl@gmail.com>
Co-authored-by: Derrick Farris <mr.dcfarris@gmail.com>
Co-authored-by: Ashcon Partovi <ashcon@partovi.net>
Co-authored-by: Dylan Conway <35280289+dylan-conway@users.noreply.github.com>
Co-authored-by: pfg <pfg@pfg.pw>
Co-authored-by: Colin McDonnell <colinmcd94@gmail.com>
Co-authored-by: dave caruso <me@paperdave.net>
Co-authored-by: zhiyuan <32867472+zhiyuang@users.noreply.github.com>
Co-authored-by: Dylan Conway <dylan.conway567@gmail.com>
Co-authored-by: Kamil Ogórek <kamil.ogorek@gmail.com>
Co-authored-by: Ciro Spaciari <ciro.spaciari@gmail.com>
2023-04-07 20:08:01 -07:00

8.4 KiB

Configuring a development environment for Bun usually takes 30-90 minutes depending on your operating system.

Linux/Windows

The best way to build Bun on Linux and Windows is with the official Dev Container. It ships with Zig, JavaScriptCore, Zig Language Server, and more pre-installed on an instance of Ubuntu.

{% image src="https://user-images.githubusercontent.com/709451/147319227-6446589c-a4d9-480d-bd5b-43037a9e56fd.png" /%}

To develop on Linux/Windows, Docker is required. If using WSL on Windows, it is recommended to use Docker Desktop for its WSL2 integration.

VSCode

If you're using VSCode, you'll need to have the Dev Containers extension installed.

To get started, open VS Code in the bun repository. The first time you try to open the dev container, the extension will automatically build it for you, based on Dockerfile.devcontainer.

To open the dev container, open the command palette (Ctrl + Shift + P) and run: Dev Containers: Reopen in Container. To later rebuild it (only needed when the devcontainer itself changes, not the Bun code), run: Dev Containers: Rebuild and Reopen in Container.

Other editors and CLI

If you're using another editor or want to manually control the dev container from the command line or a script, you'll need to install the Dev Container CLI: npm install -g @devcontainers/cli.

To create and start the dev container, in the bun repository, locally run:

# `make devcontainer-<command>` should be equivalent
# to `devcontainer <command>`, it just sets the architecture
# so if you're on ARM64, it'll do the right thing
$ make devcontainer-up

To just build the dev container image, run:

$ make devcontainer-build

To start a shell inside the container, run:

$ make devcontainer-sh

# if it attaches to the container non-interactively,
# instead use the regular docker exec command:
$ docker exec -it <container-name/id> zsh

Cloning

You will then need to clone the GitHub repository inside that container.

# First time setup
$ gh auth login # if it fails to open a browser, use Personal Access Token instead
$ gh repo clone oven-sh/bun . -- --depth=1 --progress -j8

Building

# Compile Bun dependencies (zig is already compiled)
$ make devcontainer

# It initializes and updates all submodules except WebKit, because WebKit
# takes a while and it's already compiled for you. To do it manually, use:
$ git -c submodule."src/bun.js/WebKit".update=none submodule update --init --recursive --depth=1 --progress

# Build Bun for development
$ make dev

# Run Bun
$ bun-debug

MacOS

Install LLVM 15 and homebrew dependencies:

$ brew install llvm@15 coreutils libtool cmake libiconv automake ninja gnu-sed pkg-config esbuild go rust

Bun (& the version of Zig) need LLVM 15 and Clang 15 (clang is part of LLVM). Make sure LLVM 15 is in your $PATH:

$ which clang-15

If not, run this to manually link it:

# use fish_add_path if you're using fish
$ export PATH="$PATH:$(brew --prefix llvm@15)/bin"
$ export LDFLAGS="$LDFLAGS -L$(brew --prefix llvm@15)/lib"
$ export CPPFLAGS="$CPPFLAGS -I$(brew --prefix llvm@15)/include"

Install Zig

{% callout %} ⚠️ Warning — You must use the same version of Zig used by Bun in oven-sh/zig. Installing with brew or via Zig's download page will not work! {% /callout %}

Use zigup to install the version of Zig (ZIG_VERSION) specified in the official Dockerfile. For example:

$ zigup 0.11.0-dev.1783+436e99d13

Building

To install and build dependencies:

# without --depth=1 this will take 20+ minutes on 1gbps internet
# mostly due to WebKit
$ git submodule update --init --recursive --progress --depth=1 --checkout
$ bun install
$ make vendor identifier-cache

To compile the C++ bindings:

# without -j this will take 30+ minutes
$ make bindings -j12

Testing

To verify the build worked, lets print the version number on the development build of Bun.

$ packages/debug-bun-darwin-*/bun-debug --version
bun 0.x.y__dev

You will want to add packages/debug-bun-darwin-arm64/ or packages/debug-bun-darwin-x64/ (depending on your architecture) to $PATH so you can run bun-debug from anywhere.

Troubleshooting

If you see an error when compiling libarchive, run this:

$ brew install pkg-config

If you see an error about missing files on zig build obj, make sure you built the headers.

$ make headers

JavaScript builtins

When you change anything in src/bun.js/builtins/js/*, run this:

$ make clean-bindings generate-builtins && make bindings -j12

That inlines the JavaScript code into C++ headers using the same builtins generator script that Safari uses.

Code generation scripts

Bun leverages a lot of code generation scripts.

The ./src/bun.js/bindings/headers.h file has bindings to & from Zig <> C++ code. This file is generated by running the following:

$ make headers

This ensures that the types for Zig and the types for C++ match up correctly, by using comptime reflection over functions exported/imported.

TypeScript files that end with *.classes.ts are another code generation script. They generate C++ boilerplate for classes implemented in Zig. The generated code lives in:

$ make codegen

Lastly, we also have a code generation script for our native stream implementations. To run that, run:

$ make generate-sink

You probably won't need to run that one much.

Modifying ESM core modules

Certain modules like node:fs, node:path, node:stream, and bun:sqlite are implemented in JavaScript. These live in src/bun.js/*.exports.js files.

While Bun is in beta, you can modify them at runtime in release builds via the environment variable BUN_OVERRIDE_MODULE_PATH. When set, Bun will look in the override directory for <name>.exports.js before checking the files from src/bun.js (which are now baked in to the binary). This lets you test changes to the ESM modules without needing to re-compile Bun.

Troubleshooting

If you encounter error: the build command failed with exit code 9 during the build process, this means you ran out of memory or swap. Bun currently needs about 8 GB of RAM to compile.

Valgrind

On Linux, valgrind can help find memory issues.

Keep in mind:

  • JavaScriptCore doesn't support valgrind. It will report spurious errors.
  • Valgrind is slow
  • Mimalloc will sometimes cause spurious errors when debug build is enabled

You'll need a very recent version of Valgrind due to DWARF 5 debug symbols. You may need to manually compile Valgrind instead of using it from your Linux package manager.

--fair-sched=try is necessary if running multithreaded code in Bun (such as the bundler). Otherwise it will hang.

valgrind --fair-sched=try --track-origins=yes bun-debug <args>