mirrors/bun.sh
1
0
Fork 0
mirror of https://github.com/oven-sh/bun synced 2026-02-11 11:29:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
b7ec589a265d6dfff7202fddd9744e7bdfcb42f5
bun.sh/src
History
robobun 9fd5b20aa3 feat: Add WebKit text codec support for 24 additional encodings (#21835) 
## Summary
This PR integrates WebKit's text codec implementations into Bun's
TextDecoder, adding support for 24 additional character encodings beyond
the native UTF-8, UTF-16, and Latin1.

Fixes https://github.com/oven-sh/bun/issues/11564

## What's New
### Supported Encodings (24 total)
- **11 single-byte encodings**: IBM866, ISO-8859-3/6/7/8/8-I, KOI8-U,
windows-874/1253/1255/1257
- **7 CJK encodings**: Big5, EUC-JP, ISO-2022-JP, Shift_JIS, EUC-KR,
GBK, GB18030
- **2 special encodings**: x-user-defined, replacement

### Implementation Details
- Integrated WebKit's text codec C++ implementations
- Generated static encoding tables from WHATWG spec (no ICU dependency)
- Created C++ wrapper for Zig/C++ interop
- All encoding aliases are supported (e.g., `sjis` → `shift_jis`)
- Proper whitespace trimming for encoding labels

## Testing
- ✅ Added comprehensive tests for all supported encodings
- ✅ Passes Web Platform Tests for single-byte decoders
- ✅ Passes Web Platform Tests for encoding labels
- ✅ All 2,227 tests pass

## Test Output
```
bun test v1.2.19 (9feaab47)
 2207 pass
 0 fail
 5012 expect() calls
Ran 2207 tests across 1 file. [899.00ms]
```

## Not Included
The following encodings were not added due to ICU data loading
constraints:
- ISO-8859-2, 4, 5, 10, 13, 14, 15, 16
- Windows-1250, 1251, 1254, 1256, 1258
- KOI8-R, macintosh, x-mac-cyrillic

## Example Usage
```javascript
// CJK encodings
const decoder = new TextDecoder("shift_jis");
const bytes = new Uint8Array([0x82, 0xb1, 0x82, 0xf1]);
console.log(decoder.decode(bytes)); // "こん"

// Single-byte encodings
const greekDecoder = new TextDecoder("iso-8859-7");
const greekBytes = new Uint8Array([0xC3, 0xe5, 0xe9, 0xdc]);
console.log(greekDecoder.decode(greekBytes)); // "Γειά"
```

🤖 Generated with [Claude Code](https://claude.ai/code)

---------

Co-authored-by: Claude <claude@anthropic.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2025-08-14 22:58:25 -07:00
..
allocators
Add thread safety checks to MimallocArena (#21806)
2025-08-12 22:25:04 -07:00
analytics
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
api
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
ast
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
async
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
bake
fix "String contains an invalid character" when rendering multiple frontend errors (#21844)
2025-08-14 12:31:37 -07:00
base64
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
bun.js
feat: Add WebKit text codec support for 24 additional encodings (#21835)
2025-08-14 22:58:25 -07:00
bundler
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
cli
followup #21833 (bun audit more filtering options) (#21873)
2025-08-14 19:51:56 -07:00
codegen
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
collections
Improve deepClone methods (#21747)
2025-08-11 13:29:53 -07:00
create
zig: delete deprecated bun.jsc.Maybe (#21327)
2025-07-25 12:38:06 -07:00
css
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
deps
Fix abort handler in "ws" polyfill (#21867)
2025-08-14 16:00:03 -07:00
errno
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
fs
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
glob
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
http
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
init
bun bake templates: --source-map -> --sourcemap (#21130)
2025-07-17 03:13:48 -07:00
install
followup #21833 (bun audit more filtering options) (#21873)
2025-08-14 19:51:56 -07:00
interchange
Split JS parser into multiple files (#20880)
2025-08-05 20:52:16 -07:00
io
Fix pipeline stack errors on Windows (#21800)
2025-08-14 18:03:26 -07:00
js
Introduce Bun.stripANSI (#21801)
2025-08-14 22:42:05 -07:00
js_lexer
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
libarchive
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
napi
NAPI fixes (#21775)
2025-08-12 19:59:34 -07:00
node-fallbacks
node-fallbacks:buffer: fix numberIsNaN ReferenceError (#21527)
2025-07-31 22:07:17 -07:00
paths
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
ptr
Add owned pointer types (#21776)
2025-08-12 22:25:49 -07:00
resolver
Implement wildcard sideEffects support using glob API (#21039)
2025-08-14 11:58:37 -07:00
s3
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
safety
Various safety improvements (safety.ThreadLock, stack traces, MimallocArena, RefCount, safety.alloc) (#21726)
2025-08-11 13:40:07 -07:00
semver
zig: delete deprecated some bun.jsc apis (#21309)
2025-07-23 17:10:58 -07:00
shell
Fix pipeline stack errors on Windows (#21800)
2025-08-14 18:03:26 -07:00
sourcemap
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
sql
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
string
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
sys
fix: main is not readonly in @types/node (#21612)
2025-08-04 13:07:42 -07:00
test
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
threading
Various safety improvements (safety.ThreadLock, stack traces, MimallocArena, RefCount, safety.alloc) (#21726)
2025-08-11 13:40:07 -07:00
valkey
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
vm
Add support for node:vm.SyntheticModule (#19878)
2025-06-04 19:41:26 -07:00
watcher
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
.clang-format
…
allocators.zig
Fix z_allocator implementation when use_mimalloc is false; make Bun compile with use_mimalloc false (#21771)
2025-08-11 20:20:58 -07:00
analytics.zig
Give us way to know if crashing inside exit handler (#21401)
2025-07-27 16:55:55 -07:00
asan-config.c
Fix Linux ASan debug builds (#19657)
2025-05-14 18:42:42 -07:00
asan.zig
Add asan checks to HiveArray (#21449)
2025-07-29 19:35:46 -07:00
ast.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
bake.bind.ts
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
bake.zig
Fix potential crash in new Bun.Transpiler() (#21650)
2025-08-07 15:16:44 -07:00
bits.zig
remove all usingnamespace in css (#19067)
2025-04-17 14:17:08 -07:00
boringssl.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
brotli.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
btjs.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
bun.exe.manifest
Enable Windows long path support (#21244)
2025-07-20 23:04:17 -07:00
bun.ico
…
bun.js.zig
Various safety improvements (safety.ThreadLock, stack traces, MimallocArena, RefCount, safety.alloc) (#21726)
2025-08-11 13:40:07 -07:00
bun.zig
Fix pipeline stack errors on Windows (#21800)
2025-08-14 18:03:26 -07:00
bunfig.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
c-headers-for-zig.h
node:zlib: add zstd (#20313)
2025-06-12 14:20:28 -07:00
cache.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
ci_info.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
cli.zig
implement bunx --package (#21517)
2025-08-12 17:07:46 -07:00
collections.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
compile_target.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
comptime_string_map.zig
Delete incorrect assertion in ComptimeStringMap (#21504)
2025-07-31 00:56:50 -07:00
copy_file.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
crash_handler.zig
Fix
2025-08-04 23:33:29 -07:00
csrf.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
css_scanner.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
darwin.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
defines-table.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
defines.zig
Fix potential crash in new Bun.Transpiler() (#21650)
2025-08-07 15:16:44 -07:00
dir.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
dns.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
env_loader.zig
Fix env loader buffer overflow by using stack fallback allocator (#21416)
2025-07-28 00:13:17 -07:00
env.zig
Various safety improvements (safety.ThreadLock, stack traces, MimallocArena, RefCount, safety.alloc) (#21726)
2025-08-11 13:40:07 -07:00
fallback-backend.html
Fix formatters not running in CI + delete unnecessary files (#19433)
2025-05-08 23:22:16 -07:00
fallback.html
Fix formatters not running in CI + delete unnecessary files (#19433)
2025-05-08 23:22:16 -07:00
fallback.ts
…
favicon.png
…
fd.zig
Fix pipeline stack errors on Windows (#21800)
2025-08-14 18:03:26 -07:00
feature_flags.zig
[STAB-861] Suppress known-benign core dumps in CI (#21321)
2025-07-25 16:22:04 -07:00
fixtures_example.com.html
Fix formatters not running in CI + delete unnecessary files (#19433)
2025-05-08 23:22:16 -07:00
fmt.bind.ts
…
fmt.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
fs.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
generated_perf_trace_events.zig
Use macOS signpost api for tracing (#14871)
2025-03-31 04:13:11 -07:00
generated_versions_list.zig
…
glob.zig
Revert "add support for "workspaces.nohoist" and "workspaces.hoistingLimits" (#20124)"
2025-06-03 23:51:03 -07:00
Global.zig
Fix mimalloc memory usage regression (#21550)
2025-08-01 23:38:34 -07:00
heap_breakdown.zig
Add safety checks to MultiArrayList and BabyList (#21063)
2025-07-25 18:12:21 -07:00
highway.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
hmac.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
HTMLScanner.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
http.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
identity_context.zig
implement "nodeLinker": "isolated" in bun install (#20440)
2025-07-09 00:19:57 -07:00
import_record.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
ini.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
interchange.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
js_lexer_tables.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
js_lexer.zig
Fix bounds check in lexer for single # character (#21341)
2025-07-25 13:59:33 -07:00
js_parser.zig
Split JS parser into multiple files (#20880)
2025-08-05 20:52:16 -07:00
js_printer.zig
Try mimalloc v3 (#17378)
2025-07-29 18:07:15 -07:00
jsc_stub.zig
…
linear_fifo.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
linker.lds
Better error message for NAPI modules which access unsupported libuv functions (#18503)
2025-03-26 23:57:10 -07:00
linker.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
linux.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
logger.zig
fix logger.zig from negative value error (#21876)
2025-08-14 21:12:22 -07:00
logo.svg
Add react, tailwind, react+tailwind+shadcn to bun init (#17282)
2025-02-18 10:38:37 -08:00
macho.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
main_test.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
main_wasm.zig
Various safety improvements (safety.ThreadLock, stack traces, MimallocArena, RefCount, safety.alloc) (#21726)
2025-08-11 13:40:07 -07:00
main.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
meta.zig
Add safety checks to MultiArrayList and BabyList (#21063)
2025-07-25 18:12:21 -07:00
node_fallbacks.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
open.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
options.zig
Fix potential crash in new Bun.Transpiler() (#21650)
2025-08-07 15:16:44 -07:00
output.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
OutputFile.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
patch.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
paths.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
pe.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
perf.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
pool.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
Progress.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
ptr.zig
Add owned pointer types (#21776)
2025-08-12 22:25:49 -07:00
renamer.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
result.zig
…
router.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
runtime.bun.js
…
runtime.js
Fix NODE_PATH for bun build (#20576)
2025-06-22 20:51:45 -07:00
runtime.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
safety.zig
Various safety improvements (safety.ThreadLock, stack traces, MimallocArena, RefCount, safety.alloc) (#21726)
2025-08-11 13:40:07 -07:00
semver.zig
Move semver-related structs into their own files (#16987)
2025-02-02 00:20:45 -08:00
sha.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
StandaloneModuleGraph.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
StaticHashMap.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
string.zig
Downgrade mimalloc + set libc musl flag (#21684)
2025-08-08 18:02:19 -07:00
symbols.def
Implement more v8::Array::New overloads, v8::Object::{Get,Set}, v8::Value::StrictEquals, v8::Array::Iterate (#20737)
2025-07-13 04:27:43 -07:00
symbols.dyn
Implement more v8::Array::New overloads, v8::Object::{Get,Set}, v8::Value::StrictEquals, v8::Array::Iterate (#20737)
2025-07-13 04:27:43 -07:00
symbols.txt
Implement more v8::Array::New overloads, v8::Object::{Get,Set}, v8::Value::StrictEquals, v8::Array::Iterate (#20737)
2025-07-13 04:27:43 -07:00
sys_uv.zig
zig: delete deprecated bun.jsc.Maybe (#21327)
2025-07-25 12:38:06 -07:00
sys.zig
Fix pipeline stack errors on Windows (#21800)
2025-08-14 18:03:26 -07:00
system_timer.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
threading.zig
Add edge deletion safety checks to DevServer and fix cases where it was caught (#21551)
2025-08-04 19:21:28 -07:00
tmp.zig
zig: delete deprecated bun.jsc.Maybe (#21327)
2025-07-25 12:38:06 -07:00
tracy.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
trait.zig
zig: update to 0.14.0-dev (#16862)
2025-02-01 01:11:02 -08:00
transpiler.zig
Fix env loader buffer overflow by using stack fallback allocator (#21416)
2025-07-28 00:13:17 -07:00
tsconfig.json
Fix formatters not running in CI + delete unnecessary files (#19433)
2025-05-08 23:22:16 -07:00
unit_test.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
url.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
util.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
walker_skippable.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
Watcher.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
which_npm_client.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
which.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
windows-app-info.rc
Enable Windows long path support (#21244)
2025-07-20 23:04:17 -07:00
windows.zig
Rename disabled parameter in Output.scoped (#21769)
2025-08-11 20:19:34 -07:00
work_pool.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
workaround_missing_symbols.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00
wyhash.zig
Sort imports in all files (#21119)
2025-07-21 13:26:47 -07:00
zlib.test.gz
…
zlib.test.txt
…
zlib.zig
Refactor Zig imports and file structure (part 1) (#21270)
2025-07-22 17:51:38 -07:00