Files
bun.sh/test/regression/issue/23314/zstd-async-compress.test.ts
robobun d273f7fdde fix: zstd decompression with async-compressed data (#23314) (#23317)
### What does this PR do?

Fixes #23314 where `zlib.zstdCompress()` created data that caused an
out-of-memory error when decompressed with `Bun.zstdDecompressSync()`.

#### 1. `zlib.zstdCompress()` now sets `pledgedSrcSize`

The async convenience method now automatically sets the `pledgedSrcSize`
option to the input buffer size. This ensures the compressed frame
includes the content size in the header, making sync and async
compression produce identical output.

**Node.js compatibility**: `pledgedSrcSize` is a documented Node.js
option:
-
[`vendor/node/doc/api/zlib.md:754-758`](https://github.com/oven-sh/bun/blob/main/vendor/node/doc/api/zlib.md#L754-L758)
-
[`vendor/node/lib/zlib.js:893`](https://github.com/oven-sh/bun/blob/main/vendor/node/lib/zlib.js#L893)
-
[`vendor/node/src/node_zlib.cc:890-904`](https://github.com/oven-sh/bun/blob/main/vendor/node/src/node_zlib.cc#L890-L904)

#### 2. Added `bun.zstd.decompressAlloc()` - centralized safe
decompression

Created a new function in `src/deps/zstd.zig` that handles decompression
in one place with automatic safety features:

- **Handles unknown content sizes**: Automatically switches to streaming
decompression when the zstd frame doesn't include content size (e.g.,
from streams without `pledgedSrcSize`)
- **16MB safety limit**: For security, if the reported decompressed size
exceeds 16MB, streaming decompression is used instead of blindly
trusting the header
- **Fast path for small files**: Still uses efficient pre-allocation for
files < 16MB with known sizes

This centralized fix automatically protects:
- `Bun.zstdDecompressSync()` / `Bun.zstdDecompress()`
- `StandaloneModuleGraph` source map decompression
- Any other code using `bun.zstd` decompression

### How did you verify your code works?

**Before:**
```typescript
const input = "hello world";

// Async compression
const compressed = await new Promise((resolve, reject) => {
  zlib.zstdCompress(input, (err, result) => {
    if (err) reject(err);
    else resolve(result);
  });
});

// This would fail with "Out of memory"
const decompressed = Bun.zstdDecompressSync(compressed);
```
**Error**: `RangeError: Out of memory` (tried to allocate UINT64_MAX
bytes)

**After:**
```typescript
const input = "hello world";

// Async compression (now includes content size)
const compressed = await new Promise((resolve, reject) => {
  zlib.zstdCompress(input, (err, result) => {
    if (err) reject(err);
    else resolve(result);
  });
});

//  Works! Falls back to streaming decompression if needed
const decompressed = Bun.zstdDecompressSync(compressed);
console.log(decompressed.toString()); // "hello world"
```

**Tests:**
-  All existing tests pass
-  New regression tests for async/sync compression compatibility
(`test/regression/issue/23314/zstd-async-compress.test.ts`)
-  Test for large (>16MB) decompression using streaming
(`test/regression/issue/23314/zstd-large-decompression.test.ts`)
-  Test for various input sizes and types
(`test/regression/issue/23314/zstd-large-input.test.ts`)

**Security:**
The 16MB safety limit protects against malicious zstd frames that claim
huge decompressed sizes in the header, preventing potential OOM attacks.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

Co-authored-by: Claude Bot <claude-bot@bun.sh>
Co-authored-by: Claude <noreply@anthropic.com>
2025-10-06 19:56:40 -07:00

38 lines
1.5 KiB
TypeScript

import { describe, expect, it } from "bun:test";
import zlib from "node:zlib";
// The zlib sync and async implementations create different outputs
// This may not be a bug in itself, but the async version creates data that causes an out of memory error when decompressed with Bun.zstdDecompressSync
describe("zstd compression compatibility", () => {
it("should decompress data compressed with zlib.zstdCompressSync", () => {
const input = "hello world";
const compressed = zlib.zstdCompressSync(input);
const decompressed = Bun.zstdDecompressSync(compressed);
expect(decompressed.toString()).toBe(input);
});
it("should decompress data compressed with zlib.zstdCompress (async)", async () => {
const input = "hello world";
const compressed = await new Promise<Buffer>((resolve, reject) => {
zlib.zstdCompress(input, (err, result) => {
if (err) reject(err);
else resolve(result);
});
});
const decompressed = Bun.zstdDecompressSync(compressed);
expect(decompressed.toString()).toBe(input);
});
it("should decompress data compressed with zlib.zstdCompress using Bun.zstdDecompress", async () => {
const input = "hello world";
const compressed = await new Promise<Buffer>((resolve, reject) => {
zlib.zstdCompress(input, (err, result) => {
if (err) reject(err);
else resolve(result);
});
});
const decompressed = await Bun.zstdDecompress(compressed);
expect(decompressed.toString()).toBe(input);
});
});