Commit Graph

32 Commits

Author SHA1 Message Date
Jarred Sumner
b6b3626c14 fix(bindings): handle errors from String.toJS() for oversized strings (#26213)
## Summary

- When a string exceeds `WTF::String::MaxLength` (~4GB),
`bun.String.createUninitialized()` returns a `.Dead` tag
- The C++ layer now properly throws `ERR_STRING_TOO_LONG` when this
happens
- Updated `String.toJS()` in Zig to return `bun.JSError!jsc.JSValue`
instead of just `jsc.JSValue`
- Updated ~40 Zig caller files to handle the error with `try`
- C++ callers updated with `RETURN_IF_EXCEPTION` checks

## Test plan

- [x] `bun bd test test/js/node/buffer.test.js` - 449 tests pass
- [x] `bun bd
test/js/node/test/parallel/test-buffer-tostring-rangeerror.js` - passes

🤖 Generated with [Claude Code](https://claude.ai/code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Bot <claude-bot@bun.sh>
2026-01-21 13:01:25 -08:00
Ciro Spaciari
5d3f37d7ae feat(s3): add Content-Encoding header support for S3 uploads (#26149)
## Summary

Add support for setting the `Content-Encoding` header in S3 `.write()`
and `.writer()` calls, following the same pattern as
`Content-Disposition`.

This allows users to specify the encoding of uploaded content:

```typescript
// With .write()
await s3file.write("compressed data", { contentEncoding: "gzip" });

// With .writer()
const writer = s3file.writer({ contentEncoding: "gzip" });
writer.write("compressed data");
await writer.end();

// With bucket.write()
await bucket.write("key", data, { contentEncoding: "br" });
```

## Implementation

- Extended `SignedHeaders.Key` from 6 bits to 7 bits (64→128
combinations) to accommodate the new header
- Added `content_encoding` to `S3CredentialsWithOptions`, `SignOptions`,
and `SignResult` structs
- Updated `CanonicalRequest` format strings to include
`content-encoding` in AWS SigV4 signing
- Added `getContentEncoding()` method to `Headers` for fetch-based S3
uploads
- Expanded `_headers` array from 9 to 10 elements
- Pass `content_encoding` through all S3 upload paths (upload,
uploadStream, writableStream)

## Test plan

- Added tests for "should be able to set content-encoding" 
- Added tests for "should be able to set content-encoding in writer"
- Tests verify the Content-Encoding header is properly set on uploaded
objects via presigned URL fetch
- All 4 new tests pass with `bun bd test` and fail with
`USE_SYSTEM_BUN=1` (confirming the feature is new)

## Changelog

> Describe your changes in 1-2 sentences. These will be featured on
[bun.sh/blog](https://bun.sh/blog) and Bun's release notes.

Added `contentEncoding` option to S3 `.write()` and `.writer()` methods,
allowing users to set the `Content-Encoding` header when uploading
objects.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-15 18:09:33 -08:00
Dylan Conway
959169dfaf feat(archive): change API to constructor-based with S3 support (#25940)
## Summary
- Change Archive API from `Bun.Archive.from(data)` to `new
Bun.Archive(data, options?)`
- Change compression options from `{ gzip: true }` to `{ compress:
"gzip", level?: number }`
- Default to no compression when no options provided
- Use `{ compress: "gzip" }` to enable gzip compression (level 6 by
default)
- Add Archive support for S3 and local file writes via `Bun.write()`

## New API

```typescript
// Create archive - defaults to uncompressed tar
const archive = new Bun.Archive({
  "hello.txt": "Hello, World!",
  "data.json": JSON.stringify({ foo: "bar" }),
});

// Enable gzip compression
const compressed = new Bun.Archive(files, { compress: "gzip" });

// Gzip with custom level (1-12)
const maxCompression = new Bun.Archive(files, { compress: "gzip", level: 12 });

// Write to local file
await Bun.write("archive.tar", archive);           // uncompressed by default
await Bun.write("archive.tar.gz", compressed);     // gzipped

// Write to S3
await client.write("archive.tar.gz", compressed);          // S3Client.write()
await Bun.write("s3://bucket/archive.tar.gz", compressed); // S3 URL
await s3File.write(compressed);                            // s3File.write()

// Get bytes/blob (uses compression setting from constructor)
const bytes = await archive.bytes();
const blob = await archive.blob();
```

## TypeScript Types

```typescript
type ArchiveCompression = "gzip";

type ArchiveOptions = {
  compress?: "gzip";
  level?: number;  // 1-12, default 6 when gzip enabled
};
```

## Test plan
- [x] 98 archive tests pass
- [x] S3 integration tests updated to new API
- [x] TypeScript types updated
- [x] Documentation updated with new examples

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
2026-01-12 14:54:21 -08:00
Dylan Conway
596e83c918 fix: correct logic bugs in libarchive, s3 credentials, and postgres bindings (#25905)
## Summary

- **libarchive.zig:110**: Fix self-assignment bug where `this.pos` was
assigned to itself instead of `new_pos`
- **s3/credentials.zig:165,176,199**: Fix impossible range checks -
`and` should be `or` for pageSize, partSize, and retry validation (a
value cannot be both less than MIN and greater than MAX simultaneously)
- **postgres.zig:14**: Fix copy-paste error where createConnection
function was internally named "createQuery"

## Test plan

- [ ] Verify S3 credential validation now properly rejects out-of-range
values for pageSize, partSize, and retry
- [ ] Verify libarchive seek operations work correctly
- [ ] Verify postgres createConnection function has correct internal
name

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
Co-authored-by: Claude Bot <claude-bot@bun.sh>
2026-01-08 19:46:06 -08:00
Prithvish Baidya
9ab6365a13 Add support for Requester Pays in S3 operations (#25514)
- Introduced `requestPayer` option in S3-related functions and
structures to handle Requester Pays buckets.
- Updated S3 client methods to accept and propagate the `requestPayer`
flag.
- Enhanced documentation for the `requestPayer` option in the S3 type
definitions.
- Adjusted existing S3 operations to utilize the `requestPayer`
parameter where applicable, ensuring compatibility with AWS S3's
Requester Pays feature.
- Ensured that the new functionality is integrated into multipart
uploads and simple requests.

### What does this PR do?

This change allows users to specify whether they are willing to pay for
data transfer costs when accessing objects in Requester Pays buckets,
improving flexibility and compliance with AWS S3's billing model.

This closes #25499

### How did you verify your code works?

I have added a new test file to verify this functionality, and all my
tests pass.
I also tested this against an actual S3 bucket which can only be
accessed if requester pays. I can confirm that it's accessible with
`requestPayer` is `true`, and the default of `false` does not allow
access.

An example bucket is here: s3://hl-mainnet-evm-blocks/0/0/1.rmp.lz4
(my usecase is indexing [hyperliquid block
data](https://hyperliquid.gitbook.io/hyperliquid-docs/for-developers/hyperevm/raw-hyperevm-block-data)
which is stored in s3, and I want to use bun to index faster)

---------

Co-authored-by: Alistair Smith <hi@alistair.sh>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Ciro Spaciari <ciro.spaciari@gmail.com>
2026-01-05 15:04:20 -08:00
Ciro Spaciari
f25ea59683 feat(s3): add Content-Disposition support for S3 uploads (#25363)
### What does this PR do?
- Add `contentDisposition` option to S3 file uploads to control the
`Content-Disposition` HTTP header
- Support passing `contentDisposition` through all S3 upload paths
(simple uploads, multipart uploads, and streaming uploads)
- Add TypeScript types for the new option
Fixes https://github.com/oven-sh/bun/issues/25362
### How did you verify your code works?
Test
2025-12-08 15:30:20 -08:00
Meghan Denny
51ce3bc269 [publish images] ci: ensure tests that require docker have it available (#22781) 2025-09-25 19:03:22 -07:00
Jarred Sumner
4dfd87a302 Fix aborting fetch() calls while the socket is connecting. Fix a thread-safety issue involving redirects and AbortSignal. (#22842)
### What does this PR do?

When we added "happy eyeballs" support to fetch(), it meant that
`onOpen` would not be called potentially for awhile. If the AbortSignal
is aborted between `connect()` and the socket becoming
readable/writable, then we would delay closing the connection until the
connection opens. Fixing that fixes #18536.

Separately, the `isHTTPS()` function used in abort and in request body
streams was not thread safe. This caused a crash when many redirects
happen simultaneously while either AbortSignal or request body messages
are in-flight.
This PR fixes https://github.com/oven-sh/bun/issues/14137



### How did you verify your code works?

There are tests

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Bot <claude-bot@bun.sh>
Co-authored-by: Ciro Spaciari <ciro.spaciari@gmail.com>
2025-09-25 16:08:06 -07:00
Jarred Sumner
0b549321e9 Start using test.concurrent in our tests (#22823)
### What does this PR do?

### How did you verify your code works?

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Bot <claude-bot@bun.sh>
2025-09-22 05:30:34 -07:00
robobun
f8aed4826b Migrate all Docker usage to unified docker-compose infrastructure (#22740)
## Summary

This PR migrates all Docker container usage in tests from individual
`docker run` commands to a centralized Docker Compose setup. This makes
tests run **10x faster**, eliminates port conflicts, and provides a much
better developer experience.

## What is Docker Compose?

Docker Compose is a tool for defining and running multi-container Docker
applications. Instead of each test file managing its own containers with
complex `docker run` commands, we define all services once in a YAML
file and Docker Compose handles the orchestration.

## The Problem (Before)

```javascript
// Each test file managed its own container
const container = await Bun.spawn({
  cmd: ["docker", "run", "-d", "-p", "0:5432", "postgres:15"],
  // ... complex setup
});
```

**Issues:**
- Each test started its own container (30+ seconds for PostgreSQL tests)
- Containers were killed after each test (wasteful!)
- Random port conflicts between tests
- No coordination between test suites
- Docker configuration scattered across dozens of test files

## The Solution (After)

```javascript
// All tests share managed containers
const pg = await dockerCompose.ensure("postgres_plain");
// Container starts only if needed, returns connection info
```

**Benefits:**
- Containers start once and stay running (3 seconds for PostgreSQL tests
- **10x faster!**)
- Automatic port management (no conflicts)
- All services defined in one place
- Lazy loading (services only start when needed)
- Same setup locally and in CI

## What Changed

### New Infrastructure
- `test/docker/docker-compose.yml` - Defines all test services
- `test/docker/index.ts` - TypeScript API for managing services  
- `test/docker/README.md` - Comprehensive documentation
- Configuration files and init scripts for services

### Services Migrated

| Service | Status | Tests |
|---------|--------|--------|
| PostgreSQL (plain, TLS, auth) |  | All passing |
| MySQL (plain, native_password, TLS) |  | All passing |
| S3/MinIO |  | 276 passing |
| Redis/Valkey |  | 25/26 passing* |
| Autobahn WebSocket |  | 517 available |

*One Redis test was already broken before migration (reconnection test
times out)

### Key Features

- **Dynamic Ports**: Docker assigns available ports automatically (no
conflicts!)
- **Unix Sockets**: Proxy support for PostgreSQL and Redis Unix domain
sockets
- **Persistent Data**: Volumes for services that need data to survive
restarts
- **Health Checks**: Proper readiness detection for all services
- **Backward Compatible**: Fallback to old Docker method if needed

## Performance Improvements

| Test Suite | Before | After | Improvement |
|------------|--------|-------|-------------|
| PostgreSQL | ~30s | ~3s | **10x faster** |
| MySQL | ~25s | ~3s | **8x faster** |
| Redis | ~20s | ~2s | **10x faster** |

The improvements come from container reuse - containers start once and
stay running instead of starting/stopping for each test.

## How to Use

```typescript
import * as dockerCompose from "../../docker/index.ts";

test("database test", async () => {
  // Ensure service is running (starts if needed)
  const pg = await dockerCompose.ensure("postgres_plain");
  
  // Connect using provided info
  const client = new PostgresClient({
    host: pg.host,
    port: pg.ports[5432],  // Mapped to random available port
  });
});
```

## Testing

All affected test suites have been run and verified:
- `bun test test/js/sql/sql.test.ts` 
- `bun test test/js/sql/sql-mysql*.test.ts` 
- `bun test test/js/bun/s3/s3.test.ts` 
- `bun test test/js/valkey/valkey.test.ts` 
- `bun test test/js/web/websocket/autobahn.test.ts` 

## Documentation

Comprehensive documentation added in `test/docker/README.md` including:
- Detailed explanation of Docker Compose for beginners
- Architecture overview
- Usage examples
- Debugging guide
- Migration guide for adding new services

## Notes

- The Redis reconnection test that's skipped was already broken before
this migration. It's a pre-existing issue with the Redis client's
reconnection logic, not related to Docker changes.
- All tests that were passing before continue to pass after migration.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <claude@anthropic.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
2025-09-19 04:20:58 -07:00
Ciro Spaciari
e4dd780c2a fix(s3) ref before calling unlink (#21317)
Co-authored-by: Meghan Denny <meghan@bun.sh>
2025-07-23 22:23:58 -07:00
Ciro Spaciari
93cc51c974 fix(s3) handle slashs at end and start of bucket name (#20997) 2025-07-15 16:52:47 -07:00
Ciro Spaciari
75902e6a21 fix(s3) fix loading http endpoint from env (#20876) 2025-07-07 19:24:32 -07:00
Ciro Spaciari
964f2a8941 fix(fetch/s3) Handle backpressure when upload large bodies (#20481)
Co-authored-by: cirospaciari <6379399+cirospaciari@users.noreply.github.com>
2025-06-27 20:52:46 -07:00
Jarred Sumner
14b439a115 Fix formatters not running in CI + delete unnecessary files (#19433) 2025-05-08 23:22:16 -07:00
Inqnuam
946f41c01a feat(s3Client): add support for ListObjectsV2 action (#16948) 2025-04-02 09:35:08 -07:00
Alistair Smith
9141337c7d Fix some issues in Bun types (#17424)
Co-authored-by: Michael H <git@riskymh.dev>
2025-03-03 16:04:12 -08:00
Ciro Spaciari
79afefa488 fix(s3) Support Virtual Hosted-Style endpoints (#17292)
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
2025-02-13 19:47:53 -08:00
Ciro Spaciari
25f6cbd471 fix(s3) fix queue and multipart flow (#16890)
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
2025-01-31 05:19:23 -08:00
Jarred Sumner
75a95aa5fa Rewrite the internal Web Stream native bindings to use less memory (#16349) 2025-01-26 00:04:39 -08:00
Inqnuam
98c7b8452d feat(s3Client): add support for AWS S3 Object Storage Class (#16617) 2025-01-23 05:11:41 -08:00
Ciro Spaciari
5d98e64fd6 make s3 a default client (#16574) 2025-01-21 22:08:52 -08:00
Ciro Spaciari
e0e4a270a8 fix(S3) if the part is too small and is not ended wait for more data before sending (#16453) 2025-01-18 10:47:53 -08:00
Ciro Spaciari
6cdcb1c867 fix(test) enable minio in s3.test.ts (#16298) 2025-01-16 18:39:35 -08:00
Ciro Spaciari
8c04818a80 fix(s3) fix slice with offset 0 (#16400) 2025-01-14 22:38:15 -08:00
Ciro Spaciari
81fce29fd9 S3: refactor + S3Client static method (#16198) 2025-01-06 23:52:19 -08:00
Ciro Spaciari
034f776047 WIP: S3 improvements (#16167) 2025-01-04 19:57:35 -08:00
Jarred Sumner
8a469cce7e Default to "auto" instead of "us-east-1" 2025-01-04 06:21:32 -08:00
Jarred Sumner
5fe9b6f426 Improve MinIO support in Bun.S3 2025-01-04 02:44:17 -08:00
Ciro Spaciari
a53f2e6aaa fix test on windows (#16151) 2025-01-04 01:22:48 -08:00
Jarred Sumner
d5fc928ca8 S3 cleanup (#16039)
Co-authored-by: Ciro Spaciari <ciro.spaciari@gmail.com>
2025-01-03 19:11:48 -08:00
Ciro Spaciari
fe4176e403 feat(s3) s3 client (#15740)
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
Co-authored-by: cirospaciari <cirospaciari@users.noreply.github.com>
2024-12-28 17:46:22 -08:00