Files
bun.sh/test/js/bun/shell/parse.test.ts
Zack Radisic 1b1760a9c9 feat: Bun shell (#7748)
* bring in shell impl

* add `$` to global bun scope

* Expose shell parse func on debug mode

* Expose lex tokens and add tests

* refactor parser to have better ast

* assigns and export

* pipeline kinda working

* Decouple Subprocess spawning code from JS stuff

* Subprocess works!

* Conditional execution

* Support JS objects in template expression

* More complete redirection

* Properly drain stdin/stdout/stderr and fix crash from deallocating JSC memory

* Return errors in parser

* Support command substitution

* wip brace expansion stuff

* Rearrange some files

* expansion wip

* Brace expansion working

* wip brace expansion

* refactor brace algorithm

* wip brace expansion on shell

* fix brace expansion

* Working nested brace expansion!

* brace expansion in shell variable assignment only set the last expanded

* stuff

* Small little perf things

* benchmark and test and stuff

* stuff

* fix nested braces but its also kinda broken

* attempt to fix complicated nested braces

* test

* Use fast tokenized algorithm for non nested braces, use parsed algorithm for nested braces

* fix nested braces one and for all

* small stuff

* Not sure if that made a difference

* revert that

* good speed optimization

* rip

* Environment variables, builtin/native shell cmds

* Fix tests

* Support `cd`, `pwd`, add boilerplate for glob expansion

* Support `which`

* Support `rm`

* wip

* wip

* escaping and abstract shell char iterator

* strpool unicode

* Brace expansion support unicode, disallow invalid surrogates in shell script

* shell choose ascii or unicode lexer depending on input

* fix bugs write tests

* kinda start async stuff

* HOLY SHIT big refactor of Subprocess

woops forget to commit this

...and this

* HOLY MOLY it works

* Refactor some stuff, start eval word expansion

* interpret all the nodes

* stuff

* stuff

* stuff

* kind of works but doesnt

* Buffered output works

* no need to heap allocate autosizer

* Fix bug

* Fix some stuff

* unprotect

* move out dummy shell thing

* Bring back assignments

* create expansion state so it can be non blocking for expansions that need IO (glob, cmd subst)

* glob back in action

* Setup builtin non blocking IO commands and implement export

big issue is control flow is really fcked up here need to fix that

* make Cmd state machine use a loop so control flow is a bit more clear

* rename stuff

* move that

* Implement the echo builtin again

* implement cd again but non blocking io

* Fix ls and use proper write function to prevent blockign writes

* Implement which

* holy moly big port std.fs.deleteTree

* fix compile errors

* Okay that works

* rm works thatsnice

* damn

* split it out

* rm async implementation

* fix rm bug for nested

* Work on files as well

* prevent root from being deleted

* rm error handling

* oops

* pwd and fix some script exec bug

* Implement `mv`

* stub out mv to work accross filesystems

* move it around

* woops

* boilerplate for ls and options

* more boilerplate

* stuff

* that got lost in merge

* upgrade shell stuff zig 0.12.0-dev.1828+225fe6ddb

* Implement basic ls

* smol cleanup

* Fix stream, response redirect stdin

* No longer need spawn to be abstract

* Custom promise

* move around some stuff

* shell promise returns shell output

* make tht work for builtins

* refactor IO abstractions to work with JS or mini event loop

* woops

* scaffolding for refactor

* refactor builtins to make event loop refactor easier

* Fix parsing edge case on assignments, fix expansion on cmd assignments

* change subproc to work with any event loop

* Finish refactoring subproc

* move global abstraction out

* big refactor boys

* holy moly: integrate into cli and fix allll the compile errors

* okay works in bun run now

* actually tick the event loop lol

* Fix more stuff

* Support comments

* Fix some tests

* delete that

* Properly report errors when failing to spawn command

* fix a whole bunch of tests

* fix a whole bunch of tests again

* .

* Fix rm

* Fix some exit code bugs, write force rm from deno, fix ls stderr

* fix `rm -d`

* fix `rm -d`

* Fix boolean logic

* error on subshells (e.g. `true && (echo hi && echo lol)`)

* Move out shell state from interpreter struct

* Cmd substitution supports arbitrary script, not just a single cmd

* Some escaping/quotation tests

* Fix stuff add more tests:

- cmd substitution quotations
- escape backticks in single quots

* ALOT of stuff:

- fix proper subshell inheritance of env for cmd subst
- fix: was wrong, assignments don't run in subshell in conditionals
- fix lexing chained vars `$VAR$VAR`
- more tests

* Fix subtle bugs

* Fix crazy redirect to arraybuffer bug

* more crazy echo edgecases

* Proper lexer errors instead of just panicking lol

* yoops

* Proper parsing errors

* Errors for bun run shell script

* Fix redirecting to file

* More test fix bugs yay

* Fix redirect on builtins

* Open redirection fds with O_TRUNC

* Fix lexing invalid variables and add ability to change cwd from JS api

* yoops

* Fix `.cwd()`

* `$PWD` and fix redirection bugs

* `$PWD` and fix redirection bugs

* Get rid of  some `FIXME`s

* throw errors in some places instead of panicking

* Print some errors to stderr

* Get rid of some more panics again

* Handle errors on glob

* pwd test

* `.env()`

* copy-on-write abstraction

* Reference counted env strings + fix some tests

* deinit cwd

* Put commands into a pipeline properly

* deinit Expansion and Assigns properly

* comments

* Comments

* Make it compile

* Update types

* [autofix.ci] apply automated fixes

* Only one WaiterThread

* Fix lifetimes and clean up interface

* Update shell.ts

* Add lazy test

* Remove some dead code

* Update shell.zig

* Fix memory leak

* Fix crash with empty braces

* [autofix.ci] apply automated fixes

* Linux build + bun.sh

* Update subproc.zig

* Update interpreter.zig

* Update interpreter.zig

* Fix some stuff that broke

* Fix Windows compile errors

* Fix some fd leaks

* Fix ls

* Fix a bunch of stuff

* Fix quiet

* Update leak tests fix rm bug

* More reproducible tests

* [autofix.ci] apply automated fixes

* more mem leak tests

* [autofix.ci] apply automated fixes

* Fix merge conflict

* Fix test not actually using temp directory

* Update bunshell.test.ts

* Shell instance

* Capture async context

* Increase test timeouts

* [autofix.ci] apply automated fixes

* Escape

* [autofix.ci] apply automated fixes

* Fix crash

* Add more methods

* [autofix.ci] apply automated fixes

* Fix leak

* Treat file(path) blobs as a file path string

* Create bunshell-file.test.ts

* Support Blob input

* Fix leak + organize imports

* doc

* Update shell.md

* Update shell.md

* Update shell.md

* Update shell.md

* Update CMakeLists.txt

* Fix segfault by cloning error path so it's not freed by arena

* deinit ShellErr

* Delete dead code

* fix really stupid segfault

* don't deinit shell ls task in event loop

* Fix ls bug

* Fix tests

* make truly lazy

* allow more things in the shell substitution and escape whitespace

* Fix newline and exit when finishing shell in `bun run`

---------

Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-01-19 17:30:57 -08:00

509 lines
16 KiB
TypeScript

import { $ } from "bun";
import { TestBuilder, redirect } from "./util";
const BUN = process.argv0;
describe("parse shell", () => {
test("basic", () => {
const expected = {
stmts: [
{
exprs: [
{
cmd: {
assigns: [],
name_and_args: [
{
simple: {
Text: "echo",
},
},
{
simple: {
Text: "foo",
},
},
],
redirect: redirect({}),
redirect_file: null,
},
},
],
},
],
};
const result = $.parse`echo foo`;
expect(JSON.parse(result)).toEqual(expected);
});
test("basic redirect", () => {
const expected = {
"stmts": [
{
"exprs": [
{
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "foo" } }],
"redirect": redirect({ stdout: true }),
"redirect_file": { atom: { "simple": { "Text": "lmao.txt" } } },
},
},
],
},
],
};
const result = JSON.parse($.parse`echo foo > lmao.txt`);
expect(result).toEqual(expected);
});
test("compound atom", () => {
const expected = {
"stmts": [
{
"exprs": [
{
"cmd": {
"assigns": [],
"name_and_args": [
{
"compound": {
"atoms": [{ "Text": "FOO " }, { "Var": "NICE" }, { "Text": "!" }],
brace_expansion_hint: false,
glob_hint: false,
},
},
],
"redirect": redirect({}),
"redirect_file": null,
},
},
],
},
],
};
const result = JSON.parse($.parse`"FOO $NICE!"`);
console.log("Result", JSON.stringify(result));
expect(result).toEqual(expected);
});
test("pipelines", () => {
const expected = {
"stmts": [
{
"exprs": [
{
"pipeline": {
"items": [
{
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }],
"redirect": redirect({ stdout: true }),
"redirect_file": { atom: { "simple": { "Text": "foo.txt" } } },
},
},
{
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "hi" } }],
"redirect": redirect({}),
"redirect_file": null,
},
},
],
},
},
],
},
],
};
const result = JSON.parse($.parse`echo > foo.txt | echo hi`);
// console.log(result);
expect(result).toEqual(expected);
});
test("conditional execution", () => {
const expected = {
"stmts": [
{
"exprs": [
{
"cond": {
"op": "Or",
"left": {
"cond": {
"op": "And",
"left": {
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "foo" } }],
"redirect": redirect(),
"redirect_file": null,
},
},
"right": {
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "bar" } }],
"redirect": redirect(),
"redirect_file": null,
},
},
},
},
"right": {
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "lmao" } }],
"redirect": redirect(),
"redirect_file": null,
},
},
},
},
],
},
],
};
const result = JSON.parse($.parse`echo foo && echo bar || echo lmao`);
// console.log(result);
expect(result).toEqual(expected);
});
test("precedence", () => {
const expected = {
"stmts": [
{
"exprs": [
{
"cond": {
"op": "And",
"left": {
"cond": {
"op": "And",
"left": {
"assign": [{ "label": "FOO", "value": { "simple": { "Text": "bar" } } }],
},
"right": {
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "foo" } }],
"redirect": redirect(),
"redirect_file": null,
},
},
},
},
"right": {
"pipeline": {
"items": [
{
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "bar" } }],
"redirect": redirect(),
"redirect_file": null,
},
},
{
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "lmao" } }],
"redirect": redirect(),
"redirect_file": null,
},
},
{
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "cat" } }],
"redirect": redirect({ stdout: true }),
"redirect_file": { atom: { "simple": { "Text": "foo.txt" } } },
},
},
],
},
},
},
},
],
},
],
};
const result = $.parse`FOO=bar && echo foo && echo bar | echo lmao | cat > foo.txt`;
// console.log(result);
expect(JSON.parse(result)).toEqual(expected);
});
test("assigns", () => {
const expected = {
"stmts": [
{
"exprs": [
{
"cmd": {
"assigns": [
{ "label": "FOO", "value": { "simple": { "Text": "bar" } } },
{ "label": "BAR", "value": { "simple": { "Text": "baz" } } },
],
"name_and_args": [{ "simple": { "Text": "export" } }, { "simple": { "Text": "LMAO=nice" } }],
"redirect": {
"stdin": false,
"stdout": false,
"stderr": false,
"append": false,
"__unused": 0,
},
"redirect_file": null,
},
},
],
},
],
};
const result = JSON.parse($.parse`FOO=bar BAR=baz export LMAO=nice`);
console.log("Result", JSON.stringify(result));
expect(result).toEqual(expected);
});
test("redirect js obj", () => {
const expected = {
"stmts": [
{
"exprs": [
{
"cond": {
"op": "And",
"left": {
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "foo" } }],
"redirect": redirect({ stdout: true }),
"redirect_file": { "jsbuf": { "idx": 0 } },
},
},
"right": {
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "foo" } }],
"redirect": redirect({ stdout: true }),
"redirect_file": { "jsbuf": { "idx": 1 } },
},
},
},
},
],
},
],
};
const buffer = new Uint8Array(1 << 20);
const buffer2 = new Uint8Array(1 << 20);
const result = JSON.parse($.parse`echo foo > ${buffer} && echo foo > ${buffer2}`);
// console.log("Result", JSON.stringify(result));
expect(result).toEqual(expected);
});
test("cmd subst", () => {
const expected = {
"stmts": [
{
"exprs": [
{
"cmd": {
"assigns": [],
"name_and_args": [
{ "simple": { "Text": "echo" } },
{
"simple": {
"cmd_subst": {
"script": {
"stmts": [
{
"exprs": [
{
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "1" } }],
"redirect": {
"stdin": false,
"stdout": false,
"stderr": false,
"append": false,
"__unused": 0,
},
"redirect_file": null,
},
},
],
},
{
"exprs": [
{
"cmd": {
"assigns": [],
"name_and_args": [{ "simple": { "Text": "echo" } }, { "simple": { "Text": "2" } }],
"redirect": {
"stdin": false,
"stdout": false,
"stderr": false,
"append": false,
"__unused": 0,
},
"redirect_file": null,
},
},
],
},
],
},
"quoted": true,
},
},
},
],
"redirect": {
"stdin": false,
"stdout": false,
"stderr": false,
"append": false,
"__unused": 0,
},
"redirect_file": null,
},
},
],
},
],
};
const result = JSON.parse($.parse`echo "$(echo 1; echo 2)"`);
expect(result).toEqual(expected);
});
describe("bad syntax", () => {
test("cmd subst edgecase", () => {
const expected = {
"stmts": [
{
"exprs": [
{
"cmd": {
"assigns": [],
"name_and_args": [
{ "simple": { "Text": "echo" } },
{
"simple": {
"cmd_subst": {
"script": {
"stmts": [
{
"exprs": [
{
"cmd": {
"assigns": [
{
"label": "FOO",
"value": { "simple": { "Text": "bar" } },
},
],
"name_and_args": [{ "simple": { "Var": "FOO" } }],
"redirect": {
"stdin": false,
"stdout": false,
"stderr": false,
"append": false,
"__unused": 0,
},
"redirect_file": null,
},
},
],
},
],
},
"quoted": false,
},
},
},
],
"redirect": {
"stdin": false,
"stdout": false,
"stderr": false,
"append": false,
"__unused": 0,
},
"redirect_file": null,
},
},
],
},
],
};
const result = JSON.parse($.parse`echo $(FOO=bar $FOO)`);
expect(result).toEqual(expected);
});
test("cmd edgecase", () => {
const expected = {
"stmts": [
{
"exprs": [
{
"assign": [
{ "label": "FOO", "value": { "simple": { "Text": "bar" } } },
{ "label": "BAR", "value": { "simple": { "Text": "baz" } } },
],
},
{
"cmd": {
"assigns": [
{
"label": "BUN_DEBUG_QUIET_LOGS",
"value": { "simple": { "Text": "1" } },
},
],
"name_and_args": [{ "simple": { "Text": "echo" } }],
"redirect": {
"stdin": false,
"stdout": false,
"stderr": false,
"append": false,
"__unused": 0,
},
"redirect_file": null,
},
},
],
},
],
};
const result = JSON.parse($.parse`FOO=bar BAR=baz; BUN_DEBUG_QUIET_LOGS=1 echo`);
expect(result).toEqual(expected);
});
});
});
describe("parse shell invalid input", () => {
test("invalid js obj", async () => {
const file = new Uint8Array(420);
await TestBuilder.command`${file} | cat`.error(`expected a command or assignment but got: "JSObjRef"`).run();
});
test("subshell", async () => {
await TestBuilder.command`echo (echo foo && echo hi)`
.error("Unexpected `(`, subshells are currently not supported right now. Escape the `(` or open a GitHub issue.")
.run();
await TestBuilder.command`echo foo >`.error("Redirection with no file").run();
});
});