The Command Line

The 30-second version

You've probably typed npm install or go run . into a black window. That window is the TerminalTerminal (tty)The window you type commands into and read their output from. A program running in it reads your keystrokes as its input and its output appears as text. (Under the hood the OS presents it as a special file such as /dev/pts/0.), and the program reading what you type is the ShellShellThe program that reads the commands you type in a terminal (bash, zsh, sh), starts other programs and wires up their input and output. (usually bash on Linux, zsh on macOS). They're two different programs. The terminal draws text and sends your keystrokes; the shell understands them.

Every program the shell starts has the same small contract with the outside world. Learn these three things and almost every command line, script and CI log starts to make sense:

Here's what that buys you. A web server writes one line per request into access.log: the method, the page's path and the status code (200 means OK; other numbers are errors):

$ head -3 access.log
GET /home 200
GET /login 200
POST /login 401

This one line finds which pages are failing most, using seven tiny commands that each do one thing:

$ cat access.log | grep -v ' 200$' | cut -d' ' -f2 | sort | uniq -c | sort -rn | head -3
      2 /login
      2 /cart
      1 /about

Each | is a PipePipe ( | )Connects one program's standard output to the next program's standard input, so small tools can be chained: cat log | grep ERROR | wc -l.: it connects one program's output to the next program's input. None of these programs knows the others exist. By §06 you'll be able to read that line piece by piece, and build your own.

How the shell finds and runs a command

The parts of a command

$ wc -l notes.txt
3 notes.txt

Three words. The first, wc ("word count"), is the command: the program to run. The rest are its arguments. Arguments that start with a dash, like -l ("count lines only"), are by convention options or flagsOption / flagAn argument that changes how a program behaves, usually starting with a dash: -l (short form) or --lines (long form). Some take a value: -n 3 or --lines=3.. Most tools accept a short form (-l) and a long form (--lines), and some options take a value (head -n 3 or head --lines=3).

Step through what the shell does with a line like this. ($file is a variable holding notes.txt; §03 covers variables.)

1forever:
2  line = read a line from the terminal
3  words = expand and split(line)
4  path = look up words[0] in PATH
5  child = fork()             # clone the shell
6  in child: exec(path, words, env)
7  code = wait(child)         # becomes $?
8  print the prompt again

Two words in that walkthrough are worth a closer look. forkfork()The Unix system call that clones the calling process. The copy (the child) gets a new PID and a copy of the parent's memory and open files. It returns twice: 0 in the child, the child's PID in the parent. means the shell makes a copy of itself, a new ProcessProcessOne running program (e.g. one Node server or one Go binary). It has its own memory and one or more threads. (a running program). execexec()The Unix system call that replaces the program running in the current process with a different program. The PID stays the same; the memory is thrown away and the new program is loaded. means that copy swaps its program for a different one, wc here, while keeping the same process ID. §11 looks at both in detail; for now, "fork + exec = start a program" is enough.

How the shell finds the program: PATH

The shell doesn't search your whole disk. It looks only in the folders listed in an environment variableEnvironment variableA named setting the operating system hands a program when it starts, like DATABASE_URL=postgres://… Read in Go with os.Getenv and in Node with process.env. called PATHPATHAn environment variable holding a list of folders separated by colons, like /usr/local/bin:/usr/bin:/bin. When you type a command name, the shell looks for a program with that name in each folder, left to right, and runs the first one it finds., from left to right, and runs the first match. Here's a fresh shell, a little script called hello, and the moment it becomes findable:

$ hello
bash: hello: command not found
$ mkdir -p ~/bin
$ printf '#!/bin/sh\necho hi from my script\n' > ~/bin/hello
$ chmod +x ~/bin/hello
$ hello
bash: hello: command not found
$ export PATH="$HOME/bin:$PATH"
$ hello
hi from my script
$ type hello
hello is hashed (/home/ana/bin/hello)

Line by line: ~ is short for your home folder (/home/ana). printf prints a two-line script, and > saves that output into the file ~/bin/hello instead of the screen (§05 covers >). chmod +x marks the file as runnable (§09). It still isn't found, because ~/bin isn't in PATH. PATH="$HOME/bin:$PATH" puts it at the front of the list ($HOME is the home folder, $PATH the old list), and export (§08) makes the new value visible to the programs bash starts. The next lookup finds it. "Hashed" means bash remembered where it found hello, so it won't search PATH again next time.

If the name contains a slash, like ./deploy.sh or /usr/bin/wc, the shell skips the PATH search and runs exactly that file. That's why you run a script in the current folder as ./deploy.sh: the current folder (.) is deliberately not in PATH, so a malicious file named ls in some download folder can't hijack your ls.

Builtins: commands the shell runs itself

Not every command is a separate program:

$ type cd echo ls
cd is a shell builtin
echo is a shell builtin
ls is /usr/bin/ls

A builtinBuiltin commandA command the shell runs itself instead of starting a separate program, such as cd, export, echo and type. Commands that change the shell's own state (its current folder, its variables) have to be builtins. is code inside the shell itself. Some exist for speed (echo), but some have to be builtins. Remember that a started program is a copy of the shell. If cd were a separate program, it would change the copy's folder, then exit, and your shell would stay exactly where it was. You can watch that happen with bash -c '…', which starts a second bash that runs the quoted text and exits (pwd prints the current folder):

$ pwd
/tmp/command-line/demo
$ bash -c 'cd /tmp && pwd'     # a child shell changes ITS folder…
/tmp
$ pwd                          # …and ours is unchanged
/tmp/command-line/demo

The same rule explains why running a script that does cd or sets variables doesn't affect your terminal: the script runs in a child process. (To run a file's commands inside your current shell, use source file; you'll see it with .env files in §08.)

Arguments and quoting

What the program receives

Here's a tiny program that prints each argument it gets, wrapped in quotes so you can see exactly where each one starts and ends:

package main

import (
    "fmt"
    "os"
)

func main() {
    for i, a := range os.Args[1:] { // os.Args[0] is the program's own name
        fmt.Printf("arg %d: %q\n", i+1, a)
    }
}

Now feed it some lines (compiled as ./args; node args.mjs prints the same):

$ ./args hello   world
arg 1: "hello"
arg 2: "world"
$ ./args "hello   world"
arg 1: "hello   world"

Unquoted spaces separate arguments, and any number of them count as one separator. Inside quotes, spaces are just characters.

Variables are split too, unless you quote them

A shell variable is set with name=value (no spaces around =) and used with $name. The shell replaces $name with its value before splitting the line into arguments. So a value that contains a space gets cut in two. This is called Word splittingWord splittingAfter the shell replaces an unquoted $var with its value, it cuts that value into separate arguments wherever there is a space, tab or newline. So a variable holding "my notes.txt" becomes two arguments: my and notes.txt. Double quotes turn it off.:

$ name="my notes.txt"
$ ./args $name
arg 1: "my"
arg 2: "notes.txt"
$ ./args "$name"
arg 1: "my notes.txt"
$ rm $name
rm: cannot remove 'my': No such file or directory
rm: cannot remove 'notes.txt': No such file or directory

That rm tried to delete two files that happen not to exist. If they had existed, it would have deleted them. This is the most common shell bug there is, and the fix is always the same: put double quotes around every variable, "$name".

Globs: the shell expands * into file names

An unquoted * or ? is a GlobGlob (wildcard)A pattern like *.txt that the shell expands into the matching file names before the program runs. * matches any run of characters, ? matches exactly one. The program receives the list of names, never the pattern.. The shell replaces it with the matching file names, sorted, before the program runs:

$ ls
 a.txt   b.txt  'my notes.txt'
$ ./args *.txt
arg 1: "a.txt"
arg 2: "b.txt"
arg 3: "my notes.txt"
$ ./args "*.txt"
arg 1: "*.txt"
$ ./args *.csv
arg 1: "*.csv"

Notice that ls shows 'my notes.txt' in quotes only to make the space visible; that's not part of the name. A glob result is never split again, so my notes.txt arrives as one argument. And when nothing matches, bash passes the pattern through unchanged: *.csv arrived as the literal text *.csv.

Single vs double quotes

$ ./args "$HOME" '$HOME' "it's" 'say "hi"'
arg 1: "/home/ana"
arg 2: "$HOME"
arg 3: "it's"
arg 4: "say \"hi\""
$ ./args "today is $(date +%A)"
arg 1: "today is Saturday"

The last line uses Command substitution $Command substitution $( )$(cmd) runs cmd and puts its standard output (minus the final newline) in that spot of the command line: echo "today is $(date +%A)" prints today is Saturday.: $(date +%A) runs date and puts its output, the weekday, right there in the string.

Rule of thumb: use double quotes around anything containing a variable, single quotes for fixed text that contains $, *, ! or other characters you want left alone (regular expressions, jq filters, SQL).

Try it yourself. The lab parses a line the way bash does and shows the list the program would get:

File names that look like options

One more trap: a file named -rf or --help. rm * in a folder containing a file called -rf expands to rm -rf a.txt …, and rm reads -rf as options. Two fixes: write ./* instead of * (the names become ./-rf, which no longer start with a dash), or put -- before file names. -- is a convention most tools follow: "everything after this is an argument, not an option". rm -- "$file" is safe whatever $file contains.

stdin, stdout, stderr: the program's side

Think of a kitchen with three hatches. Ingredients come in through one (Standard inputStandard input (stdin)The stream a program reads its input from (file descriptor 0). By default it's your keyboard; a pipe or < can connect it to another program or a file instead.), finished plates go out through another (Standard outputStandard output (stdout)The stream a program writes its normal output to (file descriptor 1). By default it's your terminal; > or | can send it to a file or another program.), and the third is for shouting "we're out of eggs!" (Standard errorStandard error (stderr)A second output stream (file descriptor 2) meant for errors and progress messages, so they still reach your screen when stdout is redirected to a file or a pipe.). The cook never needs to know whether plates go to a table, a delivery bag or another kitchen. That's the shell's business.

Under the hood each stream is a numbered file descriptorFile descriptor (fd)A small number the OS gives your program to identify an open file or socket — e.g. “fd 7”.: 0 for stdin, 1 for stdout, 2 for stderr. When you run a program in a terminal, all three point at the terminal: you type into fd 0, and both fd 1 and fd 2 show up on your screen. (The File Descriptors topic shows what an fd really is inside the kernel.)

Why two output streams?

Because output is often data for the next program, while errors are for you. Here's a program that does both:

package main

import (
    "fmt"
    "os"
)

func main() {
    fmt.Println("result: 42")                      // stdout (fd 1): the data
    fmt.Fprintln(os.Stderr, "warning: cache cold") // stderr (fd 2): for humans
}

$ ./report
result: 42
warning: cache cold
$ ./report > out.txt
warning: cache cold
$ cat out.txt
result: 42

On screen the two lines look identical. But > out.txt (§05) sends only stdout to the file, so the warning still reaches you and the file holds clean data. If the program had printed its warning with fmt.Println or console.log, the warning would be hidden in the file and could break whatever reads it next.

Reading stdin: a filter

A program that reads stdin, transforms each line, and writes stdout is called a filter. Most Unix tools (grep, sort, cut) are filters, and you can write your own in a dozen lines:

package main

import (
    "bufio"
    "fmt"
    "os"
    "strings"
)

func main() {
    sc := bufio.NewScanner(os.Stdin) // read fd 0 line by line
    n := 0
    for sc.Scan() {
        fmt.Println(strings.ToUpper(sc.Text())) // each line → stdout
        n++
    }
    fmt.Fprintf(os.Stderr, "upper: %d lines\n", n) // the summary is for humans → stderr
}

The loop ends when the input reaches its end, called EOFEOF (end of file)The signal that there's nothing more to read. A read returns 0 bytes (Go: io.EOF; Node: the 'end' event). For a pipe or socket this only happens once every writer has closed its end. (end of file). With a file or pipe, that's when the data runs out. Typing in a terminal, you send EOF by pressing Ctrl-D on an empty line.

$ printf 'pear\napple\n' | ./upper
PEAR
APPLE
upper: 2 lines
$ ./upper < fruit.txt > up.txt
upper: 3 lines
$ cat up.txt
APPLE
FIG
PEAR

printf 'pear\napple\n' prints two lines (\n is a newline), and | feeds them to ./upper. In the second command, < makes the file fruit.txt (three fruit names, one per line) the program's stdin and > sends its stdout to up.txt. The program didn't change at all: it reads fd 0 and writes fd 1 either way. Only the summary on stderr reached the screen.

Redirection: pointing streams somewhere else

The operators

$ echo one > log.txt
$ echo two >> log.txt
$ cat log.txt
one
two
$ ./report 2> /dev/null
result: 42
$ ./report > /dev/null 2>&1
$

The last command is silent: stdout goes to the bin, and then stderr is pointed at the same place.

Order matters: 2>&1 copies, it doesn't link

Redirections are processed left to right, and 2>&1 copies fd 1's destination at that moment. Compare:

$ ./report > all.txt 2>&1     # first fd 1 → file, then fd 2 → (copy of fd 1) = file
$ cat all.txt
result: 42
warning: cache cold
$ ./report 2>&1 > out2.txt    # first fd 2 → (copy of fd 1) = terminal, then fd 1 → file
warning: cache cold
$ cat out2.txt
result: 42

The second version is a classic bug: someone wanted "everything in the file" but the warnings still hit the screen, or, in a cron job, got lost. Build commands in the playground and watch the fd table change after each step. Try the presets in order, and especially the last one, which opens the same file twice:

result: 42
warning: cache cold

That last preset is real behavior, too:

$ ./report > o.txt 2> o.txt
$ cat o.txt
warning: cache cold

Each > opens the file separately, with its own write position starting at byte 0. The stdout line is written at byte 0, then the stderr line is written at byte 0 again, over it. 2>&1 avoids that because both fds then share one opened file, with one shared position. That middle layer, between the fd number and the file on disk, is what the playground's open # boxes show: compare > out.txt 2> out.txt (two boxes, both starting at position 0) with > out.txt 2>&1 (one box, whose position moves from 0 to 11 to 31). The kernel calls it an open file descriptionOpen file descriptionThe kernel's record of one open() call: which file or socket it is, the current read/write position (offset) and the mode (read-only, append, non-blocking…). Several fds, even in different processes, can point at the same one..

Here-documents: a few lines of input, right in the script

A Here-documentHere-document (<<EOF)A way to feed several lines of text to a command's stdin, written right in the script and ending at a line that says EOF (or whatever word you chose). With <<'EOF' in quotes, $variables inside are left alone. feeds the lines that follow to the command's stdin, up to a line containing only the marker word (EOF is the usual choice). It's handy for writing config files from a script:

$ export PORT=8080
$ cat > config.yaml <<EOF
port: $PORT
home: $HOME
EOF
$ cat config.yaml
port: 8080
home: /home/ana
$ cat <<'EOF'
literal: $PORT
EOF
literal: $PORT

Without quotes around the marker, $variables are expanded, like inside double quotes. With <<'EOF', the text is taken literally, like single quotes.

Pipes and small tools

Picture an assembly line. Each worker does one small job and passes the piece to the next. Nobody waits for the whole batch: the second worker starts as soon as the first piece arrives. The Unix version of that idea: write small programs that do one thing well, read text and write text, and let pipes combine them.

Building a pipeline, one stage at a time

The file access.log has one web request per line: method, path and status code (200 means OK; 401, 404 and 500 are errors). Question: which paths fail most often? Toggle the stages to see what each one contributes, then press ▶ to watch the lines flow.

      2 /login
      2 /cart
      1 /about

Things to notice while it runs:

• Lines stream. grep and cut pass each line on as soon as they get it. Several stages are busy at the same moment.
• sort is a dam. It can't output anything until it has read all its input, because the very last line might sort first. Everything downstream waits for it.
• uniq -c only merges neighbors. Turn off the first sort and the same path shows up twice. That's why sort | uniq -c is an idiom.
• head quits early. After 3 lines it exits. (What happens to a program still writing into a pipe whose reader has gone is a §12 story.)

You can check that both sides really run at the same time. Each sleep 2 waits two seconds:

$ time (sleep 2 | sleep 2)

real	0m2.006s
user	0m0.005s
sys	0m0.000s

Two seconds, not four. (time measures how long a command takes: real is the wall-clock time, user and sys the CPU time it used. The parentheses group the pipeline so time measures all of it.)

Errors don't go down the pipe

A pipe carries stdout only. Stderr still goes to the terminal, which is usually what you want: error messages stay visible while data flows on. To send both through, redirect first:

$ ./report | tr a-z A-Z
warning: cache cold
RESULT: 42
$ ./report 2>&1 | tr a-z A-Z
RESULT: 42
WARNING: CACHE COLD

tr a-z A-Z translates lowercase letters to uppercase, so the uppercase lines are the ones that went through the pipe. The shell connects the pipe before it processes 2>&1, so 2>&1 copies the pipe into fd 2. Bash also has the shortcut |& for 2>&1 |.

The toolbox

A dozen small tools cover most day-to-day text work. All of them read stdin when you don't give a file name:

Two of these need a closer look.

xargsxargsA tool that reads words or lines from stdin and turns them into arguments for another command: printf 'a\nb\n' | xargs rm runs rm a b. Use -0 with input separated by NUL characters to cope with spaces in names. bridges stdin and arguments. Some commands, like rm or ls, ignore stdin and only take file names as arguments. xargs reads words from its stdin and runs the command with them as arguments:

$ seq 1 5 | xargs -n 2 echo
1 2
3 4
5
$ printf 'a.txt\nmy notes.txt\n' | xargs ls
ls: cannot access 'my': No such file or directory
ls: cannot access 'notes.txt': No such file or directory
a.txt
$ find . -name '*.txt' -print0 | xargs -0 wc -l
 2 ./my notes.txt
 1 ./b.txt
 3 ./a.txt
 6 total

seq 1 5 prints the numbers 1 to 5, and -n 2 passes them two at a time. find . -name '*.txt' lists every .txt file in this folder and below. By default xargs splits at any whitespace, so a name with a space breaks, just like an unquoted variable. The fix is to separate names with the NUL character, which can't appear in a file name: find -print0 writes that way and xargs -0 reads that way.

jqjqA small command-line program for reading and reshaping JSON. jq -r '.name' prints the name field; jq '.[] | select(.status == "down")' keeps matching items from a list. is grep and cut for JSON, which matters because so many tools and APIs speak JSON:

$ cat services.json
[{"name":"api","status":"up"},{"name":"db","status":"down"},{"name":"cache","status":"up"}]
$ jq -r '.[] | select(.status == "down") | .name' services.json
db

Read the filter left to right, like a shell pipeline: .[] goes through each item of the list, select(…) keeps the items where the condition is true, .name takes their name field. -r prints raw text instead of a quoted JSON string. Note the single quotes around the filter: it's full of characters (|, ", .[]) the shell must not touch.

Exit codes: how programs say yes or no

Why is success 0, the "false" number in most languages? Because there's one way to succeed but many ways to fail, and the non-zero value can say which. The shell stores the last exit code in the special variable $?:

$ grep -q POST access.log; echo "found: $?"
found: 0
$ grep -q DELETE access.log; echo "not found: $?"
not found: 1
$ grep x nope.txt; echo "error: $?"
grep: nope.txt: No such file or directory
error: 2

; separates two commands on one line, and grep -q ("quiet") prints nothing at all: its whole answer is the exit code. grep uses three codes: 0 found a match, 1 no match, 2 something went wrong. Each tool documents its own meaning (look under "EXIT STATUS" in man grep), but a few codes come from the shell itself and are worth memorizing:

A signal is a small notification the operating system delivers to a process, such as "the user pressed Ctrl-C" (§11). A CI job that "failed with exit code 137" was almost always killed for using too much memory.

The program's side

Your program sets the code when it exits. Report errors on stderr, then exit non-zero:

package main

import (
    "fmt"
    "os"
)

func main() {
    if len(os.Args) < 2 {
        fmt.Fprintln(os.Stderr, "usage: check FILE") // explain on stderr…
        os.Exit(2)                                     // …and fail with a non-zero code
    }
    if _, err := os.Stat(os.Args[1]); err != nil {
        fmt.Fprintln(os.Stderr, "check:", err)
        os.Exit(1)
    }
    fmt.Println("ok") // returning from main = exit code 0
}

$ ./check; echo "exit $?"
usage: check FILE
exit 2
$ ./check nope.txt; echo "exit $?"
check: stat nope.txt: no such file or directory
exit 1
$ node check.mjs check.mjs; echo "exit $?"
ok
exit 0

&&, || and if: acting on the code

$ mkdir -p build && cd build && echo "in build"
in build
$ grep -q DELETE access.log || echo "no deletes"
no deletes

if in the shell doesn't test true/false values like in JS. It runs a command and checks its exit code. true and false are tiny commands that do nothing except exit with 0 and 1 (in bash they're builtins, and they also exist as programs in /usr/bin).

Chaining both operators is common, a && b || c, and it does not mean "if a then b else c". It starts with everything succeeding. Step through once, then tick "cp fails" and step through again:

1mkdir build \
2  && cp app build/ \
3  || echo "build failed"

Pipelines: only the last command counts

A pipeline's exit code is the exit code of its last command. Failures earlier in the pipe are silently dropped:

$ false | true; echo $?
0
$ false | true; echo "${PIPESTATUS[@]}"
1 0
$ set -o pipefail
$ false | true; echo $?
1

PIPESTATUS is a bash array holding every command's code from the last pipeline. set -o pipefail changes the rule: the pipeline fails if any command in it fails (it reports the rightmost non-zero code). That's almost always what you want in a script: curl … | jq … should fail when the download fails. §10 shows it in action, plus the one case where it bites back.

Environment variables

Environment variables are how you configure a program without changing its code or its arguments: which port to listen on (PORT), which database to use (DATABASE_URL), where to look for programs (PATH, §02). You've seen PATH and HOME already. Run env to list them all; this shell has 173.

Shell variables vs exported variables

A plain assignment creates a shell variable: it exists only inside the current shell. export marks it for copying into every child the shell starts:

$ GREETING=hi
$ bash -c 'echo "child sees: [$GREETING]"'
child sees: []
$ export GREETING
$ bash -c 'echo "child sees: [$GREETING]"'
child sees: [hi]

The single quotes matter: they stop our shell from replacing $GREETING, so the text reaches the child unchanged and the child looks the variable up in its own environment. You can also write export GREETING=hi in one step.

Play with the rules: who gets a copy, when, and what happens when a child sets something.

The program's side

package main

import (
    "fmt"
    "os"
)

func main() {
    port := os.Getenv("PORT") // "" if unset
    if port == "" {
        port = "8080"
    }
    token, ok := os.LookupEnv("API_TOKEN") // ok tells "unset" from "set to empty"
    fmt.Println("port:", port, "| token set:", ok, len(token) > 0)
}

Compiled as ./envdemo, the Go version is what the next examples run.

Setting a variable for one command

NAME=value command puts the variable in that one command's environment and nowhere else. It's the usual way to try settings:

$ PORT=3000 ./envdemo
port: 3000 | token set: false false
$ echo "after: [$PORT]"
after: []
$ PORT=3000 echo "[$PORT]"
[]

The last line surprises people. $PORT is replaced by the current shell while it builds echo's arguments, before echo starts, and the current shell has no PORT. Only programs that read their environment themselves see the per-command value.

Your program can also give its own children a different environment. Both languages copy the current one by default; to add a variable, copy and extend it:

os.Setenv("STAGE", "dev")                    // changes OUR environment (and future children's)
cmd := exec.Command("sh", "-c", `echo "child: STAGE=$STAGE MODE=$MODE"`)
cmd.Env = append(os.Environ(), "MODE=batch") // a copy of ours, plus one extra
cmd.Stdout = os.Stdout
if err := cmd.Run(); err != nil {           // prints: child: STAGE=dev MODE=batch
    log.Fatal(err)                           // couldn't start, or the child exited non-zero
}

Neither program can change the environment of the shell that started it. That's not a missing feature; it's how processes work. Each one owns its copy.

.env files

Projects often keep local settings in a .env file.env fileA plain text file of KEY=value lines that holds local settings for a project. Tools (or set -a; source .env) load it into environment variables. It usually holds secrets, so it must never be committed to git.:

$ cat .env
PORT=5000
DB_URL=postgres://localhost/dev
$ (set -a; source .env; set +a; ./envdemo)
port: 5000 | token set: false false
$ node --env-file=.env env.mjs
port: 5000 | token set: false

source .env runs the file's lines in the current shell (no child process), and set -a makes every variable assigned meanwhile exported automatically. The parentheses run the whole thing in a SubshellSubshellA copy of the shell started to run part of a command line, for example the inside of $( ) or ( ). Changes made in it (cd, variables) disappear when it ends., a throwaway copy of the shell, so your terminal isn't left with those variables afterwards. Node 20.6+ reads the file itself with --env-file. In Go, a library such as godotenv does the same.

Your first real script

Here's a complete, small, correct script. We'll take it apart line by line:

#!/usr/bin/env bash
set -euo pipefail

greet() {
  local name="$1"          # $1 = the function's first argument
  echo "hello, $name"
}

if [[ $# -eq 0 ]]; then    # $# = number of arguments
  echo "usage: $0 NAME..." >&2
  exit 2
fi

for person in "$@"; do     # "$@" = every argument, each kept whole
  greet "$person"
done

$ chmod +x greet.sh
$ ./greet.sh
usage: ./greet.sh NAME...
$ echo $?
2
$ ./greet.sh Ana "Bo Li"
hello, Ana
hello, Bo Li

Line 1: the shebang

#!/usr/bin/env bash is the ShebangShebang (#!)The first line of a script, like #!/usr/bin/env bash. When you run the file directly, the kernel reads this line to learn which interpreter should run the rest of the file.. When you run ./greet.sh, the kernel reads this first line to find out which interpreter runs the file. /usr/bin/env bash means "find bash on PATH", which works on Linux and on macOS (where a newer bash is often installed somewhere other than /bin). chmod +x sets the Execute permissionExecute permission (chmod +x)A permission flag on a file that allows it to be run as a program. chmod +x script.sh sets it. Without it you get “Permission denied” (exit code 126). so the file can be run at all; without it you get Permission denied and exit code 126.

Line 2: strict mode

set -euo pipefail makes the script stop at the first failure instead of plowing on. It's important enough, and has enough traps, to get its own section (§10).

Variables

name="Ana"            # no spaces around = !
greeting="hello, $name"
count=$((count + 1))  # $(( )) does integer arithmetic
files=$(ls)           # $( ) captures a command's stdout

x = 5 with spaces doesn't assign anything. The shell reads it as "run the command x with arguments = and 5" and says bash: x: command not found. Inside a function, local keeps a variable from leaking out into the rest of the script.

The script gets its arguments in special variables:

Always write "$@", with the quotes. Unquoted $@ re-splits every argument at spaces, the same bug as §03:

$ cat bad.sh
#!/usr/bin/env bash
for person in $@; do
  echo "hello, $person"
done
$ bash bad.sh Ana "Bo Li"
hello, Ana
hello, Bo
hello, Li

Conditions: if and [[ ]]

if runs a command and checks its exit code (§07). [[ … ]] is a bash command whose whole job is to test something and exit 0 (true) or 1 (false):

if [[ -f "$config" ]]; then echo "config exists"; fi       # -f: is a regular file
if [[ -d build ]]; then rm -rf build; fi                     # -d: is a directory
if [[ -z "$TOKEN" ]]; then echo "TOKEN is empty" >&2; fi     # -z: empty string
if [[ "$env" == "prod" ]]; then echo "careful!"; fi          # string comparison
if [[ $count -gt 10 ]]; then echo "more than 10"; fi         # numbers: -eq -ne -lt -le -gt -ge
if grep -q ERROR app.log; then echo "found errors"; fi       # any command works

>&2 sends that echo to stderr (it means 1>&2: make fd 1 point where fd 2 points). Use it for every error message in a script.

You'll also see the older single bracket [ … ], which is a regular command (also called test). It needs quotes everywhere, because its arguments go through word splitting like any command's:

$ f="my notes.txt"
$ [ -f $f ] && echo yes
bash: [: my: binary operator expected
$ [[ -f $f ]] && echo "[[ ]] works"
[[ ]] works

[[ ]] is part of bash's grammar, so it doesn't split variables. Prefer it in bash scripts; quoting inside it anyway is a harmless habit.

Loops

#!/usr/bin/env bash
set -euo pipefail

for f in *.txt; do                # the glob expands to one word per file, spaces and all
  lines=$(wc -l < "$f")           # $( ) runs a command and captures its stdout
  if [[ $lines -gt 1 ]]; then
    echo "$f: $lines lines"
  fi
done

count=0
while IFS= read -r line; do       # read stdin one line at a time
  count=$((count + 1))            # $(( )) does integer arithmetic
done < access.log                 # the loop's stdin is the file
echo "access.log has $count lines"

$ ls
 access.log   empty.txt   loop.sh  'my notes.txt'   todo.txt
$ ./loop.sh
my notes.txt: 2 lines
todo.txt: 3 lines
access.log has 10 lines

while IFS= read -r line is the standard incantation for "for each line". read reads one line from stdin into line, and -r keeps backslashes as they are. IFSIFSA bash variable listing the characters used to split words: space, tab and newline by default. read also trims them from the ends of each line; IFS= read -r line turns that off for one read. is the list of characters bash splits words on (space, tab and newline by default), and read also trims them off the ends of the line. Writing IFS= right before read empties that list for this one command, so the line arrives exactly as it was, leading spaces included. The < access.log after done redirects stdin for the whole loop.

Let ShellCheck read it

ShellCheckShellCheckA free linter for shell scripts. It catches missing quotes, unchecked cd, and dozens of other classic bugs, each with a code like SC2086 that links to an explanation. is a linter for shell scripts, and the fastest way to learn which habits bite. Here it is on a three-line cleanup script with two classic bugs:

$ cat clean.sh
#!/bin/bash
dir=$1
rm -rf $dir/build
cd $dir
$ shellcheck clean.sh

In clean.sh line 3:
rm -rf $dir/build
       ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
rm -rf "$dir"/build


In clean.sh line 4:
cd $dir
^-----^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
   ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean:
cd "$dir" || exit

For more information:
  https://www.shellcheck.net/wiki/SC2164 -- Use 'cd ... || exit' or 'cd ... |...
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...

Each warning has a code (SC2086) with a wiki page explaining the bug. Run it in your editor and in CI. It catches most of what the rest of this page warns about.

set -euo pipefail, its traps, and when to stop writing shell

Why the default is dangerous

Step through a four-line cleanup script. Turn on "cd fails", then compare with set -e off and on:

1# set -e   (off)
2cd /srv/app
3rm -rf ./cache/*
4echo "cache cleared"

The three settings, one at a time:

• set -e ("errexit"): if a command fails, stop the script with that command's exit code.
• set -u ("nounset"): using a variable that was never set is an error, instead of quietly becoming an empty string. That catches typos like $TARGTE_ENV:

$ cat deploy.sh
#!/usr/bin/env bash
set -eu
echo "deploying to $TARGET_ENV"
$ bash deploy.sh
deploy.sh: line 3: TARGET_ENV: unbound variable

• set -o pipefail: a pipeline fails if any command in it fails (§07). Flip the stepper above to the pipefail tab to see curl | jq with and without it.

Where set -e doesn't stop you

set -e has exceptions, and they're exactly where bugs hide. All of these were run with set -e at the top:

set -e
check() {
  false              # you'd expect set -e to stop here…
  echo "still running inside check"
}
if check; then echo "check passed?!"; fi

still running inside check
check passed?!

set -e
count=$(grep -c DELETE access.log)   # no matches → grep exits 1 → script dies silently
echo "deletes: $count"

set -e
i=0
((i++))            # the value of i++ is the OLD value, 0 → status 1 → exit
echo "i=$i"        # never runs

set -e
fail() { false; }
x=$(fail; echo "after")
echo "x=$x"        # prints x=after: the failure inside $( ) was ignored

$ bash -c 'set -o pipefail; seq 1 3 | head -1; echo "small: $?"'
1
small: 0
$ bash -c 'set -o pipefail; yes | head -1; echo "status $?"'
y
status 141

The honest summary: set -euo pipefail catches most mistakes and is still the right default, but it isn't a substitute for checking the things that matter (cd dir || exit 1).

Clean up with trap

Scripts often create temporary files. trap '…' EXIT runs a command when the script exits, however it exits: at the end, through set -e, or when it's interrupted (Ctrl-C, kill):

tmp=$(mktemp)                  # create a unique temporary file, print its name
trap 'rm -f "$tmp"' EXIT       # runs when the script ends, success or failure
curl -fsS "$URL" > "$tmp"     # -f: fail on HTTP errors, -s: no progress bar, -S: but still show errors
jq -r .name "$tmp"

When to stop writing shell

Shell is the best tool for gluing programs together: run this, then that, pipe here, retry there. It's a poor tool for programming: data structures, JSON beyond a jq one-liner, error handling with context, tests. Switch to Go or JavaScript when you notice any of these:

• The script is passing about 50–100 lines, or has functions calling functions.
• You need arrays of records, maps, or real JSON manipulation.
• Errors need handling, not just "stop": retries, partial failure, cleanup in several places.
• It must run on Windows, or behave the same on Linux and macOS (sed -i and date differ between the two).
• It handles untrusted input: file names from users, data from the network.

When your Go or JS program runs other programs, don't build a command string for a shell. Pass the program and its arguments as a list, so no shell ever parses them, and there's no quoting to get wrong:

name := "my notes.txt"                // could even be "x; rm -rf ~": still just one argument
cmd := exec.Command("wc", "-l", name) // no shell: each string is one argv entry
cmd.Stderr = os.Stderr                // let the child's errors reach our stderr
out, err := cmd.Output()              // run it, capture its stdout
if err != nil {
    fmt.Fprintln(os.Stderr, "wc failed:", err) // err includes the exit code
    os.Exit(1)
}
fmt.Printf("wc said: %s", out)        // wc said: 2 my notes.txt

Node's exec("wc -l " + name) and Go's exec.Command("sh", "-c", "wc -l " + name) hand the string to a shell, which word-splits it, expands globs, and runs anything after a ;. With a name from a user, that's a security hole: see command injection.

Under the hood: fork, exec, signals and jobs

Watching bash build a pipeline

stracestraceA Linux tool that prints every system call a process makes (open, read, write, epoll_wait…) with arguments and results — the go-to tool for seeing what a program actually asks the kernel. prints every system call a program makes. Here is bash running ls … | wc -l, filtered down to the calls that matter (0x… hides memory addresses, and clone is how Linux implements fork):

$ strace -f -e trace=clone,execve,pipe2,dup2,wait4 bash -c 'ls /tmp/command-line/ex | wc -l'
16198 execve("/usr/bin/bash", ["bash", "-c", "ls /tmp/command-line/ex | wc -l"], 0x… /* 172 vars */) = 0
16198 pipe2([3, 4], 0)                  = 0
16198 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, …) = 16201
16198 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, …) = 16202
16201 dup2(4, 1)                        = 1
16198 wait4(-1,  <unfinished ...>
16202 dup2(3, 0)                        = 0
16201 execve("/usr/bin/ls", ["ls", "/tmp/command-line/ex"], 0x… /* 172 vars */ <unfinished ...>
16202 execve("/usr/bin/wc", ["wc", "-l"], 0x… /* 172 vars */ <unfinished ...>
16202 +++ exited with 0 +++
16201 +++ exited with 0 +++
16198 <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 16201
16198 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 16202

(Trimmed slightly. The number at the start of each line is the PIDPID (process ID)The number the OS gives each running process, e.g. 4120. You use it to look a process up (ps, top) or to send it a signal (kill 4120). making the call; -f follows child processes too.) Everything this page taught is visible here:

1. pipe2([3, 4]): the kernel creates a pipe; fd 3 is its read end, fd 4 its write end.
2. Two clone calls: bash forksfork()The Unix system call that clones the calling process. The copy (the child) gets a new PID and a copy of the parent's memory and open files. It returns twice: 0 in the child, the child's PID in the parent. twice. PIDs 16201 and 16202 are copies of bash.
3. dup2(4, 1) in the first child: "make fd 1 a copy of fd 4". Its stdout is now the pipe's write end. dup2(3, 0) in the second: its stdin is the read end. That's all a pipe in the shell is. Redirections like > file and 2>&1 are the same trick: open the file, then dup2 it onto 1 or 2.
4. execve("/usr/bin/ls", ["ls", …], … /* 172 vars */): execexec()The Unix system call that replaces the program running in the current process with a different program. The PID stays the same; the memory is thrown away and the new program is loaded. takes three things: the program's path (found via PATH, §02), the argv list (§03), and the environment, all 172 exported variables (§08). The environment is literally an argument of exec.
5. wait4: bash blocks until each child exits, and reads its exit status (§07).

Why two steps (fork, then exec) instead of one "start a program" call? Because the gap between them is where the child sets itself up while it's still running the shell's code: rearrange fds, change folder, adjust the environment. Then it becomes the new program. Threads & Processes covers fork and exec in general, and File Descriptors shows the pipe wiring step by step.

Signals and Ctrl-C

A signalSignal (Unix)A tiny message the OS delivers to a running process, e.g. SIGINT (Ctrl+C), SIGTERM (“please shut down” — what kill and Kubernetes send first), SIGKILL (stop immediately; can't be caught) or SIGQUIT (Ctrl+\, Go prints every goroutine's stack). Send one with kill -TERM <pid>. is a tiny asynchronous message from the kernel to a process. Each has a number and a default action, usually "die". The ones you meet at the command line:

When you press Ctrl-C, the terminal doesn't send a character to your program. The terminal driver in the kernel sends SIGINT to every process in the foreground Process groupProcess groupA set of processes the kernel treats as one job, such as all the programs in one pipeline. Ctrl-C sends SIGINT to every process in the foreground group at once.. In an interactive shell, every pipeline is its own group, so Ctrl-C stops all of cat | grep | sort at once.

A program can catch SIGINT and SIGTERM to shut down cleanly: finish the current request, flush files, delete temp files. Then it should still exit non-zero; 130 is the convention for "stopped by Ctrl-C":

func main() {
    // ctx is cancelled when SIGINT (Ctrl-C) or SIGTERM arrives
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    fmt.Println("working… press Ctrl-C")
    select {
    case <-time.After(10 * time.Second):
        fmt.Println("done")
    case <-ctx.Done():
        fmt.Fprintln(os.Stderr, "interrupted: cleaning up")
        os.Exit(130) // 128 + 2 (SIGINT): the conventional "stopped by Ctrl-C" code
    }
}

$ ./sig
working… press Ctrl-C
^Cinterrupted: cleaning up
$ echo $?
130

This matters most in containers, where docker stop and Kubernetes send SIGTERM and wait a grace period before SIGKILL. See Containers: signals and PID 1.

Job control

Job controlJob controlShell features for juggling running commands: cmd & starts one in the background, Ctrl-Z pauses the foreground one, jobs lists them, fg and bg resume them in the foreground or background. lets one terminal juggle several commands. A real session:

$ sleep 100
^Z
[1]+  Stopped                 sleep 100
$ jobs
[1]+  Stopped                 sleep 100
$ bg
[1]+ sleep 100 &
$ jobs
[1]+  Running                 sleep 100 &
$ kill %1
$ jobs
[1]+  Terminated              sleep 100
$ sleep 50
^C
$ echo $?
130

Ctrl-Z sends SIGTSTP, which pauses the foreground job and gives you the prompt back. bg resumes it in the background; fg brings it back to the foreground. cmd & starts a command in the background from the beginning. %1 means "job number 1", and works with kill, fg and bg.

Terminals vs pipes: buffering, isatty and SIGPIPE

Programs check where their output goes

$ ls
 access.log   empty.txt   loop.sh  'my notes.txt'   todo.txt
$ ls | cat
access.log
empty.txt
loop.sh
my notes.txt
todo.txt

Same command, different output. To a terminal, ls prints columns and quotes awkward names, for a human. Into a pipe, it prints one plain name per line, for a program. It decides with a check called isattyisatty (is it a terminal?)A check a program makes to see if a file descriptor is connected to a terminal. Many tools change behavior based on it: colors and columns for a person, plain lines for a pipe or file., and you can make the same check:

info, _ := os.Stdout.Stat()
isTerminal := info.Mode()&os.ModeCharDevice != 0 // a terminal is a "character device" (so is /dev/null!)
fmt.Fprintln(os.Stderr, "stdout is a terminal:", isTerminal)

$ ./tty
stdout is a terminal: true
$ ./tty | cat
stdout is a terminal: false

The Go check is a shortcut: /dev/null is a character device too, so ./tty > /dev/null also says true. For a precise answer, use term.IsTerminal(int(os.Stdout.Fd())) from the golang.org/x/term package, which asks the kernel the real question. Node's isTTY is already precise.

Use it the same way tools do: colors, progress bars and tables when it's a person; plain, stable, parseable lines when it's a pipe. (Many tools also honor the NO_COLOR environment variable, and offer --color=always to force colors into a pipe, e.g. grep --color=always … | less -R.)

Output buffering: why your pipeline "hangs"

Every write to a pipe, file or terminal is a system call, and system calls are relatively expensive (see Syscalls: what a syscall costs). So many programs use Output bufferingOutput bufferingCollecting output in memory and writing it in bigger chunks to save system calls. Many programs flush after every line when writing to a terminal, but only every few kilobytes when writing to a pipe or file.: they collect output in memory and write it in chunks. The common rule comes from libc, the C library most Linux tools (grep, sed, awk) are built on, and Python copies it: to a terminal, write after every line; to a pipe or file, write when a few KiB have piled up (4 KiB in glibc, 8 KiB in Python), or when the program exits.

Usually that's invisible. It becomes visible with slow, continuous input, like following a log. Here logs prints one line per second, and stamp prefixes each line with the time it arrived:

stamp() {                       # print "seconds since start" before each line
  local start=$SECONDS
  while IFS= read -r line; do echo "$((SECONDS - start))s  $line"; done
}
logs() { for i in 1 2 3; do echo "ERROR $i"; sleep 1; done; }

logs | grep ERROR | stamp
echo ---
logs | grep --line-buffered ERROR | stamp

$ bash buf.sh
3s  ERROR 1
3s  ERROR 2
3s  ERROR 3
---
0s  ERROR 1
1s  ERROR 2
2s  ERROR 3

In the first run, nothing appeared for three seconds, and then everything at once: grep was holding the lines in its buffer, because its stdout was a pipe. With tail -f app.log | grep ERROR | …, the buffer might take minutes to fill, and it looks like the pipeline is stuck. Fixes: grep --line-buffered, sed -u, python -u, or stdbuf -oL cmd for many other C programs.

Your own programs: Go's os.Stdout is not buffered at all: every fmt.Println is one write system call. That's simple and never "hangs", but it's slow for big outputs. Wrap it in a buffer when you print a lot:

w := bufio.NewWriter(os.Stdout) // collect output in memory (4 KiB by default)…
defer w.Flush()                 // …and write what's left before main returns
for i := 0; i < 1_000_000; i++ {
    fmt.Fprintln(w, i)
}

Counted with strace -f -c, printing a million lines took 1,000,000 write calls unbuffered and 1,682 with bufio.Writer, and the run time dropped from about 0.4 s to 0.09 s on this machine. Don't forget the Flush: whatever is still in the buffer when the program exits is lost, and os.Exit skips defer.

SIGPIPE: when the reader goes away

$ ./slow | head -2
0
1
$ echo "${PIPESTATUS[@]}"
141 0

./slow wants to print a million numbers, but head -2 exits after two lines, closing the pipe's read end. ./slow's next write goes to a pipe nobody can read. The kernel answers with SIGPIPE / EPIPESIGPIPE / EPIPEWhat happens when you write to a pipe or socket whose reader has gone away. By default the kernel sends the SIGPIPE signal, which kills the process; if that's ignored, the write fails with the error EPIPE (“broken pipe”)., and its default action kills the writer: exit code 128 + 13 = 141.

That is a feature. Without it, ./slow would keep computing output nobody will ever read; with it, yes | head -1 finishes instantly. Every well-behaved command-line tool dies quietly from SIGPIPE, which is why you rarely notice it. You notice it in two places:

• set -o pipefail scripts, where the 141 suddenly counts as a failure (§10, pitfall #5).
• Programs that ignore the signal, whose writes then fail with the error EPIPE, "broken pipe". Go lets SIGPIPE kill the program when the broken pipe is stdout or stderr, as Unix tools do, but turns it into an error for other fds such as network sockets. Node ignores SIGPIPE, and that has a surprising consequence, below. File Descriptors: SIGPIPE has more details.

Node and EPIPE: the loop that keeps running

Here's a Node script that prints three million numbers, piped into head:

for (let i = 0; i < 3e6; i++) console.log(i);
console.error("finished the loop");

$ time (node count.mjs | head -2; echo "${PIPESTATUS[@]}")
0
1
finished the loop
0 0

real	0m5.731s

head quit after two lines, but Node kept going for almost six seconds and exited with 0. Node ignores SIGPIPE, so every write after that fails with EPIPE, and console.log silently swallows write errors. The program does all its work for nobody. (If you write with process.stdout.write and have no error handler, you get the opposite: the first EPIPE crashes the program with Error: write EPIPE and exit code 1.)

The fix is to do what SIGPIPE does for other tools: when stdout breaks, stop.

process.stdout.on("error", (err) => {
  if (err.code === "EPIPE") process.exit(0); // the reader is gone: stop quietly, like a Unix tool
  throw err;                                  // anything else is a real error
});

let i = 0;
function writeMore() {
  while (i < 3e6) {
    if (!process.stdout.write(`${i++}\n`)) return process.stdout.once("drain", writeMore); // wait while the pipe is full
  }
}
writeMore();

$ time (node count-fixed.mjs | head -2; echo "${PIPESTATUS[@]}")
0
1
0 0

real	0m0.044s

Check yourself: predict the output

Predict the answer before you click. Every one is covered above: Q1 in §03, Q2 in §05, Q3 and Q4 in §07, Q5 and Q7 in §08, Q6 in §10 and §12.

name="my notes.txt"
rm $name

./report 2>&1 > out.txt

mkdir build && cp app build/ && echo "copied" || echo "failed"

false | true
echo $?

GREETING=hi
bash -c 'echo "[$GREETING]"'

set -o pipefail
yes | head -1
echo "status $?"

PORT=3000 echo "port is [$PORT]"

Cheat sheet