System Calls

The 30-second version

Your program has never read a file. Not once. It has only ever asked the operating system to read one for it, and waited for the answer.

The kernel is the core of the operating system (Linux, macOS, Windows). It owns the hardware: the disk, the network card, memory, the CPU's time. Every other program, including yours, runs in user spaceUser space vs kernel spaceYour programs run in user space with limited rights; the OS kernel runs in kernel space with full access to hardware. Crossing from one to the other (a system call) costs time., a restricted zone where touching hardware directly is simply not possible.

Here is the whole picture, top to bottom. Your code rarely writes the slip itself: the language runtimeRuntimeCode that ships inside every program of a language to support it while it runs. Go's runtime schedules goroutines and manages memory; Node's includes the event loop. (Node's libuv, the C library Node uses for all its I/O; or the Go runtime) fills it in for you when you call fs.readFile or os.ReadFile.

Before reading further, test your instinct. Which of these can your program do on its own?

The rest of this topic follows that list: why the wall exists (§02), what one request does mechanically (§03), what it costs (§04), how to watch your programs make them (§05), and then how waiting is done well (§06 – §10), and how to run all this in production (§11).

User mode vs kernel mode: why your code can't touch the disk

Why have a wall at all?

Imagine there were no wall. Any program could send commands straight to the disk controller. Then:

• A bug in one program could wreck the whole machine. A buggy loop writing to the wrong spot on disk could overwrite another app's files, or the operating system itself.
• Programs could spy on each other. Your browser could read your password manager's memory.
• Nobody could share fairly. One program could keep the CPU forever and never let anything else run.

So the CPU has (at least) two privilege modesUser mode vs kernel modeThe CPU has two privilege levels. Your code runs in user mode and can't touch hardware or other processes. The OS kernel runs in kernel mode and can do anything. Crossing between them (a syscall, an interrupt) costs a little time., a setting the CPU checks on every instruction. In kernel mode, every instruction is allowed. In user mode, the privileged instructionsPrivileged instructionA CPU instruction that only kernel-mode code may run, such as talking to a disk controller or changing the map of which memory each program may use. If user-mode code tries one, the CPU refuses and hands control to the kernel, which usually kills the program. (talk to a device, change which memory a program can see, stop the CPU from being interrupted — see the table below) are refused: the CPU stops your program and hands control to the kernel, which usually kills it.

Rings

x86 CPUs (Intel and AMD) call these levels protection ringsProtection ringThe CPU's privilege level, numbered from 0 (most powerful) upward. On x86 the kernel runs in ring 0 and your programs in ring 3; the rings in between are almost never used today.. The kernel runs in ring 0, your programs in ring 3:

Notice the bottom box: there are only three ways for the CPU to go from your code into the kernel.

Two of these the program didn't ask for. A timer interruptTimer interruptA hardware alarm that fires regularly (e.g. every 1–4 ms) and forces the CPU to jump into the kernel — even if the running thread doesn't want to stop. It's how the scheduler takes the CPU back. (a hardware interrupt from the CPU's built-in alarm clock) is how the schedulerOS schedulerThe part of the kernel that decides which ready thread runs on which CPU core, and for how long. It runs thousands of times per second., the part of the kernel that decides which thread runs next, takes the CPU back from a thread (Threads & Processes §04). The syscall is the only door your program opens deliberately, so it's the subject of this whole topic.

Same thread, different mode

One point trips almost everyone up. During a syscall, your thread keeps running, just in kernel mode. The kernel doesn't hand your request to some other kernel worker. Your thread walks through the window, does the vault work with the kernel's code and permissions, and walks back out.

That's why a syscall isn't a context switchContext switchWhen the OS pauses one thread and resumes another on the same CPU core. It's cheap, but not free — thousands per second add up. (swapping one thread off the CPU for another; see Threads & Processes §05). It's a mode switch within the same thread, which is much cheaper. A context switch only happens if the kernel decides the thread must wait, say for data that hasn't arrived. Keep that difference in mind: §04 puts numbers on both (the context-switch figure comes from Threads & Processes).

One syscall, step by step

On the bank slip, the first box says which service you want. In a syscall that's the syscall numberSyscall numberThe number that identifies which kernel service you want: on Linux x86-64, 0 = read, 1 = write, 39 = getpid. It goes in a CPU register before the syscall instruction.. Linux on x86-64 has about 450 of them. A few you'll meet constantly:

(Recall from §02: an fd, or file descriptorFile descriptor (fd)A small number the OS gives your program to identify an open file or socket — e.g. “fd 7”., is the small number the kernel gives you for an open file or connection: fd 1 is the terminal output. The File Descriptors topic goes deep on them.)

The arguments go in other registers (registersRegisters / program counterA handful of tiny, ultra-fast storage slots inside each CPU core. They hold what the running thread is working on right now; one of them, the program counter, holds the address of the next instruction to run. are the CPU's handful of tiny built-in storage slots). Why registers and not just “call a function”? Because the kernel's code isn't in your program. You can't jump to it. Registers are the one thing both sides can see at the moment of crossing.

Step through a write of three bytes to the terminal:

1msg := []byte("hi\n")
2n, err := os.Stdout.Write(msg)
3 
4// Inside the Go runtime this becomes, roughly:
5//   RAX ← 1       which service? (1 = write)
6//   RDI ← 1       which fd? (1 = stdout)
7//   RSI ← &msg[0] where are the bytes?
8//   RDX ← 3       how many bytes?
9//   SYSCALL       the CPU instruction
10//   n = RAX       result: count, or -errno
11 
12fmt.Println(n, err) // 3 <nil>

Return values and errno

Every syscall returns one number in RAX. Zero or positive means success (often a count: bytes read, bytes written, the new fd). A small negative number means failure, and its value is the errnoerrnoThe error code a failed system call reports: a small number with a name, like ENOENT (no such file), EAGAIN (try again later) or EINTR (interrupted). Go turns it into an error value; Node into an Error with a code property.:

Careful: you'll see the same failure written in different ways. The kernel returns -errno in RAX (-2 for ENOENT). Wrappers split it in two: C's libc and Go's raw syscall package return -1 (“failed”) and put the code in a separate errno / err. strace prints it the C way (= -1 ENOENT), and Node's libuv keeps the kernel's form (errno: -2). Click through one failure:

Your language converts this for you. Same failure, both sides:

_, err := os.Open("/nope.txt")
fmt.Println(err)                          // open /nope.txt: no such file or directory
fmt.Println(errors.Is(err, fs.ErrNotExist)) // true: portable check
fmt.Println(errors.Is(err, syscall.ENOENT)) // true: the raw errno is inside

Who actually executes the instruction?

You almost never write SYSCALL yourself. Something in between fills in the slip:

• C programs and Node go through libclibc (C library)The standard library that almost every program on Linux or macOS uses to talk to the operating system — file access, memory, and hostname lookups via getaddrinfo. On most Linux systems it's glibc; on Alpine it's musl. (the standard C library), whose small wrapper functions like read() load the registers and run the instruction. Node's libuv calls those wrappers.
• Go on Linux skips libc: the Go runtime has its own wrappers written in assembly, which is part of why Go binaries are easy to copy between Linux machines. (On macOS, Go does call the system library, because Apple only supports syscalls through it.)

What a syscall costs, measured, and the vDSO shortcut

Numbers beat adjectives, so let's measure. A microbenchmarkMicrobenchmarkA tiny test that runs one operation millions of times and reports the average time per run (e.g. “90 ns/op”). Good for comparing two ways of doing one thing; the absolute numbers change from machine to machine. runs one tiny operation millions of times and divides. We'll compare four things in Go:

package bench

import (
	"os"
	"testing"
	"time"
)

//go:noinline
func add(a, b int) int { return a + b }

func BenchmarkFuncCall(b *testing.B) {
	x := 0
	for b.Loop() {
		x = add(x, 1) // a normal function call: no kernel
	}
	_ = x
}

func BenchmarkGetpid(b *testing.B) {
	for b.Loop() {
		os.Getpid() // ← a real getpid syscall, every time
	}
}

func BenchmarkTimeNow(b *testing.B) {
	for b.Loop() {
		time.Now() // ← reads the clock via the vDSO
	}
}

func BenchmarkWrite1(b *testing.B) {
	f, err := os.OpenFile(os.DevNull, os.O_WRONLY, 0)
	if err != nil {
		b.Fatal(err) // stop this benchmark with a clear message
	}
	defer f.Close()
	one := []byte("x")
	for b.Loop() {
		if _, err := f.Write(one); err != nil { // ← a write syscall of 1 byte
			b.Fatal(err)
		}
	}
}

Real output, Go 1.24 on a 4-core 2.3 GHz Xeon virtual machine, Linux 6.18 (typical lines from three runs, which varied by about ±10%):

BenchmarkFuncCall-4    1000000000      1.000 ns/op
BenchmarkGetpid-4        13093461     88.19 ns/op
BenchmarkTimeNow-4       23502009     52.17 ns/op
BenchmarkWrite1-4         6160263    192.3 ns/op

performance.now()        58 ns/op
fs.writeSync(1 byte)     352 ns/op

Put side by side (log scale, so each step right is 10× more):

Reading the numbers

• getpid ≈ 90 ns is the price of the crossing itself: the kernel does almost no work for it. That's ~100 function calls' worth.
• write ≈ 190 ns in Go, ≈ 350–400 ns from Node. The kernel does real work (check the fd, copy the byte), and Node adds its own layers: JS → C++ binding → libuv → libc.
• None of these slept. No thread switch happened. If a syscall must wait, add a context switchContext switchWhen the OS pauses one thread and resumes another on the same CPU core. It's cheap, but not free — thousands per second add up. (microseconds) plus however long the wait itself lasts.

Worked example. A Go service handling 50,000 requests a second makes ~20 small syscalls per request (read the request, a couple of writes, some epoll calls, logging). That's 1,000,000 syscalls/s × ~0.2 µs = 0.2 s of CPU per second: 20% of a core, just on crossings. Worth knowing, rarely worth panicking about. Cut the count where it's easy (§09).

Your machine will differ. Typical published ranges are ~50 ns to ~1 µs per simple syscall, depending on the CPU and especially on the kernel's security mitigationsCPU security mitigationsExtra work the kernel does on every user↔kernel crossing to block CPU side-channel attacks like Meltdown and Spectre (e.g. flushing buffers, switching page tables). They can make each syscall noticeably slower on affected CPUs.. After the 2018 MeltdownMeltdown & KPTIMeltdown (2018) let user code on many Intel CPUs read kernel memory that was mapped into its address space. The fix, kernel page-table isolation (KPTI), unmaps the kernel while user code runs, which makes every system call a bit slower. and SpectreSpectreA family of CPU flaws disclosed in 2018: code could trick the CPU into leaking memory from the same process through timing. It's why browsers put each website in its own process. disclosures, kernels added extra work to every crossing on affected CPUs. On some older Intel chips that roughly doubled the cost of a cheap syscall. Newer CPUs fix much of it in hardware. (This VM's CPU reports “Meltdown: not affected”.)

The vDSO: the kernel's answer booth on your side of the glass

Why is time.Now() cheaper than getpid(), when both “ask the OS” something? Because the kernel cheats, nicely. It maps a tiny read-only library, the vDSOvDSO“virtual dynamic shared object”: a tiny piece of kernel-provided code mapped into every Linux process. It answers a few read-only questions (mainly “what time is it?”) without entering the kernel at all, so they cost tens of nanoseconds instead of a full syscall., into every process. It holds code for a few questions whose answers the kernel keeps up to date in shared memory, mainly “what time is it?” (clock_gettime, gettimeofday). Calling it is a normal function call: no mode switch.

You can see the vDSO in any process's memory map, and prove the clock calls never reach the kernel:

$ grep vdso /proc/self/maps
7f386e974000-7f386e976000 r-xp 00000000 00:00 0      [vdso]

$ strace -c -f ./vdso-demo      # calls time.Now() 1,000× and os.Getpid() 1,000×
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 18.25    0.000695           0      1000           getpid
   …                                                (no clock_gettime line at all)

strace (§05) counted 1,000 getpid calls and zero clock calls. (Go doesn't remember your process ID between calls; each os.Getpid() really asks.)

Watching syscalls with strace

stracestraceA Linux tool that prints every system call a process makes (open, read, write, epoll_wait…) with arguments and results — the go-to tool for seeing what a program actually asks the kernel. is a Linux tool (install it with your package manager, e.g. apt install strace). Point it at a command, and every syscall comes out on stderr in a C-like format:

read(3, "hello, kernel\n", 512) = 14
│    │  │                 │      └─ return value (bytes read; -1 plus an errno on failure)
│    │  │                 └─ argument: at most 512 bytes, please
│    │  └─ argument: where the data went (strace shows what landed there)
│    └─ argument: fd 3
└─ syscall name

The flags you'll use most:

Two tiny programs, traced

Both programs read a 14-byte file, make one HTTP request to a local server, and print a line. Real output from Go 1.24 and Node 22 (libuv 1.51) on Linux 6.18, captured with strace -f -e trace=… and trimmed for reading. Click through it:

1func main() {
2    data, err := os.ReadFile("hello.txt")
3    if err != nil {
4        panic(err)
5    }
6    resp, err := http.Get("http://127.0.0.1:18573/")
7    if err != nil {
8        panic(err)
9    }
10    body, _ := io.ReadAll(resp.Body)
11    resp.Body.Close()
12    fmt.Println(len(data), resp.Status, len(body))
13}

What jumps out:

1. Your code caused syscalls it never mentioned. Go created an epoll instance; Node set up an io_uring. Runtimes do work on your behalf.
2. Sockets are non-blocking (SOCK_NONBLOCK) in both, and both runtimes happily get EAGAIN and EINPROGRESS “errors”. Those aren't failures; they're how waiting is done without blocking a thread (§06).
3. Files are handled differently from sockets. Go's epoll_ctl on the file failed with EPERM; Node never even tried, and sent the file work to helper threads. §08 explains why.

The summary view: strace -c

For a quick “what is this program spending its syscalls on”, -c beats reading lines. The same Go client, whole run (real output, trimmed to the interesting rows):

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0       114           rt_sigaction
  0.00    0.000000           0        39           nanosleep
  0.00    0.000000           0        30           futex
  0.00    0.000000           0        23           mmap
  0.00    0.000000           0         7         2 read
  0.00    0.000000           0         3         1 epoll_ctl
  0.00    0.000000           0         3           epoll_pwait
  0.00    0.000000           0         1         1 connect
   …
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000           0       292         4 total

292 syscalls for a program that “reads a file and makes one request”. Most are the runtime starting up: rt_sigaction is Go going through all ~64 signalSignal (Unix)A tiny message the OS delivers to a running process, e.g. SIGINT (Ctrl+C), SIGTERM (“please shut down” — what kill and Kubernetes send first), SIGKILL (stop immediately; can't be caught) or SIGQUIT (Ctrl+\, Go prints every goroutine's stack). Send one with kill -TERM <pid>. numbers, reading each one's current handler and then installing its own for most of them, mmap asks for memory, futex and nanosleep are threads coordinating. The same Node program made 833, most of them while booting V8 and loading modules. The errors column counts the EAGAIN/EPERM/EINPROGRESS results, which were all expected.

Blocking vs non-blocking I/O

Blocking: the default

By default, a read on a socket with no data yet is a blocking callBlocking callA call that doesn't return until its work is done. A blocking read() on a socket with no data puts the calling thread to sleep inside the kernel, possibly for minutes.. The kernel can't hand you bytes that haven't arrived, so it does the polite thing: it marks your thread as waiting, puts it on the socket's wait queueWait queueA list the kernel keeps for each thing you can wait on (a socket, a pipe, a lock) of threads sleeping until it changes. When data arrives, the kernel wakes the threads on that list. (a list of threads waiting on it), and runs some other thread on the CPU. When a packet arrives, the network card raises a hardware interruptHardware interruptA signal from a device (network card, disk, timer) that makes the CPU stop what it's doing and run a kernel handler, e.g. “a packet has arrived”. Unlike a syscall, your program didn't ask for it., the kernel files the data under the socket, sees your thread on the list and wakes it up. Your read returns as if no time had passed. Step through it:

For one connection, this is perfect: simple code, zero CPU used while waiting. The trouble starts with many connections. One thread can only sit in one chair, so the classic server gives every connection its own thread. At 10,000 connections that's 10,000 threads, each reserving stack memory and each wake-up costing a context switchContext switchWhen the OS pauses one thread and resumes another on the same CPU core. It's cheap, but not free — thousands per second add up.. Async / Await §03 tells this story (the “C10K problem”) and Threads & Processes §04 shows what all those sleeping threads cost.

Non-blocking: “don't make me wait”

Set the O_NONBLOCKNon-blocking mode (O_NONBLOCK)A flag on a file descriptor that tells the kernel: never put me to sleep. If a read or write can't make progress right now, it returns immediately with the error EAGAIN instead. flag on an fd and the kernel changes its answer. No data? It returns immediately with EAGAINEAGAINThe errno a non-blocking call returns when it would have had to wait: “nothing to read yet (or no room to write) — try again later”. Printed as “resource temporarily unavailable”. Also called EWOULDBLOCK.. Here it is with a pipePipeA one-way stream of bytes between two processes, kept by the kernel. One writes into it, the other reads out of it. The shell's `ls | grep x` is a pipe. (a kernel-managed byte channel; it behaves like a socket for this purpose):

var p [2]int // p[0] = read end, p[1] = write end
if err := syscall.Pipe(p[:]); err != nil {
	panic(err)
}
syscall.SetNonblock(p[0], true) // ← O_NONBLOCK: never put me to sleep

buf := make([]byte, 64)
n, err := syscall.Read(p[0], buf) // nothing has been written yet
fmt.Println(n, err, errors.Is(err, syscall.EAGAIN))
// -1 resource temporarily unavailable true

syscall.Write(p[1], []byte("hi"))
n, err = syscall.Read(p[0], buf)
fmt.Println(n, err, string(buf[:n]))
// 2 <nil> hi

There's no JavaScript version because JavaScript never gets this low: Node's libuv makes every socket non-blocking for you, and handles the EAGAINs you saw in the strace in §05.

So we stop waiting… and spin

Non-blocking alone doesn't solve anything. If read says “not yet”, what does the thread do next? The naive answer is to ask again, in a loop: busy pollingBusy pollingAsking “ready yet?” over and over in a loop. It reacts fast but burns a whole CPU core doing nothing useful, which is why event loops sleep in epoll_wait instead.. It reacts instantly but burns a whole CPU core on “not yet, not yet, not yet”, and with 10,000 sockets you'd be making 10,000 useless syscalls per lap.

The three options side by side:

What we want is option 3: non-blocking sockets, plus one call that sleeps until any of them is ready. That's I/O multiplexingI/O multiplexingWatching many file descriptors with one call and being told which ones are ready. select, poll, epoll and kqueue all do this., the subject of the next section.

select, poll, epoll: one thread watching thousands

Unix got this idea three times, each fixing the last one's scaling problem.

1983: select

fd_set readable;              // a bitmap: bit N = “watch fd N”
FD_ZERO(&readable);
FD_SET(sock1, &readable);     // you rebuild this before EVERY call
FD_SET(sock2, &readable);
select(maxfd + 1, &readable, NULL, NULL, NULL); // sleeps until ≥ 1 is ready
// on return, the bitmap was overwritten: test every bit to find the ready ones

Two problems. The fd_setfd_setThe fixed-size bitmap that select() uses to list the file descriptors to watch: one bit per fd number, usually 1,024 bits. An fd numbered 1024 or higher simply doesn't fit. bitmap has a fixed size, normally 1,024 bits, so a socket numbered 1024 or higher can't be watched at all (writing past the end is a classic memory-corruption bug). And every call does work proportional to all watched fds: you rebuild the bitmap, the kernel copies it in, checks each fd, copies it back, and you scan it.

1987: poll

poll replaced the bitmap with an array of {fd, events, revents} entries: no 1,024 limit. But it kept the second problem. Every call still sends the whole list in, and the kernel still checks every entry.

Worked example. A chat server holds 10,000 open connections; at any moment, about 10 have sent something. With poll, each wake-up copies 10,000 entries into the kernel, checks 10,000 sockets, copies 10,000 entries back, and your code scans 10,000 to find the 10. Do that 1,000 times a second and you're touching 10 million entries a second to do 10,000 units of real work.

2002: epoll

Linux's epollepoll / kqueue / IOCPOS features that let one thread say “here are 10,000 connections — wake me when any of them has data”. epoll is Linux, kqueue is macOS/BSD, IOCP is Windows. (kernel 2.5.44) splits the job into three calls, so the list lives inside the kernel:

1. epoll_create1() makes an epoll instance (itself an fd).
2. epoll_ctl(ep, ADD, fd, events) adds an fd to its interest listInterest listThe set of file descriptors an epoll instance is watching, kept inside the kernel. You add and remove entries with epoll_ctl, once, instead of passing the whole list on every call like poll() does.. Once per fd, not once per wait.
3. epoll_wait(ep, …) sleeps until the ready listReady listThe list, inside an epoll instance, of watched file descriptors that currently have something to do (data to read, room to write). epoll_wait copies out just these. is non-empty, then copies out only the ready entries.

The ready list is the trick. When a packet arrives on a watched socket, the kernel's own code (the part that handles the network card's interrupt) appends that socket to the ready list right then. So epoll_wait doesn't search anything: it hands over a list that's already built. Cost per call follows the number of ready fds, not watched ones.

Here's that loop in real Go, with pipes standing in for network connections:

ep, err := syscall.EpollCreate1(0) // ① a new epoll instance (it's an fd too)
if err != nil {
	panic(err)
}

var pipes [3][2]int // [i][0] = read end, [i][1] = write end
for i := range pipes {
	syscall.Pipe(pipes[i][:]) // (errors ignored here and below for brevity)
	ev := syscall.EpollEvent{Events: syscall.EPOLLIN, Fd: int32(pipes[i][0])}
	syscall.EpollCtl(ep, syscall.EPOLL_CTL_ADD, pipes[i][0], &ev) // ② watch it
}

go func() { // somewhere else, data shows up on pipes 2 and 0
	time.Sleep(100 * time.Millisecond)
	syscall.Write(pipes[2][1], []byte("hello"))
	syscall.Write(pipes[0][1], []byte("hey"))
}()

events := make([]syscall.EpollEvent, 8)
for handled := 0; handled < 2; {
	n, err := syscall.EpollWait(ep, events, -1) // ③ sleep until something is ready
	if err == syscall.EINTR {
		continue // a signal interrupted the wait: just wait again (§11)
	}
	if err != nil {
		panic(err)
	}
	for _, e := range events[:n] { // ④ only the READY fds come back
		buf := make([]byte, 16)
		k, _ := syscall.Read(int(e.Fd), buf)
		fmt.Printf("fd %d ready: %q\n", e.Fd, buf[:k])
		handled++
	}
}
// fd 8 ready: "hello"
// fd 4 ready: "hey"

You'd never write this in a real Go program: the runtime's netpollerNetpollerThe part of the Go runtime that uses epoll/kqueue to wait for network data, so goroutines can “block” on the network without blocking an OS thread. runs exactly this loop for you (§08). In Node, libuv runs it at the heart of the event loop.

Try it: interest list, ready list, and two triggering modes

Things to try: unregister a socket and make packets arrive on it (nobody's told). Then switch to edge-triggered, tick “reads only 1 KB”, let 2 packets arrive on one socket and call epoll_wait twice.

Level-triggered vs edge-triggered

• Level-triggeredLevel-triggeredAn epoll mode (the default): a file descriptor is reported on every epoll_wait for as long as it is ready, e.g. while unread data remains. Forgiving: if you read only part of the data, you'll be told again. (the default): an fd is reported as long as it's ready. Read half the data, and the next epoll_wait reports it again. Forgiving and simple.
• Edge-triggeredEdge-triggered (EPOLLET)An epoll mode where a file descriptor is reported only when it becomes ready (new data arrives), not while it stays ready. Fewer wake-ups, but you must read until EAGAIN or leftover data can sit forever. (EPOLLET): an fd is reported when it becomes ready, i.e. when new data arrives. Read half the data and you will not hear about the rest until more arrives. In exchange, there are fewer repeat notifications, so the kernel and your loop do less work per wake-up.

The names come from electronics: a level is “the signal is high”, an edge is “the signal just went from low to high”. The rule for edge-triggered is non-negotiable: after each wake-up, read (or write) until you get EAGAIN. That's exactly why the Go strace in §05 ended with one more read returning EAGAIN: Go registers sockets with EPOLLET.

Other operating systems

kqueue (FreeBSD 2000, macOS) is epoll's cousin: you pass a list of changes and collect events in the same kevent call, which saves syscalls, and it can also watch files changing, processes exiting, signals and timers. IOCPIOCP (I/O completion ports)Windows' high-performance I/O interface. You start operations and later collect “this one finished” notices from a queue: a completion model, unlike Linux's epoll. on Windows works differently: it tells you when an operation has finished, not when one could start. That difference turns out to matter a lot, and §10 picks it up with io_uring.

How Go and Node use all this

Go: the netpoller

You never see EAGAIN in Go code, yet the strace in §05 was full of them. Here's what happens when a goroutine calls conn.Read(buf) and no data has arrived:

1. The runtime calls read on the non-blocking socket. Result: EAGAIN.
2. Instead of returning that error to you, it parksParking a goroutineTaking a goroutine that has to wait off the CPU and setting it aside, so the thread can run other goroutines. A parked goroutine costs no CPU until something wakes it. the goroutine: sets it aside, remembering “wake me when fd 7 is readable”. The OS thread goes off to run other goroutines.
3. The netpollerNetpollerThe part of the Go runtime that uses epoll/kqueue to wait for network data, so goroutines can “block” on the network without blocking an OS thread. is an epoll instance with every socket registered once (edge-triggered). Whenever the scheduler has a moment, or a thread has nothing to do, it calls epoll_wait and marks the goroutines behind the ready fds as runnable.
4. Your goroutine resumes, retries read, gets the data. To your code, conn.Read just took a while.

Async / Await §09 animates this with the scheduler's desks and workers. The point for this topic: one goroutine waiting costs a few KB of memory, not a thread. Here's the proof, 100 goroutines all waiting on pipes that never get data:

package main

import (
	"fmt"
	"os"
	"strings"
	"syscall"
	"time"
)

// threads returns this process's OS thread count, as the kernel reports it.
func threads() string {
	status, _ := os.ReadFile("/proc/self/status")
	for _, line := range strings.Split(string(status), "\n") {
		if strings.HasPrefix(line, "Threads:") {
			return strings.TrimSpace(strings.TrimPrefix(line, "Threads:"))
		}
	}
	return "?"
}

func main() {
	raw := len(os.Args) > 1 && os.Args[1] == "raw" // `go run . raw` picks the blocking version

	var writers []*os.File // keep the write ends alive, so the pipes stay open
	for i := 0; i < 100; i++ {
		r, w, err := os.Pipe() // nobody ever writes: every read waits forever
		if err != nil {
			panic(err)
		}
		writers = append(writers, w)
		go func() {
			buf := make([]byte, 1)
			if raw {
				syscall.Read(int(r.Fd()), buf) // ← blocking syscall: pins an OS thread
			} else {
				r.Read(buf) // ← netpoller: the goroutine parks, the thread moves on
			}
		}()
	}
	time.Sleep(500 * time.Millisecond)
	fmt.Println("100 goroutines waiting; OS threads:", threads())
	_ = writers // still in use here, so the GC can't close the write ends early
}

$ go run .        # r.Read: through the netpoller
100 goroutines waiting; OS threads: 5
$ go run . raw    # raw blocking syscall
100 goroutines waiting; OS threads: 103

When a goroutine makes a syscall that really blocks (the second run), the runtime can't park it: the thread itself is stuck inside the kernel. So before every syscall that might block, the runtime notes “this thread is in a syscall”. If it stays there for more than a short while (sysmon checks every 20 µs when busy, up to every 10 ms when idle), a background monitor thread (sysmon) hands that thread's scheduling slot (the runtime calls it a PP (Go scheduler slot)“Processor”: one of GOMAXPROCS slots in Go's scheduler. An OS thread must hold a P to run goroutines. When a thread gets stuck in a syscall, the runtime gives its P to another thread.) to another thread so other goroutines keep running. Correct, but each stuck call costs a whole OS thread. A thousand slow blocking calls means a thousand threads, and Go stops the program once it reaches 10,000 threads (debug.SetMaxThreads).

Node: libuv's event loop and thread pool

Node's main thread runs the event loop (Async / Await §07). Under the hood, each lap includes one epoll_pwait call (the loop's poll phase), with a timeout until the next timer is due. The strace in §05 showed both halves:

• Sockets are non-blocking and registered with epoll. Reads, writes, connect, accept all happen on the main thread, only when epoll says they won't block.
• Files (and DNS lookups, some crypto, zlib) go to the libuv thread poollibuv thread poolA small pool of background threads (4 by default) inside every Node process. Node uses it for work the OS can't do asynchronously: file system calls, DNS lookups, some crypto and zlib compression.: 4 threads by default, set with the UV_THREADPOOL_SIZE environment variable (max 1,024). A pool thread makes the blocking call, then writes to an eventfdeventfdA Linux file descriptor that is just a counter. One thread writes to it to wake another thread that is sleeping in epoll_wait. Go and Node both use one to interrupt their own event loop. that epoll is watching, to wake the main thread with the result.

That pool is small and shared, and it's easy to fill. Real run: four slow password hashes occupy all four threads, and a 14-byte file read waits behind them:

The program behind it:

import fs from "node:fs/promises";
import crypto from "node:crypto";

const t0 = performance.now();
const ms = () => Math.round(performance.now() - t0);

// pbkdf2 = deliberately slow password hashing (1,000,000 rounds); it runs on the pool
for (let i = 0; i < 4; i++) {          // fill all 4 pool threads
  crypto.pbkdf2("pw", "salt", 1_000_000, 32, "sha256", () => console.log(`hash ${i} done at ${ms()} ms`));
}
await fs.readFile("hello.txt");         // needs a pool thread too
console.log(`readFile done at ${ms()} ms`);

$ node pool.mjs
hash 1 done at 129 ms
readFile done at 131 ms            ← a 14-byte read took 131 ms: it was queued
…
$ UV_THREADPOOL_SIZE=8 node pool.mjs
readFile done at 1 ms              ← a free thread was available
hash 1 done at 121 ms
…

DNS §07 shows the same trap with dns.lookup, which also uses this pool.

Why epoll can't do files

It's tempting to think “just make the file non-blocking and add it to epoll”. The kernel says no, and it's right to:

• Readiness means “won't wait for someone else.” A socket isn't ready until the other machine sends something. A file on disk has no “someone else”: its data is always there, so the kernel considers regular files always ready. epoll_ctl refuses them outright with EPERM, which is exactly the failed line in the Go strace (§05). (O_NONBLOCK is silently ignored on regular files.)
• But reading can still block. If the data isn't in the page cachePage cacheThe kernel's cache of file contents in RAM. Every normal file read and write goes through it, and mmap maps its pages straight into your process. Free RAM fills up with it, which is fine: it's dropped when memory is needed. (the kernel's memory copy of recently used file data), the read sleeps until the disk delivers, maybe 0.1 ms on an SSD, 10 ms on a spinning disk, much more on a network filesystem. Readiness can't predict that.

So on Linux, a readiness API can't make file I/O asynchronous. You need either a thread that's allowed to block (what Go and libuv do) or an API that runs the whole operation for you in the background. That second option is io_uring (§10).

Making fewer syscalls: buffering, batching, zero-copy

Buffering: the measurement

A program writes 100,000 short lines (“line 0”, “line 1”, …, about 1 MB in total) to a file, once directly and once through a bufferUser-space bufferA chunk of your program's own memory where small writes collect until there's enough to send to the kernel in one syscall (Go's bufio.Writer, a Node stream's internal queue).:

package main

import (
	"bufio"
	"fmt"
	"os"
	"time"
)

func main() {
	buffered := len(os.Args) > 1 && os.Args[1] == "buffered"
	f, err := os.Create("out.txt")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	start := time.Now()
	if buffered {
		w := bufio.NewWriter(f) // ← a 4,096-byte buffer in your memory
		for i := 0; i < 100_000; i++ {
			fmt.Fprintf(w, "line %d\n", i) // copies into the buffer: no syscall
		}
		if err := w.Flush(); err != nil { // ← sends what's left. Forget this and the tail is lost!
			panic(err)
		}
	} else {
		for i := 0; i < 100_000; i++ {
			fmt.Fprintf(f, "line %d\n", i) // ← one write() syscall per line
		}
	}
	fmt.Fprintln(os.Stderr, "took", time.Since(start).Round(time.Millisecond)) // ← 1 more write
}

Real results on this machine, with strace -c -e trace=write counting the calls:

(1,088,890 bytes ÷ 4,096 ≈ 266 buffers, plus one write for the “took 7ms” line the program prints to stderr = 267. The Node counts include a few writes of runtime housekeeping.)

~7× faster in Go, ~5× in Node, from changing three lines. The kernel work of copying 1 MB didn't change; what disappeared was 100,000 crossings and 100,000 trips through each runtime's I/O layers. See how the count falls as the buffer grows:

Node streams batch for you: writev

Node's writable streams (fs.createWriteStream, sockets, HTTP responses) already queue small writes while one is in flight, then send the queue with a single writevwritev (vectored write)A syscall that writes several separate buffers in one go — “these 30 chunks, in order” — instead of one write() per chunk. Node's streams use it to send queued chunks together. syscall. Real count for the same 100,000 out.write(…) calls on a file stream: 98 writev + 11 write, not 100,000. You get batching without writing any buffer code, as long as you write through a stream rather than calling fs.writeSync in a loop.

On a network socket you can ask for this explicitly: socket.cork() holds writes, socket.uncork() sends them together. Go's equivalent is a bufio.Writer around the connection, or net.Buffers (which uses writev too).

Batching other calls

The same idea shows up everywhere once you look for it:

• Read in bigger chunks. bufio.NewReader / bufio.Scanner in Go read 4 KB+ at a time even if you consume a line at a time. Node streams read 64 KB chunks for files by default.
• One epoll_wait, many events. Go asks for up to 128 ready fds per call, libuv for up to 1,024 (you saw both numbers in §05).
• Database and Redis pipelining batch many commands into one network write: the same trade at a higher layer.

Zero-copy: let the kernel move it

Serving a file over HTTP the naive way: read 64 KB from the file into your memory, write those 64 KB to the socket, repeat. The bytes cross the boundary twice and you pay two syscalls per chunk. sendfilesendfileA Linux syscall that copies data from a file straight into a socket inside the kernel. Go's io.Copy uses it automatically when copying an *os.File to a TCP connection. tells the kernel “copy from this file fd to that socket fd” and the data never enters your process: zero-copyZero-copyMoving data (e.g. from a file to a socket) without copying it through your program's memory. The kernel moves it directly, saving copies and syscalls. sendfile and splice are the Linux calls for this..

Go does this automatically. io.Copy(conn, file) notices it has an *os.File and a TCP connection and uses sendfile. Real strace of copying a 1 MB file to a socket:

sendfile(4, 7, NULL, 2147483647) = 1000000     ← the whole megabyte in one call
sendfile(4, 7, NULL, 2147483647) = 0           ← 0 = end of file, done

Two syscalls for a megabyte, and no copies through Go's memory. http.ServeFile and http.FileServer get this for free over plain HTTP. (With HTTPS, the bytes must pass through your process to be encrypted, so sendfile can't be used.) Node's core doesn't expose sendfile: fs.createReadStream(path).pipe(res) does read/write in 64 KB chunks, which is fine for most apps; for heavy static files, put nginx (which uses sendfile) or a CDN in front.

io_uring: shared rings, fewer syscalls

Two models: readiness and completion

Everything up to now used the readiness modelReadiness modelThe kernel tells you when an operation would succeed (“fd 7 has data”), and you then make the read/write call yourself. epoll, kqueue, poll and select work this way.: the kernel tells you an fd is ready, then you do the read or write yourself. Two weaknesses fell out of it:

1. Every operation is still its own syscall. epoll saves the waiting, not the doing. 10,000 ready sockets still means 10,000 read calls.
2. Files don't fit (§08). They're always “ready” yet can still block on the disk, so runtimes burn threads on them.

The completion modelCompletion modelYou hand the kernel the whole operation (“read 4 KB from fd 7 into this buffer”), and it tells you when it's finished, with the data already in your buffer. Windows IOCP and io_uring work this way. fixes both: you describe the whole operation (“read 4 KB from fd 7 into this buffer”) and the kernel tells you when it has finished, with the data already in place. Windows has worked this way for decades with IOCPIOCP (I/O completion ports)Windows' high-performance I/O interface. You start operations and later collect “this one finished” notices from a queue: a completion model, unlike Linux's epoll.. Linux got it in 2019 with io_uringio_uringA modern Linux interface (5.1+) where a program puts I/O requests into a shared ring buffer and collects results from another, doing many operations with very few system calls. (kernel 5.1).

The two rings

At setup, your program and the kernel share two ring buffersRing bufferA fixed-size array used as a queue: the read and write positions wrap around to the start when they reach the end, so nothing ever has to be shifted.:

• The submission queueSubmission queue (SQ)In io_uring, a ring buffer shared between your program and the kernel. You write requests into it (“read this file”, “send on this socket”); each request is an SQE, a submission queue entry. (SQ): you write request entries (SQEs) at its tail: an opcode saying which operation (read, write, accept, send, open, fsync…), fd, buffer address, length, and a tag of your choice.
• The completion queueCompletion queue (CQ)In io_uring, the second shared ring buffer. The kernel writes a result into it for each finished request (a CQE, completion queue entry), and your program reads them out without a syscall. (CQ): the kernel writes result entries (CQEs): your tag plus the result, which is the same number the classic syscall would have returned (bytes, or a negative errno).

Writing into the SQ and reading from the CQ are plain memory operations: no syscall. One io_uring_enter call says “I've added N entries” (and can also wait for completions). Try it:

The flow, in C with liburing (the small helper library most io_uring code uses), for illustration:

struct io_uring ring;
io_uring_queue_init(256, &ring, 0);                 // io_uring_setup: create both rings

for (int i = 0; i < 3; i++) {                        // queue 3 reads: NO syscalls yet
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    io_uring_prep_read(sqe, fds[i], bufs[i], 4096, 0);
    io_uring_sqe_set_data(sqe, (void *)(long)i);     // our tag: which request is this?
}
io_uring_submit(&ring);                              // ONE io_uring_enter for all three

for (int done = 0; done < 3; done++) {
    struct io_uring_cqe *cqe;
    io_uring_wait_cqe(&ring, &cqe);                  // sleeps only if the CQ is empty
    printf("req %ld → %d\n", (long)io_uring_cqe_get_data(cqe), cqe->res);
    io_uring_cqe_seen(&ring, cqe);                   // mark the slot free: plain memory write
}

Why it's fast:

• Batching: 100 operations, one io_uring_enter. Under load, completions keep arriving in the CQ while you work, and you read them without asking.
• Files are truly async: the kernel does the read in the background (with its own worker threads if it has to wait on the disk), so you don't need a thread pool.
• Optional zero-syscall mode: with IORING_SETUP_SQPOLL, a kernel thread watches your SQ and picks up new entries itself. You pay for that with a CPU core that keeps spinning while there's work.
• More than read/write: accept, connect, send, open, close, stat, fsync, timeouts, even chains of linked operations (“open, then read, then close”).

Where Go and Node stand (Go 1.24, Node 22)

When is it worth it? When syscall overhead or thread-pool hand-offs dominate: databases and storage engines doing many small random reads, proxies at very high packet rates. For a typical web service whose time goes to waiting on databases and APIs, epoll + a good runtime is already enough, and that's what Go and Node give you today.

In production: profiles, tracing, sandboxes and EINTR

1 · Spotting a syscall-heavy program

Start with top. Its CPU line splits time into us (user: your code) and sy (systemSystem time (%sys)CPU time spent running kernel code on behalf of your program (inside syscalls), as opposed to user time spent in your own code. top shows it as “sy”; a high value often means too many syscalls.: kernel code running on your behalf, mostly inside syscalls). A service at 40% sy is spending a lot of its CPU crossing into the kernel. Per process, pidstat -u 1 shows %usr and %system.

Then profile. Go's profilerProfilerA tool that measures where a program spends its time or memory (e.g. Go's pprof, Chrome DevTools, perf), usually by sampling what's running many times per second. pprof samples the stack many times a second; syscall-heavy code shows up as time in the syscall wrapper. Here is the 1-byte-write benchmark from §04, profiled (real output, trimmed). flat is time spent in that function itself; cum adds the functions it called:

      flat  flat%   sum%        cum   cum%
     430ms 60.56% 60.56%      430ms 60.56%  internal/runtime/syscall.Syscall6
      70ms  9.86% 70.42%       70ms  9.86%  runtime.exitsyscallfast
      50ms  7.04% 77.46%       50ms  7.04%  runtime.casgstatus
      40ms  5.63% 83.10%      710ms   100%  os.(*File).Write
      10ms  1.41% 95.77%       50ms  7.04%  runtime.entersyscall

Syscall6 is the Go runtime's raw syscall function: 60% of the time is the crossing itself. entersyscall/exitsyscall are the runtime's bookkeeping around every syscall that might block (§08). When you see these near the top of a real service's profile, look at cum (time including callees) to find which of your code paths makes the calls, then batch them (§09). In Node, node --cpu-prof writes a profile you open in Chrome DevTools; time spent in native writeSync/readSync frames is the equivalent signal. On Linux, perf top or perf record -g (if installed and permitted; it wasn't on this machine) show kernel functions too, so you can see where inside the kernel the time goes.

2 · Tracing without wrecking performance

strace is built on ptraceptraceThe Linux syscall debuggers and strace use to stop another process and inspect it. strace uses it to pause the traced program at every single syscall, which is why tracing can slow a program down many times over.: the kernel stops the traced thread at every syscall entry and exit and switches to strace so it can read the registers. That's at least two extra context switches per syscall. Real numbers from this machine for the 100,000-write program of §09:

The slowdown scales with the syscall count, so it hits exactly the programs you'd want to investigate. Don't attach strace to a busy production process for more than a few seconds.

eBPFeBPFA way to load small, safety-checked programs into the running Linux kernel to observe (or filter) events such as syscalls, with very low overhead. bpftrace is a one-liner language on top of it. is the production-safe alternative: small programs run inside the kernel at tracepointsTracepointA named hook point built into the kernel, such as “a syscall is starting” or “a read just finished”. Tracing tools (eBPF, perf) attach to it to count or time events. (named hook points built into the kernel), aggregate counts and histograms there, and only send the summary out. No stopping the process. bpftrace gives you one-liners. These need root, a reasonably recent kernel and often extra container privileges; they weren't available on this machine, so the lines below are illustrative:

# Which syscalls does PID 4120 make, and how often? (Ctrl-C prints the counts)
bpftrace -e 'tracepoint:syscalls:sys_enter_* /pid == 4120/ { @[probe] = count(); }'

# How long do read() calls take? A latency histogram in microseconds
bpftrace -e 'tracepoint:syscalls:sys_enter_read /pid == 4120/ { @s[tid] = nsecs; }
             tracepoint:syscalls:sys_exit_read  /@s[tid]/ { @us = hist((nsecs - @s[tid]) / 1000); delete(@s[tid]); }'

perf trace -s -p 4120        # strace-like summary, much lower overhead (perf tools)
syscount -p 4120             # from the bcc tools package: syscall counts per name

3 · Sandboxes: seccomp and containers

seccompseccompA Linux feature that lets a process lock itself (and its children) down to an allow-list of syscalls. A blocked call fails (often with EPERM) or kills the process. Docker and Kubernetes use it to sandbox containers. lets a process install a filter that every later syscall is checked against; the filter is inherited by children and can't be removed. Container runtimes use it: Docker's default profile blocks a few dozen dangerous or rarely needed syscalls (loading kernel modules, rebooting, mount…), and Kubernetes applies the runtime's default when a pod sets securityContext.seccompProfile.type: RuntimeDefault. You can see whether a process is filtered:

$ grep Seccomp /proc/self/status
Seccomp:          0          ← 0 = off, 2 = filtered
Seccomp_filters:  0

What it looks like when it bites: a call that works on your laptop fails in the container with EPERM (“operation not permitted”), or the process is killed outright. A real example from 2021: newer C libraries started using a new syscall, clone3, to create threads. Older Docker seccomp profiles answered EPERM instead of ENOSYS (“no such syscall”, which would have triggered a fallback to the old clone), so programs in containers failed to start threads until the runtimes were updated. Lessons: run your integration tests under the same seccomp profile as production, and when you see an inexplicable EPERM, strace it in the container to find the call.

4 · EINTR: when a signal interrupts a wait

A signalSignal (Unix)A tiny message the OS delivers to a running process, e.g. SIGINT (Ctrl+C), SIGTERM (“please shut down” — what kill and Kubernetes send first), SIGKILL (stop immediately; can't be caught) or SIGQUIT (Ctrl+\, Go prints every goroutine's stack). Send one with kill -TERM <pid>. (SIGTERM, SIGCHLD, …) can arrive while a thread is sleeping inside a slow syscall like epoll_wait, a blocking read on a pipe, or nanosleep. The kernel runs the signal handler, then makes the syscall return early with EINTREINTRThe errno a slow, waiting syscall returns when a signal arrives while it waits: “interrupted, nothing went wrong — call me again”. Code that calls syscalls directly must retry on it.: “interrupted, nothing went wrong, please call again”. Some calls, like a blocking read, are restarted automatically if the handler was installed with SA_RESTARTSA_RESTARTA flag a program can set when it installs a signal handler. With it, some interrupted syscalls (like a blocking read) are restarted automatically instead of failing with EINTR. Pure waits like epoll_wait and nanosleep are never restarted. (Go installs its handlers that way), but pure waits like epoll_wait and nanosleep never are. Code that calls them directly and doesn't retry treats a harmless signal as a failure.

• Go's standard library retries EINTR for you in os, net and friends. But if you call syscall or golang.org/x/sys/unix directly, you must handle it yourself, as in the epoll loop in §07. This got more common in Go 1.14, which started using a signal (SIGURG) to preemptPreemptionForcibly pausing a running task so others get a turn. The OS does this to threads when their time slice ends; Go's runtime does the same to goroutines that hog a CPU for more than ~10 ms. long-running goroutines (interrupt them so others get a turn); the Go 1.14 release notes warn that raw syscalls will see more EINTR errors.
• Node: libuv retries internally; JavaScript code never sees EINTR.
• Child processes and pipes are where it still shows up in practice: a tool written in another language, a C extension, or a raw syscall in a Go program that also runs exec.Command (SIGCHLD arrives whenever a child exits).

for {
	n, err = unix.Read(fd, buf)
	if err != unix.EINTR {
		break // success, or a real error
	}
	// interrupted by a signal: just try again
}

Pitfalls gallery

Each of these has caused real outages or real slowdowns. Most are invisible in code review and obvious under strace -c.

for (const row of rows) {
  console.log("processing", row.id); // a write() per row, blocking the loop
  handle(row);
}

let handled = 0;
for (const row of rows) {
  handle(row);
  handled++;
}
console.log("processed", handled, "rows"); // one line, one write
// for real logs: a logger configured to buffer its output

w := bufio.NewWriter(f)
for _, r := range rows {
	fmt.Fprintln(w, r)
}
return f.Close() // ← the buffer is still in memory: lost

w := bufio.NewWriter(f)
for _, r := range rows {
	fmt.Fprintln(w, r)
}
if err := w.Flush(); err != nil { // ← send the tail, and check it worked
	f.Close()
	return err
}
return f.Close()

// app = an Express server (a popular Node web framework)
app.get("/config", (req, res) => {
  const cfg = fs.readFileSync("config.json"); // blocks every request
  res.type("json").send(cfg);
});

const cfg = fs.readFileSync("config.json"); // once, at startup: fine

app.get("/config", (req, res) => {
  res.type("json").send(cfg);               // or: await fs.promises.readFile(…)
});

Check yourself: predict the outcome

Commit to an answer before you click. Each explanation points back to the section that covers it.

w := bufio.NewWriter(os.Stdout) // 4096-byte buffer
for i := 0; i < 1000; i++ {
    fmt.Fprintln(w, "ok")        // 3 bytes each
}
w.Flush()

var p [2]int
syscall.Pipe(p[:])
syscall.SetNonblock(p[0], true)
n, err := syscall.Read(p[0], make([]byte, 8))
fmt.Println(n, err)

Cheat sheet