Goroutines & the Go Scheduler

You start one by writing go in front of a function call: go sendEmail(user). The call starts, and your code carries on immediately without waiting for it. That's all the syntax there is.

The interesting part is what happens underneath. Your computer only has a few CPU cores, and the operating system runs code on them through threads. A Go program may have 100,000 goroutines. Something has to decide which of those 100,000 runs on which of the few threads, and when. That something is the Go scheduler, built into every Go program (it's part of the Go Runtime).

Here's the map of the page:

Quick reminder of what a Thread is: one line of execution, with its own Call stack (the pile of "which function called which, and their local variables"). Threads are created and switched by the operating system's kernel. Threads vs Processes §03 covers them properly.

A goroutine is also a line of execution with its own call stack. The difference is who manages it and how big it is.

Three differences that matter

How a goroutine's stack grows and moves is a story of its own: Stack & Heap §10 shows it step by step. Here, let's measure the cost.

Measuring it

This program starts n goroutines that all just wait, then reports the time it took and how much extra memory the process asked the OS for:

package main

import (
	"fmt"
	"os"
	"runtime"
	"strconv"
	"sync"
	"time"
)

func main() {
	n, _ := strconv.Atoi(os.Args[1]) // how many goroutines, from the command line

	var before runtime.MemStats
	runtime.ReadMemStats(&before) // snapshot of the runtime's memory counters

	block := make(chan struct{}) // nobody sends on it until the end, so every goroutine waits
	var wg sync.WaitGroup
	start := time.Now()
	for i := 0; i < n; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			<-block // ← wait here (parked: no CPU, no thread)
		}()
	}
	elapsed := time.Since(start)

	var after runtime.MemStats
	runtime.ReadMemStats(&after)
	fmt.Printf("%d goroutines started in %v (%.0f ns each)\n",
		n, elapsed.Round(time.Millisecond), float64(elapsed.Nanoseconds())/float64(n))
	fmt.Printf("memory from the OS: +%d MB (≈ %.1f KB per goroutine)\n",
		(after.Sys-before.Sys)>>20, float64(after.Sys-before.Sys)/float64(n)/1024)

	close(block) // release them all
	wg.Wait()
}

$ go run cost.go 10000
10000 goroutines started in 20ms (2017 ns each)
memory from the OS: +29 MB (≈ 3.0 KB per goroutine)
$ go run cost.go 100000
100000 goroutines started in 224ms (2235 ns each)
memory from the OS: +263 MB (≈ 2.7 KB per goroutine)
$ go run cost.go 1000000
1000000 goroutines started in 3.199s (3199 ns each)
memory from the OS: +2601 MB (≈ 2.7 KB per goroutine)

Read the numbers: about 2.7 KB per goroutine — the 2 KB stack plus the runtime's record for each goroutine. A million waiting goroutines fit in 2.6 GB. Creation took about 2 µs each here, and most of that is the OS handing over fresh memory, which it does in 4 KB chunks called pages. Run a second batch in the same program, reusing that memory, and it drops to 0.6–0.8 µs per goroutine. Timing numbers vary from run to run; the memory per goroutine doesn't.

Now the same question for real OS threads. (Measured with a tiny C program using pthreads, the standard Linux thread API, since Go doesn't let you create raw threads directly.)

Two things to take from this:

• Thread memory is mostly a reservation. 8 MB per thread sounds terrifying, but it's Address space, not RAM. The real cost of an idle thread here was about 24 KB (8 KB of stack actually touched, plus 16 KB the kernel keeps per thread). That's still about 9× a goroutine — and a thread that does real work touches more of its stack.
• The hard limit comes first. Thread creation failed at 32,417 threads on this machine, because every thread needs a kernel ID and the kernel's pid_max was 32,768. You can raise such limits, but 1,000,000 threads isn't a plan. 1,000,000 goroutines is just 2.6 GB.

Starting one

Put go in front of any function call:

package main

import (
	"fmt"
	"time"
)

func say(msg string) {
	fmt.Println(msg)
}

func main() {
	go say("hello from a goroutine") // starts say, doesn't wait for it
	fmt.Println("hello from main")
	time.Sleep(10 * time.Millisecond) // a crude way to wait — see below for the right way
}

$ go run hello.go
hello from main
hello from a goroutine

Notice the order: main printed first, even though the goroutine was started first. go puts the new goroutine in a queue; main keeps running on its thread, and the new goroutine gets its turn a moment later. You can't rely on the order between goroutines unless you make them coordinate.

The trap: main doesn't wait

Remove the Sleep and run it again:

func main() {
	go fmt.Println("hello from a goroutine") // any call can follow go
	fmt.Println("hello from main")
} // main returns → the whole program exits, goroutines or not

$ go run exit.go
hello from main
$ go run exit.go
hello from main
… (same three more times)

The goroutine usually doesn't get to print (once in a while it squeezes its line in first). When main returns, the program exits, and every goroutine still running or waiting simply stops. There's no "wait for background work" step. The Sleep above "worked" only because 10 ms happened to be enough — on a busy machine it might not be.

Waiting properly: sync.WaitGroup

A sync.WaitGroup is a counter. Add(1) before you start a goroutine, Done() when it finishes, and Wait() blocks until the counter is back to zero:

func fetch(id int) {
	time.Sleep(time.Duration(100*id) * time.Millisecond) // pretend to call a slow API
	fmt.Println("fetched", id)
}

func main() {
	start := time.Now()
	var wg sync.WaitGroup
	for id := 1; id <= 3; id++ {
		wg.Add(1) // one more to wait for — BEFORE the go statement
		go func() {
			defer wg.Done() // "I'm finished" — runs however this function ends
			fetch(id)
		}()
	}
	wg.Wait() // blocks until all three called Done
	fmt.Println("all done in", time.Since(start).Round(10*time.Millisecond))
}

$ go run wait.go
fetched 1
fetched 2
fetched 3
all done in 300ms

Three fetches of 100, 200 and 300 ms took 300 ms in total, not 600: they waited at the same time. Here's the timeline:

Go 1.25+: wg.Go

Add, go, defer Done is so common that Go 1.25 added a method that does all three:

var wg sync.WaitGroup
for id := 1; id <= 3; id++ {
	wg.Go(func() { fetch(id) }) // = wg.Add(1); go func() { defer wg.Done(); fetch(id) }()
}
wg.Wait()

Same output as before. Use it whenever your go.mod says go 1.25 or later; it makes the "forgot to Add" bug impossible.

Getting results back

A goroutine has no return value you can collect: x := go f() doesn't compile. To get data out, you either write into a variable each goroutine owns (like results[i] = …) and read it after Wait(), or send it over a Channel. Channels are a whole topic: Channels & select. If the goroutines can fail, errgroup is a WaitGroup that also collects the first error.

Running many goroutines on a few threads is a scheme usually called M:N scheduling (many tasks on a few threads). Go's version of it is described with three letters of its own, the G · M · P model. In the kitchen picture from §01: G is a ticket, M is a cook, P is a stove.

G — the goroutine

A G is a small record the runtime keeps for each goroutine: a pointer to its stack, where in the code it is (the next instruction to run), and its status. At any moment a goroutine is in one of three main states:

A waiting goroutine is parked: it isn't in any run queue and uses no CPU. Whatever it's waiting for (a channel, a timer, the network) keeps a note to put it back in a run queue later.

M — the thread

An M ("machine") is a real OS Thread. The runtime creates them as needed and reuses them; a typical Go server has a dozen or so. Ms are the only things the OS knows about — the OS has no idea goroutines exist.

P — the processor

A P is not hardware; it's a runtime object, best thought of as a slot for running Go code. There are exactly GOMAXPROCS of them (by default, one per CPU core — §08). Each P owns:

• a local run queue of runnable goroutines (up to 256),
• a few private caches (such as a memory cache for small allocations), so the goroutines running on it don't fight over shared data.

The rule that ties them together: an M must hold a P to run Go code. So at most GOMAXPROCS goroutines are running Go code at the same instant. An M without a P is either asleep, or stuck inside a system call (§06).

Watch it run

The simulator below follows these rules. Start with the guided tour: it spawns goroutines, overflows a queue, preempts a long-running goroutine, blocks one in a system call, parks one on the network, and ends with work stealing. Then switch to Free play and break it. (Each P also has a one-goroutine run next slot for the newest goroutine; the tour uses it, and §05 explains why it exists.)

What to notice:

• Spawning is cheap and local. go f() just puts a G on the current P: in its run-next slot, pushing the previous occupant into the local queue (§05). No thread is created unless an idle P needs one to run on.
• A full local queue spills to the global queue. Half of it moves to the Global run queue, so the queue never grows without limit and other Ps can help.
• Threads follow work. A P with nothing to do gives up its thread (the thread sleeps); a P that finds work wakes or creates one.
• A blocking system call costs a thread, not a P (the "handoff" step) — §06 explains how the runtime notices.
• A network wait costs neither. The goroutine parks on the Netpoller and the same thread runs the next one.

The next section zooms in on the queues: in exactly what order a P looks for work, and how stealing works.

Where a P looks for its next goroutine

Every time the goroutine on a P finishes, parks, or is preempted, the P needs another one. The runtime looks in a fixed order:

Why this order? Cheapest first. The local queue belongs to this P alone, so taking from it is nearly free. The global queue is shared, so it needs a Lock (only one P at a time may touch it). Stealing touches other Ps' queues. And the "every 61st turn" rule at the top is about fairness: without it, two goroutines that keep waking each other up could keep a P's local queue busy forever, and goroutines waiting in the global queue would never get a turn.

The "run next" slot

Each P has one more spot besides its queue: a single run next slot. When a goroutine starts another one with go f(), the new goroutine goes into runnext, pushing whatever was there to the back of the local queue. When the P next picks a goroutine, runnext wins.

The idea: a goroutine you just created (or just woke up by sending it a value) is probably about to be needed, and its data is still in the CPU's cache. You can see the effect with one P:

func main() {
	runtime.GOMAXPROCS(1) // one P, so only its queue matters
	var wg sync.WaitGroup
	for i := 0; i < 5; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			fmt.Print(i, " ")
		}()
	}
	wg.Wait() // main parks here; the P picks its next goroutine
	fmt.Println()
}

$ go run runnext.go
4 0 1 2 3
4 0 1 2 3
4 0 1 2 3
4 0 1 2 3
1 0 4 2 3

Goroutine 4 was started last, so it sat in runnext; 0–3 were in the queue in order. When main parked in Wait(), the P ran runnext first, then the queue. And then the fifth run came out differently. The runtime makes no promise about order, and real programs have far more going on. Treat this as a peek inside, never as something to rely on.

Work stealing

When a P's own queue and the global queue are both empty, it becomes a thief. Work stealing works like this:

1. Visit the other Ps in random order (so thieves don't all pile onto P0).
2. Find one with goroutines in its queue and take half of them in one go — not one at a time, so the thief won't have to come back right away.
3. Run one, keep the rest in its own queue.
4. Try up to four rounds. Still nothing? Go to sleep.

A thread that holds a P and is busy hunting for work like this is a Spinning thread. Spinning burns CPU while finding nothing, so the runtime caps it at roughly half the busy Ps; the other idle threads sleep. In the other direction, whenever new work appears and nobody is spinning, the runtime wakes one thread to go looking. Together that means new goroutines get picked up within microseconds, without a crowd of threads burning CPU on empty queues.

"My goroutine is waiting" can mean four different things to the runtime:

The netpoller: why a network wait doesn't block a thread

When you call conn.Read(buf) and no data has arrived yet, Go doesn't let the thread wait in the kernel. Under the hood every network Socket is in non-blocking mode, so the read returns at once with "nothing yet". The runtime then:

1. registers "wake this goroutine when this socket has data" with the Netpoller, which is built on the OS's epoll / kqueue / IOCP (kqueue on macOS),
2. parks the goroutine, and
3. lets the thread pick the next goroutine from its P.

Whenever a P runs out of work (step 4 of §05's list) — and at least every 10 ms via a background check — the runtime asks epoll "which sockets are ready?" and puts those goroutines back in a run queue. Then conn.Read simply returns with the data. To your code it looks like an ordinary blocking call.

Compare that with the alternatives on six connections:

The netpoller gets close to "one thread per connection" speed (140 ms vs 110 ms in this toy) while using one thread. At 6 connections it doesn't matter. At 10,000 it decides whether your server fits in memory. Here's the real thing: 5,000 goroutines, each waiting on its own TCP connection that never sends anything.

package main

import (
	"fmt"
	"net"
	"os"
	"runtime"
	"strings"
	"time"
)

// threads reads this process's OS thread count from Linux (/proc/self/status).
func threads() string {
	status, _ := os.ReadFile("/proc/self/status")
	for _, line := range strings.Split(string(status), "\n") {
		if v, ok := strings.CutPrefix(line, "Threads:"); ok {
			return strings.TrimSpace(v)
		}
	}
	return "?"
}

func main() {
	ln, err := net.Listen("tcp", "127.0.0.1:0") // a local server on any free port
	if err != nil {
		panic(err)
	}
	var keep []net.Conn // hold the server side of each connection open
	go func() {         // the server: accept connections, never send anything
		for {
			conn, err := ln.Accept()
			if err != nil {
				return
			}
			keep = append(keep, conn)
		}
	}()

	const n = 5000
	for i := 0; i < n; i++ {
		conn, err := net.Dial("tcp", ln.Addr().String())
		if err != nil {
			panic(err)
		}
		go func() {
			buf := make([]byte, 1)
			conn.Read(buf) // ← waits forever: the server never writes
		}()
	}
	time.Sleep(time.Second)
	fmt.Println("goroutines:", runtime.NumGoroutine())
	fmt.Println("OS threads:", threads())
}

$ go run netpark.go
goroutines: 5002
OS threads: 8

5,000 goroutines blocked on the network, 8 threads. (5,002 = the 5,000 readers + main + the accept loop. The exact thread count varies a little from run to run. Each connection uses two file descriptors, one per end; if you get "too many open files", raise the limit with ulimit -n 20000 — File Descriptors §07 explains.)

Timers

time.Sleep, time.After, time.Ticker and context.WithTimeout all use runtime timers. Each P keeps its own timers in a heap (a structure that always knows the soonest deadline). When the scheduler looks for work it first fires any timers that are due, making their goroutines runnable. If there's nothing at all to do, a thread waits inside epoll_wait (the system call that asks epoll "wake me when any socket is ready") with a timeout set to the next deadline — so one call wakes it for either "network data arrived" or "a timer is due".

That's why sleeping goroutines are as cheap as network waits. Compare 1,000 goroutines sleeping with time.Sleep against 1,000 asking the kernel to sleep directly:

func main() {
	raw := len(os.Args) > 1 && os.Args[1] == "raw"
	for i := 0; i < 1000; i++ {
		go func() {
			if raw {
				ts := syscall.Timespec{Sec: 5}
				syscall.Nanosleep(&ts, nil) // ← asks the kernel to sleep: the thread is stuck
			} else {
				time.Sleep(5 * time.Second) // ← a runtime timer: the goroutine parks
			}
		}()
	}
	time.Sleep(time.Second)
	fmt.Println("1000 sleeping goroutines; OS threads:", threads()) // threads() as in netpark.go
}

$ go run sleepers.go
1000 sleeping goroutines; OS threads: 6
$ go run sleepers.go raw
1000 sleeping goroutines; OS threads: 1005

Same goroutines, same sleep, 6 threads vs 1,005. The raw system call blocks inside the kernel, and the runtime can't run anything else on a thread that's stuck there.

Blocking system calls: the handoff

So what exactly happens to the P when its thread disappears into a slow System call?

1. Entering the call, the goroutine marks itself as "in a system call" and keeps its P. Most system calls (a write to a pipe, reading the clock) finish in a few microseconds, and handing the P over every time would cost more than it saves.
2. The watchdog notices. sysmon, a runtime thread that runs without a P, wakes up every 20 µs to 10 ms. If a goroutine has been in the same system call for at least one of its ticks, and there's work waiting in that P's queue (or no other P is free to pick up new work), sysmon takes the P away and gives it to another thread — a sleeping one, or a new one. After ~10 ms it does so regardless. That's the Handoff. (A few calls the runtime knows will block for a long time hand off right away.)
3. When the call returns, the thread tries to get a P back: its old one if it's free, otherwise any idle one. If every P is busy, the goroutine goes on the global run queue and the thread goes to sleep.

That's how the raw Nanosleep program ended up with 1,005 threads: 1,000 stuck in the kernel, and a few more holding the Ps. The System Calls topic shows the same effect with pipe reads, and why plain file reads take this path even in normal Go code.

Everything so far relied on goroutines stopping now and then: finishing, waiting on a channel, the network or a timer. Each of those is a moment where the scheduler gets control back. But what about a goroutine that just computes?

for { } // or: crunching numbers in a loop for 5 seconds, never calling anything

If nothing ever interrupts it, it keeps its P, and with GOMAXPROCS=1 nothing else in the program ever runs again. Giving a long-running task's turn to someone else by force is called Preemption.

Before Go 1.14: cooperative, at function calls

Go used to be cooperative (Cooperative scheduling). Recall from Stack & Heap §10 that every Go function starts with a tiny check: "is there room on my stack?" The runtime reused that check. To ask a goroutine to stop, sysmon set a flag that made the check fail; the next time the goroutine called a function, it noticed and yielded its P.

That worked for almost all code, because almost all code calls functions all the time. But a tight loop that never calls anything — or only calls functions the compiler inlined — never reaches a check. Real programs hit this: a loop decoding a big buffer, a busy-wait. The result was a stalled program, or a garbage collection that couldn't start because it needed every goroutine to pause briefly.

Go 1.14+: asynchronous, with a signal

Go 1.14 added Asynchronous preemption. When sysmon sees that a goroutine has been running for more than ~10 ms:

1. It sends the goroutine's thread an OS signal, SIGURG (chosen because hardly any program uses it for anything else).
2. The thread's signal handler, inside the Go runtime, checks whether the exact instruction it interrupted is a Safe point, a spot where the runtime knows precisely what's on the stack. Thanks to extra information the compiler now records, almost every instruction is.
3. If it is, the handler makes the goroutine "call" a small runtime function at that spot, which saves everything and puts the goroutine on the global run queue. The P is free for someone else.

Try the three cases:

func spin() {
    for { } // no function calls inside
}

func main() {
    runtime.GOMAXPROCS(1) // one P
    go spin()
    time.Sleep(10 * time.Millisecond)
    fmt.Println("main got the CPU back")
}

And here's the same program, for real:

func spin() {
	for { // a tight loop: no function calls, no channel operations
	}
}

func main() {
	runtime.GOMAXPROCS(1) // one P: only one goroutine can run at a time
	go spin()
	time.Sleep(10 * time.Millisecond) // main parks; spin gets the only P
	fmt.Println("main got the CPU back")
}

$ time ./spin
main got the CPU back
real    0m0.013s

$ GODEBUG=asyncpreemptoff=1 timeout 5 ./spin   # switch async preemption off
$ echo $?
124

With asynchronous preemption, main gets its turn back after about 13 ms. With it switched off (the GODEBUG setting asyncpreemptoff=1), the program prints nothing and timeout has to kill it after 5 seconds: exit code 124 means "timed out".

Reading and setting it

func main() {
	fmt.Println("CPUs:", runtime.NumCPU())
	fmt.Println("GOMAXPROCS:", runtime.GOMAXPROCS(0)) // 0 = just read it, don't change it
}

$ go run procs.go
CPUs: 4
GOMAXPROCS: 4
$ GOMAXPROCS=2 go run procs.go     # the environment variable overrides the default
CPUs: 4
GOMAXPROCS: 2
$ taskset -c 0 go run procs.go     # only allow CPU 0 (Linux)
CPUs: 1
GOMAXPROCS: 1

taskset restricts which CPUs a process may run on, and Go has always respected that. What it didn't respect, until Go 1.25, was a CPU limit.

The container problem

A Container usually doesn't get a set of dedicated CPUs. It gets a CPU limit: "1.5 CPUs" means "at most 150 ms of CPU time in every 100 ms window, spread over any of the machine's cores". The Linux feature behind it is cgroups. In Kubernetes it's each container's resources.limits.cpu.

Say the container has a 1.5-CPU limit and runs on a 64-core machine. Before Go 1.25, a Go program in it still saw all 64 cores and created 64 Ps. Here's what that does, in numbers:

1. During a busy moment (say, a garbage collection, or a burst of requests) all 64 Ps run at once: that burns 64 ms of CPU time per real millisecond.
2. The 150 ms budget for this 100 ms window is gone after about 2.3 ms.
3. The kernel then pauses the whole container for the remaining ~98 ms of the window. That pause is called CPU throttling.

So a request that needed 5 ms of CPU could take 100 ms because it landed in a pause. With 2 Ps the program can burn at most 2 ms of CPU time per millisecond. If it stays that busy, the 150 ms budget still runs out, but only after ~75 ms, and the pause lasts ~25 ms instead of ~98. Matching GOMAXPROCS to the limit doesn't make throttling impossible. It makes the pauses rarer (you need a longer stretch of overload to hit them) and much shorter.

We checked this on the 4-core test machine with a 1.5-CPU limit: one goroutine per P spinning for 3 seconds, timing every gap in its progress (a gap = the container was frozen). Two runs of each:

GOMAXPROCS=4: frozen 58–67% of the time, longest pause 62–68 ms
GOMAXPROCS=2: frozen 23–24% of the time, longest pause 26–28 ms

That's the ~62 ms and ~25 ms the arithmetic predicts. Before Go 1.25, teams worked around this by setting GOMAXPROCS by hand or adding the go.uber.org/automaxprocs library.

What Go 1.25 changed

From the Go 1.25 release notes, two changes to the default:

1. On Linux, the default respects the cgroup CPU limit. If the limit is lower than the number of CPUs, GOMAXPROCS defaults to the limit. The limit is rounded up to a whole number, and never goes below 2 (from the runtime source). It does not look at Kubernetes CPU requests, only limits.
2. On all systems, the runtime re-checks periodically and updates GOMAXPROCS if the number of CPUs or the limit changes while the program runs.

Both are switched off if you set GOMAXPROCS yourself (the environment variable or a runtime.GOMAXPROCS(n) call), and both are tied to the go line in your Go module's go.mod: a program whose go.mod says go 1.24 keeps the old behavior even when built with a newer Go.

Here's the same procs.go, built with Go 1.27 (go 1.27 in go.mod), on the 4-CPU test machine, inside a cgroup with different CPU limits. This is how the limit was set (as root, on a machine using cgroup v1; docker run --cpus=1.5 sets the same kind of limit):

mkdir /sys/fs/cgroup/cpu/demo
echo 150000 > /sys/fs/cgroup/cpu/demo/cpu.cfs_quota_us   # 150 ms per 100 ms period = 1.5 CPUs
echo $$ > /sys/fs/cgroup/cpu/demo/cgroup.procs           # move this shell into the cgroup
./procs

limit 1.5 CPUs                       → GOMAXPROCS: 2    (rounded up)
limit 0.5 CPUs                       → GOMAXPROCS: 2    (minimum 2)
limit 3 CPUs                         → GOMAXPROCS: 3
limit 1.5, GOMAXPROCS=4 set by hand  → GOMAXPROCS: 4    (you win)
limit 1.5, GODEBUG=containermaxprocs=0 → GOMAXPROCS: 4  (feature off)
no limit                             → GOMAXPROCS: 4
limit 1.5, go.mod says go 1.24      → GOMAXPROCS: 4    (old behavior)

And the periodic update, with a program that prints GOMAXPROCS every 2 seconds while we lower the limit from 3 to 1.5 CPUs:

GOMAXPROCS: 3
GOMAXPROCS: 3
GOMAXPROCS: 3
(limit lowered to 1.5 CPUs)
GOMAXPROCS: 2
GOMAXPROCS: 2

A leak in nine lines

Here's a common pattern: ask three replicas the same question and take whichever answers first.

// firstResult asks three replicas and returns whichever answers first.
func firstResult() (string, error) {
	ch := make(chan string) // unbuffered: a send waits until someone receives
	for _, name := range []string{"a", "b", "c"} {
		go func() {
			time.Sleep(10 * time.Millisecond) // pretend to call replica `name`
			ch <- "answer from " + name       // ← the two losers wait here forever
		}()
	}
	select {
	case r := <-ch:
		return r, nil // we take ONE value and leave
	case <-time.After(time.Second):
		return "", errors.New("timeout")
	}
}

func main() {
	fmt.Println("goroutines at start:", runtime.NumGoroutine())
	for i := 0; i < 1000; i++ { // e.g. 1000 requests to a server
		firstResult()
	}
	time.Sleep(100 * time.Millisecond)
	fmt.Println("goroutines after 1000 calls:", runtime.NumGoroutine())
}

$ go run leak.go
goroutines at start: 1
goroutines after 1000 calls: 2001

Each call receives one value and returns. The other two goroutines are still trying to send on ch, and nobody will ever receive from it again. They're parked forever, each holding its stack and everything its variables point to. In a server, that's two more stuck goroutines per request until the process runs out of memory.

The fixes, from simplest:

• Leave room for every answer: make(chan string, 3). With a buffer of 3, every send succeeds immediately even if nobody receives; the goroutines finish, and the channel is garbage-collected.
• Give the goroutines a stop signal: pass a context.Context and select on ctx.Done() next to the send, then cancel when you return. Channels §08 builds this step by step.

Finding leaks in a running program

1. Watch the count. Export runtime.NumGoroutine() as a metric. A healthy service's count goes up and down with traffic; a leak is a line that only goes up, even at night.

2. Look at where they're stuck: the goroutine profile. The standard pprof package can list every goroutine, grouped by the line where it's waiting. In a server you'd import net/http/pprof and open /debug/pprof/goroutine?debug=1; in a small program you can print it directly:

pprof.Lookup("goroutine").WriteTo(os.Stdout, 1) // from "runtime/pprof"

goroutine profile: total 2001
2000 @ 0x47f06e 0x41491c 0x414517 0x4db129 0x485601
#	0x4db128	main.firstResult.func1+0x68	/tmp/goroutines/leak/main.go:18
…

Read it as: 2,000 goroutines are all stopped at line 18 of main.go (the full file, with its package and import lines, where line 18 is the ch <- send), inside the anonymous function in firstResult (func1). The hex numbers are code addresses; ignore them. With debug=2 you get every goroutine separately, with what it's waiting for:

goroutine 33 [chan send]:
main.firstResult.func1()
	/tmp/goroutines/leak/main.go:18 +0x69
created by main.firstResult in goroutine 1
	/tmp/goroutines/leak/main.go:16 +0xa5

[chan send] is the reason it's parked, and "created by" tells you who started it. When the same line shows up thousands of times, you've found your leak.

3. The leak profile (Go 1.27+). Go 1.27 has a goroutineleak profile, read with pprof.Lookup("goroutineleak") or /debug/pprof/goroutineleak. It uses the garbage collector's view of memory: if a goroutine is waiting on a channel (or mutex, or condition variable) that no running goroutine can reach anymore, nobody can ever wake it, so it's reported as leaked. Same program, with pprof.Lookup("goroutineleak").WriteTo(os.Stdout, 1) as its last line:

$ go run .
goroutines at start: 1
goroutines after 1000 calls: 2001
goroutineleak profile: total 2000
2000 @ 0x47f06a 0x41421c 0x413e17 0x4e0909 0x484ec1
#	0x4e0908	main.firstResult.func1+0x68	/tmp/goroutines/leak127/main.go:18

Unlike the plain goroutine profile, it lists only the stuck ones: the main goroutine isn't counted. It can miss leaks whose channel is still reachable, for example through a global variable or a goroutine that is still running.

Catching leaks in tests

The cheapest place to catch a leak is a test. Without any extra library, compare the goroutine count before and after:

// checkNoLeaks fails the test if goroutines started during it are still alive at the end.
func checkNoLeaks(t *testing.T) {
	before := runtime.NumGoroutine()
	t.Cleanup(func() { // runs when the test finishes
		deadline := time.Now().Add(time.Second)
		for runtime.NumGoroutine() > before { // give stragglers a moment to exit
			if time.Now().After(deadline) {
				t.Errorf("goroutine leak: %d before, %d after", before, runtime.NumGoroutine())
				return
			}
			time.Sleep(10 * time.Millisecond)
		}
	})
}

func TestFirstResult(t *testing.T) {
	checkNoLeaks(t)
	firstResult() // the unbuffered version
}

--- FAIL: TestFirstResult (1.02s)
    fetch_test.go:16: goroutine leak: 2 before, 4 after

This simple version can be confused by goroutines from other tests running in parallel. The popular go.uber.org/goleak library (a third-party dependency) does it properly: it checks the stacks of the goroutines that remain, not just the count. Add defer goleak.VerifyNone(t) to a test, or call goleak.VerifyTestMain(m) once for a whole package.

Or use testing/synctest (standard library, Go 1.25+). testing/synctest runs the test in a "bubble" with a fake clock, so the 10 ms sleeps take no real time — and when the test function returns while goroutines in the bubble are still blocked, it fails:

func TestFirstResult(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		firstResult()
	})
}

--- FAIL: TestFirstResult (0.00s)
panic: deadlock: main bubble goroutine has exited but blocked goroutines remain [recovered, repanicked]

With make(chan string, 3) the same test passes instantly (0.00s): the 10 ms sleeps run on the fake clock, so no real time passes.

The same program, both ways

Get one greeting, then three at once. Each greeting waits 100 ms for a pretend network call.

type User struct{ Name string }

// An ordinary function. It waits, and nobody has to know.
func getUser(id int) User {
	time.Sleep(100 * time.Millisecond) // pretend network call
	return User{Name: fmt.Sprint("user", id)}
}

func greeting(id int) string { // calls a waiting function — still ordinary
	return "hi " + getUser(id).Name
}

func main() {
	start := time.Now()
	fmt.Println(greeting(1)) // one at a time: just call it

	var wg sync.WaitGroup // concurrency is chosen HERE, by the caller
	out := make([]string, 3)
	for i := range 3 { // i = 0, 1, 2
		wg.Go(func() { out[i] = greeting(i + 2) })
	}
	wg.Wait()
	fmt.Println(out, time.Since(start).Round(10*time.Millisecond))
}
// Output: hi user1
//         [hi user2 hi user3 hi user4] 200ms

Both print the same thing and take about 200 ms (100 for the first greeting, 100 for the three together). The difference is in the signatures. In JavaScript, getUser waits, so it must be async; then greeting calls it, so it must be async; and so would every function above it. In Go, getUser and greeting are plain functions, and the only place concurrency shows up is in main, where the caller decided to run three at once.

Function coloring

This spreading is known as Function coloring, after a 2015 blog post that described async functions as "red" and normal ones as "blue". The rules: you can call blue from red freely, but calling red from blue means dealing with a Promise you can't simply wait on. So once one function deep in your code becomes async, every function on the path up to it has to change color too. Async / Await §09 has an animation of the spreading.

Go has one color. A function that does I/O looks exactly like one that doesn't. That's possible because of what gets set aside when a task waits:

A goroutine is a stackful Coroutine: it has a whole call stack of its own, so it can pause at any depth and resume later with every caller's frame intact. A JS async function is stackless: it saves only its own local variables (in a heap object, as a Continuation). It can only pause itself, and only at an await — which is why each caller has to be async too.

JavaScript runs all of this on a single thread with an Event loop: the Event Loop topic walks through it, and Async / Await covers the whole JS model.

Neither is free

Go's choice moves the cost from your code to the runtime: the scheduler, the netpoller, growable stacks and preemption all exist so that plain blocking code is cheap. The price you pay is the last row but one — real parallelism means real data races if two goroutines touch the same memory without coordination (§12).

Which tool for which question:

schedtrace: one line per tick

The GODEBUG setting schedtrace makes the runtime print a scheduler summary to stderr every N milliseconds. Here it is on a program that starts 200 goroutines, each doing about 50 ms of pure CPU work, on 4 Ps:

func work() int { // pure CPU work: about 50 ms on one core of this machine
	x := 0
	for i := range 130_000_000 {
		x += i % 7
	}
	return x
}

func main() {
	start := time.Now()
	var wg sync.WaitGroup
	for i := 0; i < 200; i++ { // 200 CPU-hungry goroutines, only 4 Ps
		wg.Add(1)
		go func() { defer wg.Done(); work() }()
	}
	wg.Wait()
	fmt.Fprintln(os.Stderr, "all done in", time.Since(start).Round(10*time.Millisecond))
}

$ GODEBUG=schedtrace=500 ./busy
SCHED 0ms: gomaxprocs=4 idleprocs=2 threads=4 spinningthreads=1 needspinning=0 idlethreads=0 runqueue=0 [ 1 0 0 0 ] schedticks=[ 0 0 0 0 ]
SCHED 504ms: gomaxprocs=4 idleprocs=0 threads=5 spinningthreads=0 needspinning=1 idlethreads=0 runqueue=7 [ 76 1 76 2 ] schedticks=[ 29 33 31 40 ]
SCHED 1007ms: gomaxprocs=4 idleprocs=0 threads=5 spinningthreads=0 needspinning=1 idlethreads=0 runqueue=24 [ 52 2 51 1 ] schedticks=[ 53 71 56 78 ]
…
all done in 2.42s

200 goroutines × ~50 ms = ~10 s of CPU work, shared by 4 Ps: ~2.5 s, and the run took 2.42 s. How to read the 504 ms line:

• gomaxprocs=4 idleprocs=0 — four Ps, none idle: every core is busy.
• threads=5 — five OS threads in total: four running Go code, plus runtime helpers such as sysmon.
• spinningthreads=0 — nobody is hunting for work; there's plenty.
• runqueue=7 — 7 goroutines in the global queue.
• [ 76 1 76 2 ] — the local queues of P0–P3. With the global queue, 162 goroutines are waiting for a turn and 4 are running. The other 34 have already finished (4 Ps × 500 ms ÷ 50 ms each ≈ 40).
• schedticks=[…] (Go 1.25+) — how many times each P has picked a goroutine so far.

The numbers are a snapshot, and different on every run. Two things to notice anyway. The queues are lopsided (P0 has 76, P1 has 1) — but P1 isn't idle: it's running a goroutine, and when that one is done or preempted, P1 will take from its queue, the global queue, or steal (§05). And over time the total shrinks as goroutines finish. A service whose local queues stay long, run after run, simply has more CPU work than CPUs. Add scheddetail=1 (GODEBUG=schedtrace=1000,scheddetail=1) to get one line per P, per M and per goroutine as well.

runtime/metrics: scheduler counters you can export

For production you want numbers on a dashboard, not text on stderr. The runtime/metrics package exposes them; Go 1.26 added goroutine counts by state and thread counts (still there in 1.27):

func main() {
	for i := 0; i < 100; i++ { // 100 CPU-hungry goroutines on 4 Ps
		go func() {
			for {
			}
		}()
	}
	time.Sleep(100 * time.Millisecond)

	samples := []metrics.Sample{
		{Name: "/sched/goroutines/running:goroutines"},
		{Name: "/sched/goroutines/runnable:goroutines"},
		{Name: "/sched/goroutines/waiting:goroutines"},
		{Name: "/sched/threads/total:threads"},
		{Name: "/sched/goroutines:goroutines"},
	}
	fmt.Println("NumGoroutine:", runtime.NumGoroutine())
	metrics.Read(samples) // fills in each sample's Value
	for _, s := range samples {
		fmt.Printf("%-40s %d\n", s.Name, s.Value.Uint64())
	}
}

NumGoroutine: 101
/sched/goroutines/running:goroutines     4
/sched/goroutines/runnable:goroutines    97
/sched/goroutines/waiting:goroutines     5
/sched/threads/total:threads             5
/sched/goroutines:goroutines             106

4 running (one per P, and one of them is main itself, reading the metrics), 97 runnable, waiting in queues. (The documentation calls the per-state counts approximate and doesn't promise that they add up exactly; here they happen to.) The metrics count 106 goroutines where runtime.NumGoroutine() says 101: the 5 "waiting" ones are the runtime's own background goroutines (for the garbage collector and such), which NumGoroutine leaves out.

The most useful one for latency is older: /sched/latencies:seconds (since Go 1.17), a histogram of how long goroutines sat runnable before getting a P. If that grows, requests are waiting for CPU, not for the database.

go tool trace: every event on a timeline

An Execution trace records every scheduler event: each goroutine created, started, blocked (and on what), unblocked, each P's activity, each GC, each system call. You record one with runtime/trace:

func main() {
	f, err := os.Create("trace.out")
	if err != nil {
		panic(err)
	}
	defer f.Close()
	if err := trace.Start(f); err != nil { // start recording ("runtime/trace")
		panic(err)
	}
	defer trace.Stop()

	// … the code you want to look at …
}

go run trace.go
go tool trace trace.out             # opens a timeline in your browser
go test -trace=trace.out ./mypkg    # or record a test run

The viewer shows one row per P with each goroutine's slices of running time, plus views like "goroutine analysis" (for each goroutine: how long it ran, waited to be scheduled, blocked on the network, on sync, in syscalls). It's the tool for "why was this one request slow?". A web server can serve traces too: net/http/pprof exposes /debug/pprof/trace?seconds=5.

Recording costs some overhead, and a trace of a busy service is big, so you can't leave it on. Go 1.25 added the Flight recorder: trace.NewFlightRecorder(cfg) followed by Start() keeps only the last few seconds in memory, and when something goes wrong (a request took over a second, say) your code calls WriteTo(file) to save exactly those seconds.

Answer before you click. Q1 is covered in §03, Q2 in §06, Q3 in §07, Q4 in §04 and §06, Q5 in §09, Q6 in §08.

func main() {
    go fmt.Println("from the goroutine")
    fmt.Println("from main")
}

func first() string {
    ch := make(chan string)
    for i := 0; i < 3; i++ {
        go func() { ch <- fetch(i) }()
    }
    return <-ch
}
// a server calls first() 1,000 times