Debugging
Debugging is a skill you can learn, not a talent: reproduce the bug, form hypotheses and test them, read stack traces, use debuggers, binary-search with git bisect, and debug production systems you can't pause.
The 30-second version
Most beginners debug by staring at the code, changing something, and re-running to see if the problem went away. Sometimes it works. When it doesn't, an afternoon disappears and you still don't know what's wrong. This page teaches the method that replaces staring, plus the tools that make each step fast.
Collect facts, not opinions. What exactly happens, and what did you expect? “Saturday orders from Vietnam get no weekend discount” is a fact. “Pricing is broken” is an opinion.
The loop above is the whole topic. Everything else on this page makes one of its stages quicker:
Reproduce (§03)
Look (§04–§07)
Halve (§08–§09)
git bisect, or inputs, config and code.What this page assumes
You can read basic JavaScript and a little Go. If Go's syntax is new, the 90-second primer in Async / Await §02 covers what you need. You should know that a commit is a saved snapshot of your project; Git, From the Inside explains the rest. A few examples use a terminal; The Command Line covers the basics.
All the program output on this page is real. It came from running each example with Go 1.26 and Node 22 (the version installed where these examples were run). Newer Node versions print the same lines for your own code; only the line numbers inside Node's own node:internal files differ.
Guess, predict, test: debugging as science
Step 1: write down what you actually know
Start with facts, stated precisely. "Checkout is broken" is useless. "Since Friday, 37 customers in Vietnam got no weekend discount on Saturday orders; customers elsewhere are fine" already points somewhere. Separate what you saw from what you assume: "the discount code hasn't changed" is an assumption until you've checked the history.
Step 2: list several guesses, not one
A hypothesis is a specific, checkable guess about the cause. The trap is falling in love with your first one and spending an hour proving it. Write down three to five guesses, including boring ones (stale cache, wrong config, old app version). The more rivals you have, the easier it is to design an experiment that splits them.
Step 3: for each experiment, predict before you run
Before you run anything, ask of each guess: if this were the cause, what would I see? If every guess predicts the same result, the experiment teaches you nothing, however clever it is. If the guesses predict different results, one run crosses several off at once.
Try it. The bug report is real-world shaped: a weekend discount that doesn't apply for some customers. One fact you need: the servers' clocks are set to UTC, the world's reference time zone, which is 7 hours behind Hanoi. You have five guesses and seven possible experiments, each with a cost.
| If this guess is true… | …you'd expect |
|---|---|
| The discount code is broken | spread over the whole day |
| The weekday is worked out in UTC | only between 00:00 and 06:59 |
| A stale cache serves old prices | spread over the day |
| Big orders skip the discount | spread over the day |
| The old mobile app sends bad data | spread over the day |
Three lessons hide in that widget:
- Look at the data you already have first. The 3-minute query split the guesses better than any code change. Logs, timestamps, error rates and "what changed recently" are free experiments.
- Changing code isn't an experiment unless it tests a prediction. The four-hour rewrite might make the symptom go away, but you'd learn nothing about why, and you could add new bugs. (We'll see this exact time-zone bug in code in §10.)
- Confirm the last guess directly. "Everything else was ruled out" isn't proof. Once you think you know the cause, make a change based on it and check that it fixes the real failing cases.
Keep a lab notebook
Write down every experiment, its result and what it ruled out: a text file, a comment on the ticket, anything. It feels slow, but it stops the worst time-waster in debugging: testing the same guess twice because you forgot you already ruled it out. It also makes asking for help easy: you can paste the notebook instead of saying "I tried everything".
14:02 37 complaints, all VN, all Saturday. Others fine.
14:05 query: all 37 placed 00:10–06:55 Sat (Hanoi). → not the discount code, not cache, not app, not big orders
14:09 server TZ = UTC. 01:30 Sat Hanoi = 18:30 Fri UTC. → weekday computed in UTC?
14:30 patched weekday to use customer TZ, replayed 37 orders: 37/37 discounted. ✔When you're stuck: explain it out loud
If you've run out of guesses, explain the code, line by line, to a colleague. No colleague around? Explain it to a rubber duck on your desk: that's rubber-duck debugging, and it works because speaking forces you to check each step instead of skimming it. You'll often stop mid-sentence: "…and then this returns the user, which… oh. It can return null."
Reproduce it, then shrink it
A reproduction ("repro") is a reliable recipe for the failure: these steps, with this input, on this version, give this wrong result. With a repro, every guess from §02 becomes a quick experiment. Without one, you're guessing and waiting for the next customer complaint to find out if you were right.
Capture everything that could differ
"Works on my machine" means some input differs between your machine and the failing one. Collect them before you start:
| Ask | Why |
|---|---|
| Exact input: request body, file, user ID, URL | The bug may only hit one kind of data: a name with no space, an empty list, a guest with no account. |
| Exact version: commit hash, app version, dependency versions | You might be debugging code that isn't what's running. |
| Environment: OS, Time zone, locale, environment variables, config files | A server set to UTC and a laptop set to Hanoi time see different dates. |
| Timing and load: how often, since when, under what traffic | "Fails 1 in 50" and "fails since Tuesday's deploy" point at very different causes. |
| The full error: the whole message and Stack trace, not a summary | "It crashed" loses the one line that says where. |
Turn the recipe into code
Clicking through an app to reproduce a bug takes minutes and goes slightly differently each time. A small test or script takes milliseconds and does the same thing every time.
Here's the example we'll use through §04 and §12: a tiny shop. An order is a list of item prices in cents plus an optional customer. total(order) adds up the prices and adds shipping: 300 cents if the customer is in Vietnam, 1,500 otherwise. Guest checkouts have no customer at all, and in production they crash. (§04 shows the full code and the crash.)
type Order struct {
ID int
Customer *Customer // the buyer's name and country; nil for guest checkouts
Items []int // prices in cents
}Captured as a test, the crash fails exactly like production:
// Reproduces the crash: a guest order has no Customer.
func TestTotalGuestCheckout(t *testing.T) {
o := &Order{ID: 2, Items: []int{500}} // ← the exact input that crashed
if got := total(o); got != 2000 {
t.Errorf("total = %d, want 2000 (500 + 1500 shipping)", got)
}
}Run it with go test or node --test. Both fail with the same crash as production. Now every experiment takes one command, and when you're done, this repro becomes your Regression test (§12).
Shrink it
A repro that needs your whole app, a database and 16 setup steps still hides the bug among a lot of noise. Cut it down: remove a step, a field, a line of input, a middleware, and check it still fails. Stop when removing anything more makes it pass. What's left is a minimal reproduction, and it usually points straight at the cause: if the only thing left is "an order with no customer", you know where to look.
The fastest way to shrink is to remove half at a time; §09 has a widget for exactly that. Shrinking is also what you do before asking for help or filing a bug against a library: maintainers answer a 10-line repro in an hour and a "my app crashes" issue never.
Reading a stack trace
The call stack, in one picture
When main calls total, and total calls shippingFor, the runtime keeps a list of the calls in progress: the Call stack. Each entry is a Stack frame: which function, and which line of it is running. The newest call sits on top, like a stack of plates:
When something goes wrong, the runtime prints that list, top to bottom. That's all a stack trace is.
A crash in both languages
Here's a tiny shop. Each order's total is its items plus shipping, and shipping depends on the customer's country. One order is a guest checkout, with no customer at all:
type Customer struct {
Name string
Country string
}
type Order struct {
ID int
Customer *Customer // nil for guest checkouts
Items []int // prices in cents
}
func shippingFor(o *Order) int {
if o.Customer.Country == "VN" { // line 17
return 300
}
return 1500
}
func total(o *Order) int {
sum := 0
for _, p := range o.Items {
sum += p
}
return sum + shippingFor(o) // line 28
}
func main() {
orders := []*Order{
{ID: 1, Customer: &Customer{"An", "VN"}, Items: []int{1200, 800}},
{ID: 2, Items: []int{500}}, // guest checkout: no customer
}
for _, o := range orders {
fmt.Println("order", o.ID, "total", total(o)) // line 37
}
}(The import "fmt" and package main lines are left out of the Go file to save space; the line numbers are from the full file.) Order 1 prints, then order 2 crashes. Go calls this a panic / recover; JavaScript throws a TypeError. Both are a nil pointer dereference: reading a field through something that points at nothing.
$ go run .
order 1 total 2300
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x49e146]
goroutine 1 [running]:
main.shippingFor(...)
/tmp/debugging/shop/main.go:17
main.total(...)
/tmp/debugging/shop/main.go:28
main.main()
/tmp/debugging/shop/main.go:37 +0x206
exit status 2How to read one, in four moves
- Read the message. "nil pointer dereference" / "Cannot read properties of undefined (reading 'country')" already narrows it to: something before
.Countryis missing. - Find the topmost frame from your code. Frames are newest-first. Skip frames from the Runtime, the standard library and
node:internal; the first frame from your own files is where your code was when it broke. - Open that file at that line, and ask which value on it could be the bad one. On line 17, the only pointer followed is
o.Customer. - Read down the stack to answer "how did that value get here?"
mainpassed order 2, a guest checkout, tototal, which passed it toshippingFor.
Go writes each frame as two lines (function, then file:line). Node writes one line per frame: at function (file:line:column). Now try it on four real traces, including one where your code is buried under library frames.
The second trace comes from this small web server. Requesting /greet?name=Cher crashes the handler, because lastName assumes every name has a space. The file is shown in full, so the line numbers match the trace:
package main
import (
"net/http"
"strings"
)
func lastName(full string) string {
parts := strings.Split(full, " ")
return parts[1] // ← line 10: assumes every name has a space
}
func greet(w http.ResponseWriter, r *http.Request) {
name := r.URL.Query().Get("name")
w.Write([]byte("Hello, " + lastName(name))) // line 15
}
func main() {
http.HandleFunc("/greet", greet)
http.ListenAndServe("127.0.0.1:8089", nil)
}The last two traces are Node programs from §05: greet.mjs has the same lastName bug, reached through await.
The bug is often not on the crashing line
The trace tells you where the bad value was used, not where it came from. Line 17 is fine for every order that has a customer. The real question is a design one: guest orders have no customer, and nobody wrote code for that case. Fixing it might mean a nil check on line 17, or giving guests a country some other way. Always ask "where did this value come from?" before patching the line that crashed.
When the trace isn't enough
Go: one goroutine, and who started it
By default a Go panic prints only the Goroutine that crashed. Its stack starts at the goroutine's first function, so the trace can't show what led up to it. Go fills the gap with a final created by line naming the go statement that launched it. From the HTTP example in §04:
net/http.(*conn).serve(0xd481cbc21b0, {0x72c9f8, 0xd481cb8cff0})
/usr/local/go/src/net/http/server.go:2073 +0x650
created by net/http.(*Server).Serve in goroutine 1
/usr/local/go/src/net/http/server.go:3464 +0x485Read it as: "this goroutine was started by Server.Serve, which was running in goroutine 1, at line 3464". In your own code, it names the go worker(…) line that started the crashing worker. To see every goroutine at the moment of a crash, set GOTRACEBACK=all in the environment before starting the program. That's how you spot the other goroutine that was halfway through changing the same data.
Go errors carry no trace at all
A panic prints a trace. An ordinary error value doesn't: it's just a message. If every layer returns err unchanged, you end up with a log line like invalid character 'p' looking for beginning of object key string, and no idea which of your twenty JSON files it came from. The fix is wrapping: each layer adds what it knows.
func loadConfig(path string) error {
var cfg map[string]any
data := []byte("{ port: 8080 }") // pretend we read this from path
if err := json.Unmarshal(data, &cfg); err != nil {
return fmt.Errorf("load config %s: %w", path, err) // ← add context, keep the original
}
return nil
}
func main() {
if err := loadConfig("app.json"); err != nil {
fmt.Fprintln(os.Stderr, "error:", err) // print to stderr…
os.Exit(1) // …and exit with code 1
}
}$ go run .
error: load config app.json: invalid character 'p' looking for beginning of object key string
exit status 1%w puts the original error inside the new one, so errors.Is can still find it (Go idioms: wrapping has the details). Good wrapping reads like a path: checkout order 42: charge card: POST /payments: timeout. Each prefix answers "where was I, and with what?", which a stack trace wouldn't tell you anyway.
JavaScript: async traces work through await, not callbacks
When an async function resumes after await, it runs from the Event loop: the functions that originally called it are no longer on the call stack. Since Node 12, V8 (the JavaScript engine inside Node and Chrome) rebuilds those callers by following the chain of awaits, and marks them async:
import { setTimeout as sleep } from "node:timers/promises";
async function findUser(id) {
await sleep(10); // pretend this is a database call
return id === 1 ? { name: "Nguyen An" } : { name: "Cher" };
}
function lastName(full) {
return full.split(" ")[1].toUpperCase(); // ← assumes a space
}
async function greet(id) {
const user = await findUser(id);
return "Hello, " + lastName(user.name);
}
async function main() {
console.log(await greet(1));
console.log(await greet(2));
}
main();TypeError: Cannot read properties of undefined (reading 'toUpperCase')
at lastName (file:///tmp/debugging/webjs/greet.mjs:9:28)
at greet (file:///tmp/debugging/webjs/greet.mjs:14:22)
at async main (file:///tmp/debugging/webjs/greet.mjs:19:15)(The file is shown in full, so you can check the line numbers. sleep is the promise version of setTimeout from node:timers/promises.) That rebuilding only follows await. A plain callback, like setTimeout(fn) or an event handler, cuts the chain. Here the same bug runs inside a timer callback:
function lastName(full) {
return full.split(" ")[1].toUpperCase();
}
function greetLater(name) {
setTimeout(() => {
console.log("Hello, " + lastName(name));
}, 10);
}
greetLater("Cher");TypeError: Cannot read properties of undefined (reading 'toUpperCase')
at lastName (file:///tmp/debugging/webjs/lost.mjs:2:28)
at Timeout._onTimeout (file:///tmp/debugging/webjs/lost.mjs:7:29)
at listOnTimeout (node:internal/timers:585:17)
at process.processTimers (node:internal/timers:521:7)greetLater is gone. The trace ends in Node's timer code, because by the time the callback ran, greetLater had long returned. This is one more reason to prefer await over callbacks (Async / Await: errors covers the rest). Browser DevTools can stitch callback chains back together while you're debugging, but plain Error.stack text can't.
Keep the original error: cause
Re-throwing a new error loses the old trace, unless you attach it. Since ES2022 (Node 16.9+), new Error(msg, { cause }) does that, and Node prints both:
async function loadConfig(path) {
try {
JSON.parse("{ port: 8080 }"); // pretend we read this from path
} catch (err) {
throw new Error(`load config ${path}`, { cause: err }); // ← keep the original
}
}
try {
await loadConfig("app.json");
} catch (err) {
console.error(err);
}Error: load config app.json
at loadConfig (file:///tmp/debugging/webjs/cause.mjs:5:11)
at file:///tmp/debugging/webjs/cause.mjs:10:9
... 2 lines matching cause stack trace ...
at async asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:117:5) {
[cause]: SyntaxError: Expected property name or '}' in JSON at position 2 (line 1 column 3)
at JSON.parse (<anonymous>)
at loadConfig (file:///tmp/debugging/webjs/cause.mjs:3:10)
...
}It's the JavaScript twin of Go's %w: your context on top, the original error and its trace underneath.
Minified code and source maps
Production JavaScript is usually bundled and minified: one long line with short names. The same greet.mjs, minified by a Bundler, crashes like this:
$ node dist/app.min.mjs
TypeError: Cannot read properties of undefined (reading 'toUpperCase')
at n (file:///tmp/debugging/webjs/dist/app.min.mjs:1:165)
at r (file:///tmp/debugging/webjs/dist/app.min.mjs:1:216)
at async i (file:///tmp/debugging/webjs/dist/app.min.mjs:1:292)Line 1, column 165, of function n. Useless. A Source map is a file the bundler writes next to the output that maps each position back to your original file and line. Node uses it when you pass --enable-source-maps:
$ node --enable-source-maps dist/app.min.mjs
TypeError: Cannot read properties of undefined (reading 'toUpperCase')
at lastName (/tmp/debugging/webjs/greet.mjs:9:28)
at r (/tmp/debugging/webjs/greet.mjs:14:22)
at async i (/tmp/debugging/webjs/greet.mjs:19:15)The files and line numbers are back, which is what matters. Some function names stay minified (r, i), depending on how much naming information the bundler put in the map. Browsers apply source maps automatically in DevTools. Error-tracking services need you to upload the maps at deploy time, since you usually don't serve them to the public. Build tools: source maps explains how they're made.
Print debugging, done well
o.Customer is nil here." Print exactly what tests the guess, with a label, and delete it when you're done.Print debugging gets a bad name, but it's often the right tool. It works everywhere (in tests, in CI, on a server you can't attach a debugger to), it shows the order things happened in, and it's quick to add. What makes it slow is printing randomly: console.log("here"), console.log("here2"), then scrolling through 400 lines of output hoping something jumps out.
Print values, with labels and types
The default way of printing a value often hides exactly what you need. Both languages have better options:
func shippingFor(o *Order) int {
log.Printf("shippingFor: %+v", o) // ← field names and values
log.Printf("shippingFor: %#v", o.Customer) // ← Go syntax, shows the type
debug.PrintStack() // ← "who called me?"
// …
}
func main() {
log.SetFlags(log.Ltime | log.Lshortfile) // ← add file:line to every log line
shippingFor(&Order{ID: 2, Items: []int{500}})
}08:14:00 main.go:20: shippingFor: &{ID:2 Customer:<nil> Items:[500]}
08:14:00 main.go:21: shippingFor: (*main.Customer)(nil)
goroutine 1 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:26 +0x5e
runtime/debug.PrintStack()
/usr/local/go/src/runtime/debug/stack.go:18 +0x13
main.shippingFor(0x31bf30c82420)
/tmp/debugging/prints/main.go:22 +0x105
main.main()
/tmp/debugging/prints/main.go:28 +0x88(The Go file also has the imports and the Order types from §04; line numbers are from the full file.) Notice the plain console.log hid the nested data as [Object], and console.dir(…, { depth: null }) showed it. (The trace lines below the … are Node internals, as in §04.)
Habits that make prints fast
- One guess, one print. Before adding it, say what you expect to see. If the output matches, the guess survives; if not, it's ruled out.
- Print at boundaries. Where data enters a function, where it's returned, before and after a call to another service. Halfway through a 50-line function rarely helps.
- Print the type and the edges.
"42"and42look the same in a plain print;%#v,%q,JSON.stringify(x)andtypeof xdon't. Print the length of lists, not just the first item. - Make each line findable. Prefix temporary prints with something unique like
DEBUG-42:so a quick search (grep -rn DEBUG-42 .) finds every one to remove before you commit.
Browser consoles show objects live
In Chrome DevTools, console.log(order) shows a collapsed object. When you expand it later, the browser reads the object's values at that moment, not when the line ran. If the code changed order in between, you see the new values and draw the wrong conclusion. Log a copy (console.log(structuredClone(order)) or JSON.stringify(order)) when values change over time.
Debuggers: pause, look, step
You set a Breakpoint on a line. When the program reaches it, it pauses before running that line. While it's paused, you can see every local variable, the whole Call stack, and the result of any expression you type. Then you choose how far to go next:
| Control | What it does | Delve | VS Code | Chrome DevTools |
|---|---|---|---|---|
| Continue | Run until the next breakpoint | continue (c) | F5 | F8 |
| Step over | Run this line, stop at the next one | next (n) | F10 | F10 |
| Step into | Go inside the function called on this line | step (s) | F11 | F11 |
| Step out | Finish this function, stop in its caller | stepout (so) | Shift+F11 | Shift+F11 |
These are the stepping controls. Here they are in action, on a bug whose cause you can't see by reading quickly: the average of 90, 70 and 80 comes out as 50.
1package main23import "fmt"45func average(xs []int) int {6 sum := 07 for i := 1; i < len(xs); i++ {8 sum += xs[i]9 }10 return sum / len(xs)11}1213func main() {14 scores := []int{90, 70, 80}15 fmt.Println("average:", average(scores))16}
average of 90, 70 and 80 to be 80, but the program prints 50. We set a breakpoint on line 14 and start the program. It runs until line 14, then pauses before running it. Nothing on line 14 has happened yet, so scores doesn't exist yet.The same session in Delve
Delve is Go's debugger. Install it with go install github.com/go-delve/delve/cmd/dlv@latest. (As of October 2026, Delve 1.27 needs Go 1.25 or newer; with Go 1.24 it refuses to start.) dlv debug compiles your package with optimizations off and starts it paused. Here's a real session on the program above:
$ dlv debug
Type 'help' for list of commands.
(dlv) break main.go:8
Breakpoint 1 set at 0x4bc989 for main.average() ./main.go:8
(dlv) continue
> [Breakpoint 1] main.average() ./main.go:8 (hits goroutine(1):1 total:1) (PC: 0x4bc989)
=> 8: sum += xs[i]
(dlv) locals
sum = 0
i = 1
(dlv) print xs[i]
70
(dlv) stack
0 0x00000000004bc989 in main.average
at ./main.go:8
1 0x00000000004bca73 in main.main
at ./main.go:15
2 0x0000000000452547 in runtime.main
at /usr/local/go/src/runtime/proc.go:290
3 0x000000000048a0a1 in runtime.goexit
at /usr/local/go/src/runtime/asm_amd64.s:1771
(dlv) clearall
Breakpoint 1 cleared at 0x4bc989 for main.average() ./main.go:8
(dlv) break main.go:10
Breakpoint 2 set at 0x4bc9bc for main.average() ./main.go:10
(dlv) continue
> [Breakpoint 2] main.average() ./main.go:10 (hits goroutine(1):1 total:1) (PC: 0x4bc9bc)
=> 10: return sum / len(xs)
(dlv) locals
sum = 150(Delve also prints a few lines of source around the current line after each stop; they're trimmed here, and the Go installation path is shortened to /usr/local/go.) locals lists local variables (the function's parameters, here xs, are listed by args instead), print evaluates any expression, stack is the same call stack a panic would print, and clearall removes every breakpoint. To debug a test instead of main, use dlv test -- -test.run '^TestTotal$'.
In VS Code, the Go extension runs Delve for you: click left of a line number to set a breakpoint, then press F5. GoLand has the same thing built in.
Skip the boring iterations: conditions, logpoints, watches
A breakpoint inside a loop that runs 10,000 times is useless if the bug only shows on item 9,412. Three tools fix that:
- A conditional breakpoint only pauses when an expression is true. In Delve:
cond 1 i == 2. In VS Code and DevTools: right-click the breakpoint → Edit condition. - A logpoint prints a message instead of pausing: a print statement you add without editing the file or restarting.
- A watch expression is re-evaluated at every pause, so you see
sumororder.items.lengthchange as you step.
(dlv) break main.go:8
Breakpoint 1 set at 0x4bc989 for main.average() ./main.go:8
(dlv) cond 1 i == 2
(dlv) continue
> [Breakpoint 1] main.average() ./main.go:8 (hits goroutine(1):1 total:1) (PC: 0x4bc989)
=> 8: sum += xs[i]
(dlv) locals
sum = 70
i = 2Debugging Node
Node has a debugger built in, the inspector. Start your program with --inspect-brk so it pauses on the first line and waits for you to connect:
node --inspect-brk avg.mjs
# then open chrome://inspect in Chrome and click "inspect" under your script,
# or in VS Code: open a "JavaScript Debug Terminal" and just run `node avg.mjs`Either way you get the same breakpoints, stepping, variables pane and console as in the browser's DevTools. You can also write debugger; on a line in your code: when an inspector is attached, execution pauses there, and when none is attached the statement does nothing. (Remove it before committing; linters flag it.)
There's also a bare-bones terminal debugger, node inspect. A trimmed real session on the JavaScript version of average:
$ node inspect avg.mjs
Break on start in avg.mjs:1
> 1 function average(xs) {
debug> setBreakpoint(4)
debug> cont
break in avg.mjs:4
3 for (let i = 1; i < xs.length; i++) {
> 4 sum += xs[i];
debug> exec i
1
debug> watch("sum")
debug> cont
break in avg.mjs:4
Watchers:
0: sum = 70
debug> backtrace
#0 average avg.mjs:4:4
#1 (anonymous) avg.mjs:10:24When the debugger misleads you
- "Optimized out" variables. In an optimized build, the compiler may delete or merge variables.
dlv debuganddlv testturn optimizations off for you; for a binary you build yourself, usego build -gcflags=all="-N -l". - Pausing changes timing. While one goroutine sits at a breakpoint, the others keep running, or time out. A race between two goroutines may never happen while you're watching (§10). Delve's
goroutinescommand lists them all, andgoroutine 7switches to one. - You can't pause production. A paused server serves nobody, and its clients time out. §11 covers the tools that work on live systems.
git bisect: binary search through history
You don't need to understand the bug to find the commit that caused it. You only need a way to tell a good build from a bad one. That's why bisecting is so powerful on unfamiliar code: once you have the one commit that broke things, you have a small diff to read instead of the whole codebase.
The idea: binary search over commits
It's the same Binary search you'd use to find a word in a dictionary. Check out the commit in the middle of the range and test it. If it's good, the bug arrived later, so the first half is cleared. If it's bad, the bug arrived at or before it, so the second half is cleared. Each test halves the suspects. Run it yourself:
Discount(5000) should be 500.A real session, by hand
Here's that exact history in a real repo: 16 commits, with a Discount function that gave 10% off orders of 5,000 cents or more. Someone refactored it, and now an order of exactly 5,000 gets nothing. The tag v1.3 is known good; main is bad.
$ git bisect start
status: waiting for both good and bad commits
$ git bisect bad # the commit you're on (main) is broken
status: waiting for good commit(s), bad commit known
$ git bisect good v1.3 # this one worked
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[2936e1a434d6aa90044bba2b1d28b20322939758] Update docsgit has checked out the commit in the middle. Your files now look exactly as they did then. Build, test, and report:
$ go test -run TestDiscountAtThreshold . # passes here
$ git bisect good
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[228ae6f622fa61b82f00c93982cbf6c82fc26be0] Add Vietnamese strings
$ go test -run TestDiscountAtThreshold . # fails here
$ git bisect bad
Bisecting: 1 revision left to test after this (roughly 1 step)
[8754b75f1e3dedf494a7859fd6f3d3c1d8597d0b] Speed up totals
$ git bisect good
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[4306cdbf6ce6d913e46d7ed73282036e4b8ddf19] Refactor discount threshold
$ git bisect bad
4306cdbf6ce6d913e46d7ed73282036e4b8ddf19 is the first bad commit
commit 4306cdbf6ce6d913e46d7ed73282036e4b8ddf19
Author: Dev <dev@example.com>
Date: Sun Oct 11 10:00:00 2026 +0000
Refactor discount threshold
price.go | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
$ git bisect reset # go back to where you started (main)Four tests among 15 suspects. Now read just that commit's diff with git show 4306cdb:
-// Discount returns the discount in cents: 10% off orders of 5000 cents or more.
+const minTotal = 5000 // orders of at least this many cents get 10% off
+
+// Discount returns the discount in cents.
func Discount(total int) int {
- if total >= 5000 {
+ if total > minTotal {
return total / 10
}>= became >. A classic off-by-one, hidden inside a refactor that looked harmless. Nobody would have spotted it by reading 15 commit messages.
Let a script do the testing: git bisect run
Marking by hand is slow, and one wrong answer sends the search to the wrong place. If a script can tell good from bad, give it to git. It runs the script at each step and reads the script's Exit code: 0 means good, 1 to 127 means bad, and 125 means "can't test this one, skip it" (for example, a commit that doesn't compile).
Old commits don't contain the test you just wrote, so the script copies it in from outside the repo:
#!/bin/sh
# Exit 0 = this commit is good, 1 = bad. git bisect run reads the exit code.
go build ./... || exit 125 # doesn't compile? skip this commit
cp /tmp/debugging/tools/discount_test.go .
go test -run TestDiscountAtThreshold . > /dev/null
status=$?
rm discount_test.go
exit $status$ git bisect start main v1.3 # bad first, then good
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[2936e1a434d6aa90044bba2b1d28b20322939758] Update docs
$ git bisect run /tmp/debugging/tools/check.sh
running '/tmp/debugging/tools/check.sh'
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[228ae6f622fa61b82f00c93982cbf6c82fc26be0] Add Vietnamese strings
running '/tmp/debugging/tools/check.sh'
Bisecting: 1 revision left to test after this (roughly 1 step)
[8754b75f1e3dedf494a7859fd6f3d3c1d8597d0b] Speed up totals
running '/tmp/debugging/tools/check.sh'
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[4306cdbf6ce6d913e46d7ed73282036e4b8ddf19] Refactor discount threshold
running '/tmp/debugging/tools/check.sh'
4306cdbf6ce6d913e46d7ed73282036e4b8ddf19 is the first bad commit
…
bisect found first bad commit
$ git bisect reset(The … stands for the same commit details as before.) The script ran four times and found the same commit, with no chance of a wrong answer. For a JavaScript project, the script might run node --test discount.test.mjs, or npm test if the test already exists in the old commits. Any command whose exit code means good or bad works, including curl-ing a local server or comparing a program's output with grep.
Tips from people who bisect a lot
- Bisect only needs a yes/no. It can find when something got slow (a script that fails above 200 ms) or when an output changed, not just crashes.
- Use
git bisect skip(or exit 125) on commits that don't build. git picks a nearby commit instead. Too many skips and it can only tell you "one of these few". - Keep commits small and buildable. Bisect points at one commit. A 2,000-line "WIP" commit is a much less useful answer than a 20-line one.
git bisect logprints your answers so far, andgit bisect replayre-runs them, so a wrong answer doesn't mean starting over.- Merges work too: bisect handles branching history, though it may land you on a merge commit, and then you look at what the merge brought in.
Halve anything: inputs, config, code
Shrinking a failing input
An import of 16 customer rows crashes. Which row is to blame? Testing them one at a time takes up to 16 runs. Halving takes about 4, because you only need to test one half each round: if the first half passes, the bad row must be in the other half. Try it, then switch to the second scenario, where the bug needs two rows at once:
The second scenario shows the catch with plain halving: when the bug needs pieces from both halves, neither half fails on its own. The fix is to remove smaller chunks (a quarter, then an eighth) and keep any removal that still fails. That smarter version has a name, delta debugging, and tools do it for you. Go's built-in fuzzer (go test -fuzz) shrinks every failing input it finds before saving it, and property-based testing libraries such as fast-check for JavaScript shrink failing cases automatically.
Shrinking code: comment out half
The same move works on code. If a 200-line function produces a wrong result, check the value halfway through (a print or a breakpoint). Right at line 100? The bug is in the second half. Wrong already? The first half. When a page renders wrongly, delete half the components (in a scratch copy) and see if the problem stays.
- tweak the CSS → still broken
- update a library → still broken
- restart the server → still broken
- …2 hours later, no idea which part matters- remove the right half of the page → still broken
- remove half of the left half → fixed!
→ the bug is in those 3 components
- put them back one at a time → it's <PriceTag>Shrinking config and environment
"It works on staging but not production" means some difference between the two matters. List the differences (environment variables, config keys, feature flags, versions) and copy half of production's values into staging. Still fine? The culprit is in the other half. Twenty settings take about five tries.
Shrinking dependencies
"It broke when we upgraded from library v2.1 to v2.9." Try v2.5 first. If it's broken there, try v2.3, and so on. Once you know the exact version, read its changelog, or git bisect the library's own repository between those two tags. The same works for language upgrades (Go 1.25 vs 1.26, Node 22 vs 24), compiler flags and base Docker images.
The usual suspects
1. Off-by-one
A loop or range that's one step too long or too short: starting at 1 instead of 0, <= instead of <, > instead of >=. We've met two already: for i := 1 in §07 and total > minTotal in §08. Suspect it whenever the result is wrong by exactly one item or fails only at a boundary (exactly 5,000 cents, the last page, the 31st of the month).
First experiment: test the edges on purpose: 0, 1, the exact threshold, the last index. Prefer for _, x := range xs (Go) and for (const x of xs) (JS), which can't get the start or the end wrong.
2. Nil, undefined and "nothing" values
A value you expected to be there isn't: a nil pointer, an undefined property, an empty list, a missing map key. It's the most common crash in both languages (§04). The two languages fail differently, and JavaScript's way is sneakier:
func average(xs []int) int {
sum := 0
for _, x := range xs {
sum += x
}
return sum / len(xs)
}
func main() {
fmt.Println(average([]int{90, 70, 80}))
fmt.Println(average(nil)) // no scores yet
}
// 80
// panic: runtime error: integer divide by zeroGo crashes at the division, with a stack trace pointing right at it. JavaScript returns NaN ("not a number"), which flows on through the program: into a total, into a database row, onto a web page as "NaN". By the time someone notices, the trace is long gone. Likewise, parts[1] on a too-short array gives undefined silently (§05's greet example), and a Go map returns the Zero value for a missing key without complaint.
First experiment: print the value and where it came from one step earlier. Then find out which inputs produce "nothing": guest users, new accounts, empty carts, deleted records. In JavaScript, ?. (optional chaining) and ?? stop crashes, but only use them when "missing" is really allowed; otherwise they just move the problem somewhere else.
3. A wrong assumption about the data
The code is correct for the data the author imagined. Real data includes names without a space ("Cher"), names with three parts, emoji, prices with a comma as the decimal point, an order with 0 items, a list with 10 million entries, two rows with the same ID. Our lastName function (§04's web server and §05's greet.mjs) assumed every name had exactly two words.
First experiment: get the actual failing input (from logs, the database, the user) and look at it with types visible: %q and %#v in Go, JSON.stringify in JS (§06). Hidden spaces, "42" instead of 42, and null instead of [] jump out.
4. Time zones and clocks
The discount bug from §02, in code. The server's clock is set to UTC, and asking a date for its weekday uses the machine's Time zone unless you say otherwise:
placed, _ := time.Parse(time.RFC3339, "2026-10-31T01:30:00+07:00")
fmt.Println(placed.UTC().Weekday()) // what a UTC server sees: Friday
hanoi, _ := time.LoadLocation("Asia/Ho_Chi_Minh")
fmt.Println(placed.In(hanoi).Weekday()) // the customer's day: SaturdayReal output: Friday, Saturday from Go; 5, Saturday from Node (run with TZ=UTC). The same instant is Friday evening in London and Saturday morning in Hanoi. Suspect time whenever a bug depends on when it runs: only at night, only at month end, only on one laptop, only after the clocks change.
First experiment: run it with another zone: TZ=Asia/Ho_Chi_Minh go test ./.... Flaky Tests: clocks and time zones has more.
5. Environment differences
"Works on my machine" means something differs between machines: an Environment variable that's set locally but not on the server, a different Node or Go version, a different library version, a file path whose case matters on Linux but not on macOS (Config.json vs config.json), the locale, the amount of memory.
First experiment: print the environment where it fails (versions, the relevant variables, the config actually loaded), and compare it with yours, then halve the differences (§09).
6. You're not running the code you think
The fix "didn't work" because the old code is still running: the server wasn't restarted, the browser cached the old JavaScript, the Docker image wasn't rebuilt, you edited a copy of the file, or the request went to a different server. It's embarrassing, and it happens to everyone.
First experiment: add one obviously new print (log.Println("v2 is running")) and check it appears. If it doesn't, stop debugging the code and debug the deploy.
7. Concurrency bugs that hide when you look
A heisenbug changes or disappears when you observe it. Most are races: two goroutines (or two async tasks) touch the same data, and the result depends on which one gets there first. Adding a print or pausing in a debugger changes the timing, so the bad ordering stops happening, and the bug "goes away" while you watch.
First experiment: don't rely on luck. In Go, run the tests with the Race detector (go test -race -count=100 ./...), which reports unsynchronized access to the same memory even on runs that happen to get the right answer. Race conditions: the race detector explains how to read its report. If a program hangs instead, take a Goroutine dump before you restart it (Debugging a hang).
8. The test is flaky, not the code (or both)
A test that passes and fails with no code change might be a broken test (it sleeps, depends on test order or the clock) or a real intermittent bug. Don't assume which. Flaky Tests is the whole playbook.
| Symptom | Suspect first |
|---|---|
| Wrong by exactly one item, or only at a boundary | off-by-one (1) |
nil pointer dereference, undefined, NaN | a missing value, and where it came from (2) |
| Only some users, rows or files | the actual input data (3) |
| Only at night, at month end, or in one region | time zones and clocks (4) |
| Works locally, fails on CI or in production | environment differences (5) |
| Your fix changes nothing at all | old code still running (6) |
| Fails 1 in 50; disappears when you add a print | a race (7) |
Debugging in production
Stop the bleeding first
If users are hurting right now, mitigating comes before understanding: roll back the deploy, turn off the feature flag, add capacity. Investigate once users are safe, and keep the evidence (logs, a dump, the bad build) so you still can. Reliability: mitigate first covers this. The first question of any investigation is the same either way: what changed? A deploy, a config change, a flag flip, a dependency's release, a traffic spike, a new kind of customer. A large share of incidents start with a change, which is why good dashboards draw deploys as lines on every graph.
Narrow it down with the three signals
Treat your telemetry like the hypothesis board in §02. Each question below rules out whole groups of guesses:
- Metrics answer "how much, since when, where?" Is the error rate up for every endpoint or one? Every server or one? Did it start at 14:02, two minutes after a deploy?
- Logs answer "what exactly happened to this request?" A Request ID on every line lets you pull one failing request's whole story out of millions of lines.
- Traces answer "where did the time go, and which service failed?" Distributed tracing follows one request across every service it touched.
Observability teaches all three, and Observability: debugging in production shows how to jump from a spike on a graph to the exact slow request.
Add observation, then wait
When the telemetry can't tell your guesses apart, add the missing measurement: a log line with the values that would decide it, a counter for the suspicious branch, a span around the slow call. Ship it, and let the next occurrence answer. This is print debugging (§06) on a slower loop, so pick the measurement carefully: each round trip may take hours. Being able to turn on DEBUG logs for one service or one customer, without redeploying, makes these loops much shorter.
Evidence from a crash: core dumps
A core dump is a copy of the process's memory, written when it crashes. Open it in a debugger later and you can inspect the variables of the crashed program, as if you'd had a breakpoint on the crashing line. Go writes one on a crash if you set GOTRACEBACK=crash and the operating system allows core files:
$ go build -gcflags=all="-N -l" -o shop . # optional: keep every variable visible
$ ulimit -c unlimited # allow core files in this shell
$ GOTRACEBACK=crash ./shop
order 1 total 2300
panic: runtime error: invalid memory address or nil pointer dereference
… (the trace, now including runtime frames and every goroutine)
Aborted (core dumped)
$ dlv core ./shop core
(dlv) bt
…
10 0x00000000004bc95b in main.shippingFor
at ./main.go:17
11 0x00000000004bca77 in main.total
at ./main.go:28
12 0x00000000004bcdea in main.main
at ./main.go:37
…
(dlv) frame 10 print *o
main.Order {
ID: 2,
Customer: *main.Customer nil,
Items: []int len: 1, cap: 1, [500],}Frames 0 to 9 are the runtime dying; frame 10 is our shippingFor, and print *o shows the exact order that crashed it: ID 2, Customer nil. A stack trace says where it broke; a core dump also says with what data. Where the file lands depends on the machine: on many Linux servers, systemd collects it and coredumpctl lists it; in containers, you usually have to configure a path on purpose.
Evidence from a live Node process: heap snapshots and reports
Node can write a Heap snapshot (every object in memory, and who references whom) without stopping for long, when you send it a Signal:
$ node --heapsnapshot-signal=SIGUSR2 server.mjs &
$ kill -USR2 <pid>
$ ls
Heap.20261031.080432.21355.0.001.heapsnapshotLoad the file in Chrome DevTools' Memory tab. Take two snapshots a few minutes apart and compare them to see what's piling up (GC: memory leaks walks through it). --heapsnapshot-near-heap-limit=1 writes one automatically just before Node runs out of memory. --report-on-signal and --report-on-fatalerror write a JSON diagnostic report: the JavaScript stack, the native stack, resource usage and the versions of everything, often enough to explain a crash.
Evidence from a hung Go process
If a Go service is stuck rather than crashed, take a Goroutine dump before you restart it: every goroutine's stack and what it's waiting on. Race conditions: debugging a hang shows how, with kill -QUIT and pprof's /debug/pprof/goroutine?debug=2.
Attaching a debugger to a live process: rarely, and carefully
dlv attach <pid> and Node's inspector (kill -USR1 <pid> turns it on in a running Node process) work in production, but think first:
- A breakpoint pauses the whole process. Every request it's serving stalls, health checks fail, and the orchestrator may kill it. Take the instance out of the load balancer first, or use logpoints (§07), which don't pause.
- The inspector is remote code execution. Anyone who can reach its port can run any code in your process. Keep it bound to
127.0.0.1(the default) and reach it through an SSH tunnel, never by opening the port. - Dumps contain everything. Core dumps and heap snapshots hold passwords, tokens and customers' personal data. Store them like secrets, and delete them when the investigation is done.
After the fix: tests, causes and notes
1. Red, then green
Turn your repro from §03 into a Regression test, and run it before the fix. It must fail, and fail for the right reason. A test that passes before the fix proves nothing: it might not even reach the broken code.
$ go test .
--- FAIL: TestTotalGuestCheckout (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x52d98b]
goroutine 35 [running]:
…
shop.shippingFor(...)
/tmp/debugging/shop/main.go:17
shop.total(...)
/tmp/debugging/shop/main.go:28
shop.TestTotalGuestCheckout(0x28b73a0e2248?)
/tmp/debugging/shop/main_test.go:8 +0x6b
…
FAIL shop 0.017s
FAILThe same crash, at the same line, reached from the test. (Go names the package shop here, after the module, instead of main. [recovered, repanicked], printed since Go 1.25, means the test framework caught the panic to mark the test as failed, then let it continue.) Now the fix:
func shippingFor(o *Order) int {
if o.Customer != nil && o.Customer.Country == "VN" { // ← guests have no customer
return 300
}
return 1500
}$ go test -v .
=== RUN TestTotalGuestCheckout
--- PASS: TestTotalGuestCheckout (0.00s)
PASS
ok shop 0.004sRed, then green: now you know the fix works, and the test stays in the suite so the bug can't quietly come back. Name it after the behavior (TestTotalGuestCheckout), not the ticket number; Testing fundamentals explains why.
2. Fix the cause, not just the symptom
The nil check stops the crash. But keep asking "why?" until you reach something you can change. Some teams call this the five whys (five is a rule of thumb; here four were enough):
- Why did it crash?
o.Customerwas nil. - Why was it nil? Guest checkouts have no customer.
- Why did shipping read the customer at all? Shipping depends on where the parcel goes, which is the shipping address, not the account. Guests have a shipping address too.
- Why didn't tests catch it? Every test order had a customer.
Answer 3 suggests a better fix: base shipping on the order's shipping address, which every order has. Answer 4 says to add guest orders to the test data. Both prevent bugs you haven't found yet.
3. Look for siblings
A bug is rarely alone. If o.Customer.Country crashed here, search for every other place that does the same thing:
grep -rn "\.Customer\." --include='*.go' .Fix them in the same pull request, or make the mistake impossible: for example, a method o.Country() that handles guests in one place, used everywhere.
Don't swallow the crash
It's tempting to wrap everything in a recover() (Go) or a try { … } catch {} (JS) "so it never crashes again". That doesn't fix the bug: it hides it. The program carries on with wrong data, and the next bug shows up far away, with no stack trace. Catch errors only where you can actually handle them (return a 400, retry, use a default you've chosen on purpose), and log them with full detail when you do. See Go idioms: panic and recover.
4. Write short, blameless notes
For anything that reached users or cost more than an hour, write a few lines in the ticket or pull request. Keep it blameless: describe what the system allowed, not who slipped. "The shipping code assumed every order had a customer, and no test had a guest order" is useful. "Dana forgot guests" teaches nothing and makes people hide mistakes.
Symptom Guest checkouts failed with a 500 since the 14 Oct release.
Cause shippingFor read o.Customer.Country; guests have no Customer.
Found by Stack trace → main.go:17; repro test with a guest order.
Fix Shipping now uses the shipping address (every order has one).
Prevent Guest orders added to test fixtures; grep found 2 more sites.
Time lost 40 min. A request ID in the error log would have made it 5.For a real incident, this grows into a Postmortem: Reliability: blameless postmortems has the full format. Notes like these are also how a team learns its own usual suspects (§10): after a year of them, you'll know that your system's bugs are mostly time zones and missing customers.
Check yourself
Answer before you click. Q1 is covered in §04, Q2 in §02, Q3 in §08, Q4 in §10 and Q5 in §12.
A Go service logs this. Which line do you open in your editor first?
goroutine 12 [running]:
net/http.(*conn).serve.func1()
/usr/local/go/src/net/http/server.go:1907 +0xbd
panic({0x6eccc0?, 0xc000012345?})
/usr/local/go/src/runtime/panic.go:860 +0x13a
main.parsePrice(...)
/app/price.go:41
main.checkout(...)
/app/handlers.go:88 +0x96Exports fail for “some” customers. Your guesses: (A) customers with non-ASCII names, (B) customers with more than 1,000 orders, (C) a slow database. Which experiment should you run first?
A bug appeared somewhere in the last 500 commits. You have a script that tells good from bad. About how many times will git bisect run run it?
A Go test fails about once in 50 runs with a wrong total. You add a fmt.Println in the loop and it stops failing. In the debugger it never fails. What's the most likely cause, and the next step?
You found that guest checkouts crash because o.Customer is nil, and you added a nil check. What should go into the same pull request?
Cheat sheet
Commands
| I want to… | Go | JavaScript / Node |
|---|---|---|
| Print a value with its type | log.Printf("%#v", x) | console.dir(x, { depth: null }) |
| See who called this | debug.PrintStack() | console.trace("why?") |
| Start a debugger | dlv debug · dlv test | node --inspect-brk app.mjs + chrome://inspect or VS Code |
| Breakpoint / conditional | break main.go:8 · cond 1 i == 2 | click the line number · right-click → Edit condition · debugger; |
| Step over / into / out | next · step · stepout | F10 · F11 · Shift+F11 |
| Show variables / stack | locals · print x · stack | Scope and Call Stack panes |
| Every goroutine's stack on crash | GOTRACEBACK=all ./app | — |
| Crash dump | GOTRACEBACK=crash + dlv core ./app core | --report-on-fatalerror |
| Memory snapshot | heap profile via pprof | --heapsnapshot-signal=SIGUSR2, then kill -USR2 <pid> |
| Original file/line in minified code | — | node --enable-source-maps |
| Find a race | go test -race -count=100 | look for un-awaited work |
git bisect
git bisect start <bad> <good> # e.g. git bisect start main v1.3
git bisect good | bad | skip # after testing each commit git checks out
git bisect run ./check.sh # exit 0 = good, 1–127 = bad, 125 = skip
git bisect log # your answers so far (replay with git bisect replay)
git bisect reset # back to where you startedSymptom → first suspect
| You see | Look at |
|---|---|
| Wrong only at a boundary or by one item | off-by-one (§10) |
nil dereference, undefined, NaN | the missing value and where it came from (§04, §10) |
| Only some users or rows | the real input data, with types visible (§06) |
| Only at night, month end, one region | time zones (§10) |
| Works locally, fails elsewhere | environment differences; halve them (§09) |
| "It worked last month" | git bisect (§08) |
| Disappears when you print or debug | a race: -race (§10) |
| Only in production | what changed; logs, metrics, traces; dumps (§11) |