Virtual Memory

The 30-second version

In computer terms: the key card numbers are virtual addresses, the real rooms are slots in RAM called page framesPage frameA page-sized slot (usually 4 KiB) of real, physical RAM. Virtual pages are what your program sees; frames are where the bytes actually live. The page table maps one to the other., and the front desk's map is the page table. The checking happens in hardware, inside the CPU, on every single memory read and write (a load or storeLoad / storeThe CPU's names for its two basic memory operations: a load reads a value from memory into the CPU, a store writes one back. Every variable read or assignment in your code becomes loads and stores.).

The four things it buys you

The fourth one surprises people: because RAM is handed out lazily, programs can reserve far more memory than they use, and in total even more than the machine has (§09 reserves 8 GiB and uses 11 MB). Even a tiny Go "hello world" on the machine used for this article reported this in /proc/self/status, a file where Linux reports a process's memory stats:

VmSize:	 1225224 kB    ← address space reserved: about 1.2 GB
VmRSS:	    1920 kB    ← actually in RAM: about 2 MB

Both numbers are true. One counts rooms booked, the other counts rooms with someone in them. By the end of §07 you'll know which one to put on a dashboard.

What you'll learn

• §02–§04: how a virtual address becomes a physical one (pages, page tables, the TLBTLBTranslation lookaside buffer: a small cache inside the CPU that remembers recent page-table lookups, so translating an address is usually instant.), with a translator you can drive yourself.
• §05–§06: page faults (how memory appears on first touch) and mmap (files as memory), measured from Go and Node.
• §07–§09: the production side: VSZ vs RSS, why memory doesn't drop after garbage collection, overcommit, swap and the OOM killer.
• §10–§11: staff-level trade-offs: mmap in databases, huge pages, NUMA, ASLR and Meltdown.

Pages and frames: memory in 4 KiB chunks

Why chunks?

Suppose the OS kept a map entry for every single byte: "virtual byte 1000 lives at physical byte 73,512". Each entry needs about 8 bytes to store a physical address. So describing 1 GB of memory would take 8 GB of map. The map would be bigger than the memory it describes.

So memory is handled in pages instead: blocks of 4096 bytes (4 KiB; 1 KiB = 1024 bytes). One map entry covers a whole page. Now 1 GB of memory needs about 262,000 entries, around 2 MB of map. That's affordable.

• A page is a 4 KiB chunk of virtual memory: what your program sees.
• A Page framePage frameA page-sized slot (usually 4 KiB) of real, physical RAM. Virtual pages are what your program sees; frames are where the bytes actually live. The page table maps one to the other. (or just "frame") is a 4 KiB slot of physical RAM: where the bytes really are.
• The Page tablePage table (and pages)Memory is handled in chunks called pages (usually 4 KB). A page table is the per-process lookup table that says “virtual page X lives at physical RAM spot Y”. The CPU consults it on every memory access. says which frame each page lives in, plus what you're allowed to do with it (read, write, run as code).

Splitting an address

4096 is exactly 2¹², so the position inside a page always fits in the lowest 12 bits of the address. That gives a beautifully simple rule:

• the low 12 bits are the offsetPage offsetThe position of a byte inside its page: the low 12 bits of the address for a 4 KiB page (0 to 4095). Translation keeps the offset unchanged and only swaps the page number for a frame number. (0 to 4095): where in the page;
• everything above is the page number: which page.

In HexadecimalHexadecimal (hex)Base-16 numbers using the digits 0–9 and a–f. One hex digit is exactly 4 bits, so a byte is two hex digits: ff = 255. MAC and IPv6 addresses are written this way., each digit is exactly 4 bits, so 12 bits = the last three hex digits. You can split an address by eye.

Worked example. Take the 16-bit address 0x5A3C:

Now look page 5 up in the page table. Say it lives in frame 0x11. Translation swaps the page number for the frame number and keeps the offset: physical address = 0x11 × 0x1000 + 0xA3C = 0x11A3C. Multiplying by 0x1000 just appends three zeros in hex (0x11000), so the result is simply the frame digits followed by the offset digits.

Drive it yourself. Try the presets too: reading code is fine but writing it isn't, a "first touch" of reserved memory triggers a fault that the kernelUser space vs kernel spaceYour programs run in user space with limited rights; the OS kernel runs in kernel space with full access to hardware. Crossing from one to the other (a system call) costs time. (the core of the OS, which runs with full rights) quietly fixes, and a null pointer lands on page 0, which is never mapped.

Who does the translating?

Not your program, and not the OS on each access. It's a piece of the CPU called the MMUMMU (memory management unit)The part of the CPU that translates every virtual address into a physical one, using the page table, in hardware. It also checks permissions and raises a page fault when a page isn't mapped or isn't allowed.. It translates every load and store in hardware, and checks the permission bits at the same time. The OS's job is to fill in the page tables, and to step in when the MMU raises a Page faultPage faultThe CPU stops a memory access because the page table doesn't allow it (yet) and hands control to the kernel. Often harmless: the kernel fixes things up — e.g. makes a private copy of a copy-on-write page — and the program carries on without noticing. because an entry is missing or forbids the access.

That's also where the Segmentation faultSegmentation fault (segfault)What happens when a program touches a memory address it isn't allowed to use. The CPU stops it and the OS kills the whole process (“Segmentation fault (core dumped)”, exit code 139). you know comes from. A null pointer is address 0, which is on page 0. The OS deliberately never maps page 0. So the MMU faults, the kernel sees there's nothing to fix, and it kills the process.

Seeing it from Go

Go lets you print a variable's virtual address. Here we split one by hand, exactly like the table above. (The package-level keep variable forces the int onto the heap; go build -gcflags=-m confirms new(int) escapes to heap. Stack & Heap explains why.)

package main

import (
	"fmt"
	"os"
	"unsafe"
)

var keep *int // package-level, so the int below must live on the heap

func main() {
	keep = new(int)                       // an int on the heap
	addr := uintptr(unsafe.Pointer(keep)) // its virtual address, as a number

	fmt.Println("page size:  ", os.Getpagesize())
	fmt.Printf("address:     %#x\n", addr)
	fmt.Printf("page number: %#x\n", addr>>12)   // ← drop the low 12 bits
	fmt.Printf("offset:      %#x\n", addr&0xFFF) // ← keep only the low 12 bits
}

page size:   4096
address:     0xc0000100f0
page number: 0xc000010
offset:      0xf0

JavaScript deliberately has no way to see memory addresses: the engine moves objects around, and hiding addresses is also a security feature. Node doesn't expose the page size either; from a shell, getconf PAGESIZE prints it.

Page tables: a four-level lookup

The flat table doesn't fit

On x86-64, programs use 48-bit virtual addresses (newer CPUs can use 57; more on that below). 2⁴⁸ bytes is 256 TiB of address space per process. With 4 KiB pages, that's 2³⁶, about 69 billion pages.

A flat table with one 8-byte entry per page would be 2³⁶ × 8 bytes = 512 GiB, for every process. Impossible. But almost all of that space is empty: a typical program uses a few small islands (code, heap, stacks, libraries) scattered across a gigantic empty ocean.

A tree that skips the empty parts

The fix is the same as a printed street index: you don't list every house in the country on one page. You look up the region, then the city, then the street, then the house. And if a region has no houses at all, it gets no pages in the index.

The 48-bit address is cut into five pieces:

Nine bits can count from 0 to 511, so each table has 512 entries of 8 bytes = exactly 4 KiB, one page. Each entry in a level-4, -3 or -2 table points to a table one level down; a level-1 entry points to the frame holding the data. An empty region just has "not present" at the top, and none of the tables below it exist.

Here's what that looks like for a small Go program. Press the button to touch more heap and watch which tables appear:

Now step through a real translation, one path down that tree (the address is the Go heap address printed in §02):

1// What the MMU does in hardware (pseudo-code)
2func translate(va uint64) uint64 {
3    table := cr3 // this process's top-level table
4    for _, shift := range []uint{39, 30, 21} {
5        e := table[(va>>shift)&0x1FF] // 9-bit index
6        if !e.present { pageFault(va) }
7        table = e.next // go one level down
8    }
9    e := table[(va>>12)&0x1FF] // level 1
10    if !e.present { pageFault(va) }
11    return e.frame<<12 | va&0xFFF // frame + offset
12}

How much memory do the tables take?

Only what's used. Linux reports it as VmPTE in /proc/<pid>/status. From the real runs in §01 and §07:

Compare that with the 512 GiB a flat table would need. The tree is why 64-bit address spaces are practical at all.

The cost: four reads before the real one

The tree has a price. To translate one address, the CPU reads four table entries, one per level, and each read depends on the one before (you can't know where level 3 is until you've read level 4). This is called a Page walkPage walkWhat the CPU does when it has no cached translation: it reads the page table level by level (4 reads on x86-64) to find the physical frame. Each read is a memory access, so a walk can cost tens to hundreds of CPU cycles.. Only after it can the CPU fetch the byte you actually asked for.

Those table entries live in ordinary memory, so they can sit in the CPU cacheCPU cacheSmall, very fast memory on the CPU chip (L1, L2, L3) that keeps copies of recently used data. Reading from cache takes ~1–10 ns; reading from RAM takes ~100 ns. (a few nanoseconds) or, in the worst case, all the way out in RAM (~100 ns each). A walk costs anywhere from a few dozen to several hundred CPU cyclesCPU cycleOne tick of the processor's clock. At 3 GHz a cycle is a third of a nanosecond; a simple instruction takes about one cycle, a read from RAM a few hundred.. If every instruction paid that, programs would crawl. The next section is about the small cache that makes almost all of them free.

The TLB: remembering recent translations

How small?

Exact sizes vary by CPU model, so treat these as rough, typical numbers for a recent x86-64 server core:

That last column is the TLB reach: how much memory you can touch without ever missing. A few megabytes. A service that hops randomly around a 2 GB cache is far outside it.

Hits depend on how you walk memory

Every value on the same page shares one TLB entry. So a loop over an array in order gets 511 hits for every miss (4096 bytes ÷ 8-byte values = 512 values per page). Random access over a big range is the opposite: nearly every access lands on a different page, and nearly every one misses.

Try it. Pick "random" over 1 GiB: with 64 entries and 4 KiB pages, almost every access misses. Switch to 2 MiB pages and it only climbs to about 12%, because 64 × 2 MiB = 128 MiB still doesn't cover 1 GiB. Now pick the realistic 1,536-entry size: 1,536 × 2 MiB = 3 GiB covers everything, and the hit rate goes to nearly 100%. Finally try "one per page" over a range just bigger than the TLB, and watch a loop evict exactly the entry it will need next.

Huge pages: more reach per entry

A Huge pageHuge pageA bigger page: 2 MiB (or 1 GiB) instead of 4 KiB on x86-64. One TLB entry then covers 512× more memory, so fewer translation misses, at the price of coarser, sometimes wasted, memory. is 2 MiB instead of 4 KiB, so one TLB entry covers 512× more memory (and the page walk is one level shorter). 1,500 entries × 2 MiB = 3 GiB of reach.

Here's the same random-read loop over 1 GiB, once with normal pages and once asking Linux for huge pages. (The core of the real program; the full version also builds idx and checks AnonHugePages in /proc/self/smaps_rollup to confirm the huge pages really arrived.)

// Get 1 GiB of plain memory straight from the kernel (no file behind it).
mem, err := syscall.Mmap(-1, 0, 1<<30,
	syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_PRIVATE|syscall.MAP_ANON)
if err != nil {
	panic(err)
}
if huge {
	// ← "please use 2 MiB pages here". Fails (EINVAL) if the kernel has no THP support.
	if err := syscall.Madvise(mem, syscall.MADV_HUGEPAGE); err != nil {
		log.Println("no huge pages:", err)
	}
}
for i := range mem { // touch everything first, so we time reads, not page faults
	mem[i] = byte(i)
}

start := time.Now()
var sum byte
for _, i := range idx { // idx: 20 million random positions in mem
	sum += mem[i]
}
fmt.Println(time.Since(start), sum) // ← print sum, or the compiler may skip the reads

4 KiB pages:  ~41–44 ns per read
2 MiB pages:  ~21–30 ns per read

Up to twice as fast, for one hint. Both versions miss the data caches equally (same random positions); the difference is the page walks. The numbers are approximate and vary between runs and machines.

Page faults: memory on demand

This is Demand pagingDemand pagingThe OS gives a process real RAM only when it first touches a page, not when it asks for memory. Asking just reserves addresses; the first access causes a page fault that brings the page in.. It's the hotel from §01 preparing a room only when you first open its door.

What happens on first touch

1. Your program calls make([]byte, 100<<20) (100 MiB). The Go runtime asks the kernel for address space. The kernel notes "this range belongs to the process" but puts nothing in the page table.
2. Your code writes buf[0] = 1. The MMUMMU (memory management unit)The part of the CPU that translates every virtual address into a physical one, using the page table, in hardware. It also checks permissions and raises a page fault when a page isn't mapped or isn't allowed. finds no entry for that page and raises a Page faultPage faultThe CPU stops a memory access because the page table doesn't allow it (yet) and hands control to the kernel. Often harmless: the kernel fixes things up — e.g. makes a private copy of a copy-on-write page — and the program carries on without noticing.: the CPU stops the instruction and jumps into the kernel.
3. The kernel checks its notes: "that range is reserved, fine". It takes a free framePage frameA page-sized slot (usually 4 KiB) of real, physical RAM. Virtual pages are what your program sees; frames are where the bytes actually live. The page table maps one to the other., fills it with zeros (so you can't read the previous owner's data), and writes the mapping into the page table.
4. The kernel returns, and the CPU retries the same instruction. This time it works. Your code never knew.

That's a Minor page faultMinor page faultA page fault the kernel can fix without touching the disk: give a fresh zeroed frame, make a copy-on-write copy, or map a file page that is already cached in RAM. Costs roughly a microsecond.: fixed without touching the disk. Reading an untouched page faults too (Linux then maps a shared, read-only page of zeros, and the first write later faults again to get a real frame), so the rule is "first touch", read or write. Let's count them for real.

package main

import (
	"fmt"
	"syscall"
)

// minorFaults asks the kernel how many minor page faults this process has had so far.
func minorFaults() int64 {
	var ru syscall.Rusage
	syscall.Getrusage(syscall.RUSAGE_SELF, &ru)
	return ru.Minflt
}

func main() {
	const size = 100 << 20 // 100 MiB
	before := minorFaults()
	buf := make([]byte, size) // ← reserve: no real RAM yet
	afterAlloc := minorFaults()

	for i := 0; i < len(buf); i += 4096 { // ← touch one byte per 4 KiB page
		buf[i] = 1
	}
	afterTouch := minorFaults()

	fmt.Println("faults from make():   ", afterAlloc-before)
	fmt.Println("faults from touching: ", afterTouch-afterAlloc)
	fmt.Println("pages in 100 MiB:     ", size/4096)
}

# Go                              # Node
faults from make():    82         faults from allocation: 15
faults from touching:  25668      faults from touching:   25762
pages in 100 MiB:      25600      pages in 100 MiB:       25600

Allocating 100 MiB caused almost no faults. Touching it caused one fault per page, give or take some bookkeeping by the runtime. The memory was "free" until the loop wrote to it.

What a fault costs

On this machine, touching 1 GiB for the first time (262,144 faults) took about 1 second; touching it a second time took about 10–25 ms. That's roughly 4 µs per fault, including zeroing the page. (This is a virtual machine; on bare metal a minor fault is often closer to 1 µs. Either way it's thousands of times more than a normal memory write.)

So first-touch cost is real. It shows up as a slow first request after startup, or as a benchmark that's mysteriously slower on its first iteration. Benchmark warm, or pre-touch memory you know you'll need.

The three kinds of fault

Major faults are the dangerous ones: one is like a tiny, invisible disk read in the middle of an ordinary line of code. §06 produces 51,000 of them on purpose.

Copy-on-write: the fault that makes fork() cheap

Threads vs Processes §06 showed that forkfork()The Unix system call that clones the calling process. The copy (the child) gets a new PID and a copy of the parent's memory and open files. It returns twice: 0 in the child, the child's PID in the parent. doesn't copy the parent's memory. Here's how that works at the page level: the child gets a copy of the page table, both tables point at the same frames, and every shared page is marked read-only in both.

Reading is fine. The first write to a shared page faults; the kernel copies just that one 4 KiB page, gives the writer its private copy, and retries the write. That's Copy-on-writeCopy-on-write (COW)A lazy copy: after fork(), parent and child share the same memory pages, marked read-only. A page is only really copied when one of them writes to it. This makes fork() fast., and it's a minor fault too.

mmap: files as memory

First, where file data lives: the page cache

When any program reads a file, Linux keeps the file's pages in RAM afterwards, in the Page cachePage cacheThe kernel's cache of file contents in RAM. Every normal file read and write goes through it, and mmap maps its pages straight into your process. Free RAM fills up with it, which is fine: it's dropped when memory is needed.. The next read of the same data comes from RAM instead of the disk. (That's why free on a long-running server shows most RAM as buff/cache: it's file data, and the kernel drops it the moment someone needs the memory.)

There are two ways to get those bytes into your program:

With read(), every call is a System callSystem call (syscall)A request from your program to the operating system — “read this file”, “open a connection”. While a blocking syscall runs, the calling thread waits inside the OS., and the kernel copies the bytes from the page cache into your buffer. With mmap, there's one syscall up front; after that, the page table points straight at the page-cache frames. No copy (zero-copyZero-copyMoving data (e.g. from a file to a socket) without copying it through your program's memory. The kernel moves it directly, saving copies and syscalls. sendfile and splice are the Linux calls for this.), and no syscall per read.

mmap in Go

package main

import (
	"bytes"
	"fmt"
	"os"
	"syscall"
)

func main() {
	f, err := os.Open("access.log")
	if err != nil {
		panic(err)
	}
	defer f.Close()
	fi, err := f.Stat()
	if err != nil {
		panic(err)
	}

	// Map the whole file into our address space, read-only.
	data, err := syscall.Mmap(int(f.Fd()), 0, int(fi.Size()),
		syscall.PROT_READ, syscall.MAP_SHARED)
	if err != nil {
		panic(err)
	}
	defer syscall.Munmap(data) // ← unmap when done; never use data afterwards

	// data is a plain []byte: no Read() calls, no copying into a buffer.
	fmt.Println("lines:", bytes.Count(data, []byte{'\n'}))
}

lines: 200000

Shared or private?

The flags argument decides what happens when you write to a mapping (opened with PROT_READ|PROT_WRITE):

• MAP_SHARED: your writes go into the page cache, so they reach the file and are visible to every other process mapping it.
• MAP_PRIVATE: the first write to a page triggers Copy-on-writeCopy-on-write (COW)A lazy copy: after fork(), parent and child share the same memory pages, marked read-only. A page is only really copied when one of them writes to it. This makes fork() fast. (§05); you get a private copy and the file never changes.

Here's both at once, on the same 12-byte file:

// (errors ignored for brevity; f.Close() omitted)
os.WriteFile("greeting.txt", []byte("hello world\n"), 0o644)
f, _ := os.OpenFile("greeting.txt", os.O_RDWR, 0)
rw := syscall.PROT_READ | syscall.PROT_WRITE
shared, _ := syscall.Mmap(int(f.Fd()), 0, 12, rw, syscall.MAP_SHARED)
private, _ := syscall.Mmap(int(f.Fd()), 0, 12, rw, syscall.MAP_PRIVATE)

copy(private[0:5], "HOWDY") // copy-on-write: only this mapping sees it
copy(shared[6:11], "WORLD") // goes into the page cache → into the file

onDisk, _ := os.ReadFile("greeting.txt")
fmt.Printf("%q\n%q\n%q\n", private, shared, onDisk)
// "HOWDY world\n"   ← private copy was taken before WORLD was written
// "hello WORLD\n"
// "hello WORLD\n"   ← the file changed

Mappings come in four flavours, and you use all four every day without knowing:

Major faults, measured

With mmap, a page that isn't in the page cache yet is loaded by a Major page faultMajor page faultA page fault that has to wait for the disk: the data is in a file that isn't cached, or the page was swapped out. Costs from tens of microseconds (SSD) to milliseconds (spinning disk).. To see them, this test maps a 200 MiB file with syscall.Mmap and runs the same loop as faults.go in §05 (one byte per page, counting faults with Getrusage). Rows 3 and 4 add one line after mapping: syscall.Madvise(data, syscall.MADV_RANDOM). Before each "cold" run it empties the page cache (echo 3 > /proc/sys/vm/drop_caches, needs root).

                                  time     major faults   minor faults
cold cache                        99 ms         2            1,607
warm cache (already in RAM)        4 ms         0            1,605
cold, MADV_RANDOM (no readahead)  2.6 s    51,078              128
warm, MADV_RANDOM                 13 ms         0            3,200

Three lessons hide in that table:

1. Major faults are expensive. Without help, each one waited about 50 µs for the disk. 51,000 of them took 2.6 seconds.
2. ReadaheadReadaheadWhen the kernel sees you reading a file in order, it loads the next chunks from disk before you ask, so most of your reads find the data already in RAM. hides most of them. In the first row, the kernel saw sequential access and loaded upcoming pages before they were touched. Nearly every fault found its page already in RAM, so only 2 were major. madvise(MADV_RANDOM) switches readahead off, which is right for truly random access and terrible here.
3. There are fewer faults than pages (51,200 pages, ~1,600–3,200 faults). For file mappings, Linux maps several neighbouring cached pages on each fault (fault-aroundFault-aroundOn a page fault in a file mapping, Linux also maps the neighbouring pages that are already cached (64 KiB worth by default), so one fault can serve many pages.), so one fault can serve 16 or more pages.

Here's readahead in slow motion. Run it once with readahead on, then switch it off:

Node: no built-in mmap

Node has no mmap in its standard library. (Community native add-ons exist, but they need a C++ build step and can crash the process on the errors described below.) The usual answer is to read in chunks into one reused buffer: a syscall per chunk, and one copy, but steady memory use for files of any size:

const fs = require("node:fs");

const fd = fs.openSync("access.log", "r");
const buf = Buffer.alloc(1 << 20); // one reusable 1 MiB buffer
let lines = 0, n;
while ((n = fs.readSync(fd, buf, 0, buf.length, null)) > 0) { // ← null = "continue where the last read stopped"
  for (let i = 0; i < n; i++) if (buf[i] === 10) lines++; // 10 = '\n'
}
fs.closeSync(fd);
console.log("lines:", lines); // lines: 200000

For streaming work, fs.createReadStream does the same chunking for you without blocking the event loop.

VSZ, RSS, PSS: which memory number is real?

ps shows VSZ and RSS; top calls them VIRT and RES. The kernel exposes the details in /proc/<pid>/status, where /proc/self/... always means "the process reading it".

Two hello-worlds, read from the inside

# Go hello-world               # Node 22, empty script
VmSize:  1225224 kB  ← VSZ      VmSize:   748100 kB
VmRSS:      1920 kB  ← RSS      VmRSS:     42592 kB
RssAnon:     512 kB             RssAnon:    6388 kB
RssFile:    1408 kB             RssFile:   36204 kB
VmPTE:        80 kB             VmPTE:       480 kB

RssAnon is Anonymous memoryAnonymous memoryMemory not backed by any file: heaps, stacks, Go and V8 heap arenas. It starts as zeros. If the system needs the RAM back, it can only go to swap (or be lost when the process exits). in RAM (heap, stacks), and RssFile is file-backed pages in RAM (the program's own code, shared libraries). Node's RSS is mostly pages of the ~120 MB node executable file that it has touched. Here's how to read these numbers from code:

// Print the VmRSS line from /proc/self/status (Linux only).
b, _ := os.ReadFile("/proc/self/status")
for _, line := range strings.Split(string(b), "\n") {
	if strings.HasPrefix(line, "VmRSS:") {
		fmt.Println(line) // VmRSS:	    1920 kB
	}
}

Why is VSZ so big? Read the map

/proc/self/maps lists every reserved range, one per line: address range, permissions, and the file behind it (if any). Here's the Go hello-world, trimmed and annotated:

00400000-0049f000 r-xp ... /tmp/vm/maps   ← program code: read + execute
0049f000-0056a000 r--p ... /tmp/vm/maps   ← constants: read-only
0056a000-00575000 rw-p ... /tmp/vm/maps   ← global variables: read + write
c000000000-c000400000 rw-p ...            ← Go heap: 4 MiB usable so far
c000400000-c004000000 ---p ...            ← 60 MiB more heap, reserved, no access yet
7f0fd7600000-7f0fe7780000 ---p ...        ← 257 MiB reserved for the runtime's bookkeeping
7f0fe7781000-7f1007780000 ---p ...        ← 512 MiB more of the same
7f101c0fc000-7f101c0fe000 r-xp ... [vdso] ← kernel helper code (see the System Calls topic)
7ffdffa2a000-7ffdffa4d000 rw-p ... [stack] ← main thread's stack

The permissions column is read, write, execute, then private or shared. The lines marked ---p have no permissions at all. They're pure reservations: "keep these addresses for me, I'll use them later". They cost no RAM, but they count in full towards VSZ. Go reserves room for its heap and for its bookkeeping about the heap up front; V8 does something similar for its JavaScript heap. That's why a Go or Node process routinely shows a VSZ of gigabytes while using megabytes.

When RSS lies: shared pages

RSS counts every page in RAM that this process can see, including pages it shares with others. Ten Node processes on one machine all map the same node binary; each one's RSS includes the ~36 MB of it that's in RAM, but RAM holds it only once.

PSS fixes this by dividing each shared page by the number of processes sharing it. From the same Node process, via /proc/self/smaps_rollup:

Rss:               42616 kB
Pss:               16282 kB   ← other node processes on this machine share the binary's pages
Shared_Clean:      35908 kB
Private_Dirty:      6452 kB   ← this process's own memory

Adding up RSS across a pool of forked workers overcounts badly; adding up PSS gives the true total. (Some tools also show USSUSS (unique set size)Only the pages of a process that no other process shares. It's roughly what you'd get back if this process exited.: only the private pages, i.e. what you'd get back if this process exited.)

Now drive one process through its life and watch which number moves:

Why RSS doesn't drop after garbage collection

A classic incident: "We fixed the leak, memory usage dropped in our heap metrics, but the container's memory graph stayed flat at 1.8 GB." Nothing is wrong. There are three layers, and each one keeps memory for a while:

1. Your objects. The garbage collector finds the dead ones and marks their space free (Garbage Collection covers how).
2. The runtime's heap. That free space stays in Go's or V8's heap, ready for the next allocation. Still in RAM, still in RSS. Asking the kernel for fresh memory and faulting it in again costs page faults (§05), so keeping it is usually the fast choice.
3. The kernel. Only when the runtime calls madvisemadviseA system call that gives the kernel hints about a memory range: “I'll read this randomly”, “use huge pages here”, or “I don't need these pages any more, take them back” (MADV_DONTNEED / MADV_FREE). on whole, empty pages does the kernel take the frames back. Now RSS drops. The addresses stay reserved, so VSZ doesn't change.

Measured: Go

var sink [][]byte

func main() {
	fmt.Println("start:                ", rss())
	for i := 0; i < 50; i++ {
		b := make([]byte, 10<<20) // 50 × 10 MiB = 500 MiB
		for j := range b {
			b[j] = 1 // touch every byte, so it's really in RAM
		}
		sink = append(sink, b)
	}
	fmt.Println("after 500 MiB:        ", rss())

	sink = nil   // drop every reference…
	runtime.GC() // …and collect the garbage
	fmt.Println("after GC:             ", rss())

	time.Sleep(2 * time.Second)
	fmt.Println("2 s later:            ", rss())

	debug.FreeOSMemory() // ← force: GC + return free pages to the OS now
	fmt.Println("after FreeOSMemory(): ", rss())
}

start:                 2036 kB
after 500 MiB:         515300 kB
after GC:              515316 kB    ← the heap is empty, RSS hasn't moved
2 s later:             512644 kB    ← the background scavenger has returned a tiny bit
after FreeOSMemory():  3204 kB      ← pages handed back to the kernel

Go returns memory on its own through a background scavenger that hands back pages it doesn't expect to need soon, gradually so it doesn't slow the program down. With GOMEMLIMIT set (Go 1.19+), it works harder as memory approaches the limit. Calling debug.FreeOSMemory() yourself is rarely needed. It makes sense after a one-off spike in a long-running process (a nightly import, a huge report), not in a request handler.

Measured: Node

const mb = (n) => `${Math.round(n / 1024 / 1024)} MiB`;
const show = (label) => {
  const m = process.memoryUsage();
  console.log(label.padEnd(16), "rss", mb(m.rss).padStart(8), "  heapUsed", mb(m.heapUsed).padStart(8));
};

show("start:");
let objs = [];
for (let i = 0; i < 3_000_000; i++) objs.push({ id: i, name: "user" + i });
show("after alloc:");
objs = null;     // drop the only reference
globalThis.gc(); // force a full GC (only exists with --expose-gc)
show("after gc():");
setTimeout(() => show("3 s later:"), 3000);

start:           rss   40 MiB   heapUsed    4 MiB
after alloc:     rss  365 MiB   heapUsed  267 MiB
after gc():      rss  309 MiB   heapUsed    4 MiB   ← objects gone, most pages kept
3 s later:       rss  309 MiB   heapUsed    4 MiB

heapUsed fell by 263 MiB; RSS fell by 56 MiB. V8 keeps most of the empty heap space for future allocations and gives it back gradually (for example when the process goes idle or memory gets tight); Node has no public "give it back now" call. Memory outside the JS heap behaves differently again: a big Buffer or ArrayBuffer is allocated by the C library, which often returns large blocks to the OS as soon as they're freed.

DONTNEED vs FREE: two ways to give pages back

MADV_FREE is more efficient, but it makes RSS look high for memory the kernel could take at any moment, and that confuses dashboards and container limits. Go used MADV_FREE in 1.12–1.15 and switched back to MADV_DONTNEED by default on Linux in Go 1.16 for exactly that reason (GODEBUG=madvdontneed=0 opts back in). In /proc/<pid>/smaps_rollup, the LazyFree line shows memory in this "free but still counted" state.

Overcommit, swap and the OOM killer

The promise and the bet

Because RAM is only handed out on first touch (§05), the kernel can promise much more than it has. On the machine used for this article, /proc/meminfo said:

Committed_AS:   70818944 kB   ← memory promised to all processes: ~70 GB
MemTotal:       16091 MB      ← memory that exists: 16 GB (and no swap)

That's OvercommitOvercommitLinux promises processes more memory than really exists, betting that most reserved memory is never touched. Allocations succeed; if everyone does touch their memory, the OOM killer settles the difference.: 70 GB promised, 16 GB real. It works because most of those promises are reservations like the ones in §07 that are never filled. Here's what one Go program can get away with:

big := make([]byte, 8<<30) // ask for 8 GiB…
big[0] = 1                 // …but touch only the first page
// VmSize: 9623204 kB   VmRSS: 11204 kB   (real run)

8 GiB "allocated" on a 16 GB machine, 11 MB used. Ask for 64 GiB and even Linux refuses (real output: fatal error: runtime: out of memory), because the default mode rejects a single request that could never fit.

The three overcommit modes

Set in /proc/sys/vm/overcommit_memory:

The consequence of mode 0 is the important part: "the allocation succeeded" doesn't mean the memory exists. The failure comes later, at whatever line happens to touch a page when no frame is left.

Plan A: swap

If SwapSwapDisk space the OS uses as overflow for RAM: rarely used memory pages are written out to make room. Reading them back is thousands of times slower than RAM, so heavy swapping makes everything crawl. is configured, the kernel first tries to make room: it writes rarely used Anonymous memoryAnonymous memoryMemory not backed by any file: heaps, stacks, Go and V8 heap arenas. It starts as zeros. If the system needs the RAM back, it can only go to swap (or be lost when the process exits). pages to disk and frees their frames (clean file pages are simply dropped, since they can be re-read from the file). Touching a swapped-out page later is a major fault (§05).

A little swap of truly cold pages is harmless. But if the memory programs actively use doesn't fit in RAM, every access evicts something that's needed a moment later: ThrashingThrashingWhen the working memory of running programs doesn't fit in RAM, the OS spends its time moving pages to and from swap instead of running code. The machine is up but practically frozen.. The machine doesn't crash; it becomes so slow that health checks time out and it looks dead. Many production setups run with no swap at all, or very little, preferring a fast, clean kill over a slow death. vm.swappiness (0–200, default 60) tunes how eagerly the kernel swaps anonymous memory versus dropping file cache.

Plan B: the OOM killer

When nothing more can be freed, the kernel's OOM killer picks a victim and sends it SIGKILL (Threads vs Processes §11 covers the signal and exit code 137). It picks using each process's OOM scoreOOM scoreA number from 0 to 1000 per process (/proc/PID/oom_score) that says how good a victim it is for the OOM killer, mostly based on how much memory it uses. oom_score_adj (−1000…1000) lets you shift it., which is mostly "how much memory does it use" on a 0–1000 scale:

$ cat /proc/self/oom_score      # how attractive a victim am I? (real: 666 here)
$ cat /proc/self/oom_score_adj  # my adjustment, −1000 … 1000 (default 0)
$ echo -1000 > /proc/<pid>/oom_score_adj   # never kill this one (needs privileges)

The kernel logs every kill, which is the first place to look after a mysterious restart (dmesg or journalctl -k). The format varies by kernel version; it looks roughly like this (illustrative):

Out of memory: Killed process 4120 (node) total-vm:11834512kB, anon-rss:1984220kB, file-rss:0kB, oom_score_adj:0

total-vm is VSZ and anon-rss is the anonymous part of RSS, the numbers from §07.

In containers: the cgroup's own OOM killer

A container's memory limit is a cgroupcgroupsControl groups: a Linux kernel feature that limits how much CPU, memory and how many processes a group of processes may use. It controls how much a container can use. setting (memory.max in cgroup v2). It's enforced per cgroup: your container can be OOM-killed while the host still has gigabytes free. Before the kill, the kernel tries to reclaim inside the cgroup, starting with page cache, so a container that reads lots of files is not in danger just because its usage graph includes cache. There's also memory.high, a soft limit: above it, the kernel slows the cgroup down and reclaims aggressively instead of killing.

In Kubernetes, the result of a kill is the pod status OOMKilled and exit code 137. Since SIGKILL can't be caught, your program logs nothing. That's why you tell each runtime its budget, so it can work harder before the kernel acts:

# Go 1.19+: a soft limit for the whole Go heap. The GC runs more often near it,
# and the scavenger returns memory to the OS (§08).
GOMEMLIMIT=450MiB ./server        # container limit: 512Mi

mmap vs read(): what databases choose

Two camps

Why the big databases avoid it

A 2022 paper from Carnegie Mellon, "Are You Sure You Want to Use MMAP in Your Database Management System?" (Crotty, Leis, Pavlo), lists the problems. The main ones, in this topic's terms:

1. No say in eviction or write-back. The kernel decides which pages to drop from RAM and when to write dirty pages to disk. A database with a Write-ahead logWrite-ahead log (WAL)An append-only file where a database records each change before applying it. After a crash it replays the log to recover; replicas can follow it to copy changes. must guarantee that a changed page is not written to disk before its log record is safe (Transactions §03 covers why); with a shared mapping, the kernel may write it whenever it likes.
2. Errors become signals. A disk error or a truncated file shows up as SIGBUSSIGBUS (bus error)The signal a process gets when it touches a mapped address whose data can't be produced, e.g. a file page beyond the end of a file that was truncated. Like a segfault, it kills the process unless handled. on some random read (§06), not as an error the code can handle.
3. Stalls you can't see or schedule. A Major page faultMajor page faultA page fault that has to wait for the disk: the data is in a file that isn't cached, or the page was swapped out. Costs from tens of microseconds (SSD) to milliseconds (spinning disk). blocks the thread on an ordinary-looking memory read. In Go, the runtime can't park the goroutine and run another on that thread the way it does for network I/O (Async/Await §09); the OS thread is simply stuck until the disk answers.
4. Scaling costs. Evicting a mapped page means removing it from page tables, and that means TLB shootdownsTLB shootdownWhen the kernel changes or removes a mapping, it must make every CPU core forget its cached (TLB) copy of it. It interrupts the other cores to do that, which gets expensive with many cores. (§04) across every core. The paper measured throughput collapsing on fast NVMe drives once the data no longer fit in RAM.

Why the small ones love it

For a read-mostly store whose hot data fits in RAM, mmap is hard to beat: reads are plain memory loads, the cache survives process restarts (the page cache belongs to the kernel), and several processes can share one copy. LMDB and bbolt also keep the design simple by writing new pages copy-on-write style and never overwriting live data in place, which sidesteps problem 1.

Should you mmap?

For application code, a simple rule:

• Big, read-only, random-access file you'll read many times (a lookup table, a model file, an index you built): mmap is a good fit, in Go via golang.org/x/exp/mmap or syscall.Mmap.
• Streaming through a file once (logs, CSV, uploads): ordinary buffered reads. Readahead makes them fast anyway, and errors stay errors.
• Writing data that must survive a crash: use write + fsyncfsyncA system call that forces data the OS has buffered in memory to actually be written to the disk. Slow (often milliseconds), but without it a power cut can lose “written” data., not a shared mapping. Getting durability right with msync (mmap's version of fsync) is subtle.
• In Node: streams or fs.read with a reused buffer (§06). There's no built-in mmap, and for most Node workloads it isn't the bottleneck.

Huge pages, NUMA, ASLR and Meltdown

Transparent huge pages: free speed, with strings attached

§04 showed random reads getting up to twice as fast with 2 MiB pages. Transparent huge pagesTransparent huge pages (THP)A Linux feature that backs normal memory with 2 MiB huge pages automatically, without the program asking for them explicitly. Mode always, madvise (only where a program asks) or never. tries to give you that without code changes: the kernel backs suitable memory with 2 MiB pages, either directly on a page fault or later, when a background thread called khugepaged merges 512 small pages into one huge one. There are three modes:

$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never          # ← this machine: only where a program asks (madvise)

The catch is that every one of these is a trade:

• Latency spikes. A huge page needs 2 MiB of physically contiguous free RAM. On a long-running server memory gets fragmented, and the kernel may have to stop and shuffle pages around ("compaction") inside a page fault. That's milliseconds, at a random moment, on a request path.
• Memory bloat. Touch one byte of a 2 MiB region and you may get the whole 2 MiB. Sparse heaps can use far more RSS than with 4 KiB pages.
• Bigger copy-on-write. After a forkfork()The Unix system call that clones the calling process. The copy (the child) gets a new PID and a copy of the parent's memory and open files. It returns twice: 0 in the child, the child's PID in the parent., one small write copies 2 MiB instead of 4 KiB (§05). Redis's own docs warn about this and recommend disabling THP.

Database vendors have gone back and forth on this. Redis recommends never; several others historically did too, while MongoDB's recent versions changed their advice towards enabling it. Read your database's current docs rather than a blog post from 2016. madvise mode is a sensible middle ground: programs that know their memory benefits ask for it, nobody else is affected.

Go changed how its runtime interacts with THP in Go 1.22 (backported to 1.21.1), and on some common kernel settings that can make the heap use noticeably more memory. GODEBUG=disablethp=1 (Go 1.21.6+) switches THP off for the Go heap, and the Go GC guide has a section on the recommended Linux THP settings.

NUMA: not all RAM is equally close

Big servers have two or more CPU sockets, and each socket has its own RAM attached. A core reading its own socket's RAM is fast; reading the other socket's goes across a link and is noticeably slower. That's NUMANUMA (non-uniform memory access)On big servers with several CPU sockets, each socket has its own RAM. Memory attached to your own socket is “local” and fast; memory attached to another socket is “remote” and noticeably slower., covered properly in CPU Caches.

Virtual memory decides which socket's RAM a page lands on. Linux's default policy is first touch: the frame comes from the socket of the core that first wrote to the page (§05). So a program that initialises a big array on one thread, then processes it from threads on both sockets, has made half of its accesses remote. Tools: numactl --hardware to see the layout, numastat to see where a process's memory ended up. (This machine is a VM with a single node, so it shows nothing interesting.)

ASLR: random addresses on purpose

If an attacker knows exactly where your stack, heap and libraries are, many memory-corruption exploits become easy. ASLRASLRAddress space layout randomization: the OS puts the stack, heap and libraries at random addresses on each run, so an attacker can't know in advance where useful code or data is. shuffles them on every run. Two runs of Node, reading their own /proc/self/maps:

35069000-3525d000 rw-p ... [heap]    7ffcdb040000-7ffcdb063000 rw-p ... [stack]
42f1d000-43111000 rw-p ... [heap]    7ffc2a7af000-7ffc2a7d2000 rw-p ... [stack]

Same program, different addresses. (/proc/sys/kernel/randomize_va_space is 2, full randomization, on most systems.)

Go is a partial exception, and you saw it in §07: on linux/amd64, Go builds ordinary executables by default, so its code sits at a fixed 0x400000, and the Go heap starts at a fixed hint, 0xc000000000. That's acceptable because Go is memory-safe: without unsafe or cgo, there's no buffer overflow to aim with. go build -buildmode=pie produces a randomizable executable when policy requires one, and some Linux distributions build their Go packages that way.

Meltdown and KPTI: the price of a security fix

For decades, the kernel was mapped into every process's address space (marked "kernel only"), so a System callSystem call (syscall)A request from your program to the operating system — “read this file”, “open a connection”. While a blocking syscall runs, the calling thread waits inside the OS. didn't need to switch page tables. In 2018, Meltdown showed that on many Intel CPUs, user code could read those kernel-only pages anyway. Modern CPUs use Speculative executionSpeculative executionCPUs run instructions ahead of time, guessing they'll be needed, and throw the results away if the guess was wrong. The thrown-away work can still leave measurable traces in the CPU cache.: they run instructions ahead, before they've finished checking whether an access is allowed; the forbidden read is rolled back, but it leaves traces in the CPU cache that an attacker can measure.

The fix, Meltdown & KPTIMeltdown & KPTIMeltdown (2018) let user code on many Intel CPUs read kernel memory that was mapped into its address space. The fix, kernel page-table isolation (KPTI), unmaps the kernel while user code runs, which makes every system call a bit slower., gives each process two sets of page tables: one with the kernel mapped, used while in the kernel, and one with almost nothing of the kernel, used while your code runs. Every system call and interrupt now switches page tables. Reported overhead ranged from negligible to tens of percent for syscall-heavy workloads (databases, proxies doing small reads and writes); PCIDPCIDProcess-context identifiers: an x86 feature that tags each TLB entry with which address space it belongs to, so switching processes doesn't have to throw the whole TLB away. (§04) keeps the TLB from being flushed on each switch and recovers much of it. CPUs designed after 2018 aren't vulnerable, and Linux turns KPTI off on them: this machine reports Not affected in /sys/devices/system/cpu/vulnerabilities/meltdown. System Calls covers what a syscall costs in general.

Pitfalls gallery

Check yourself

Predict the answer before you click. Q1 is covered in §05, Q2 in §02, Q3 in §04, Q4 in §07, Q5 in §08 and Q6 in §09.

buf := make([]byte, 1<<30) // 1 GiB
buf[0] = 1
buf[len(buf)-1] = 1
// read VmRSS from /proc/self/status here

// node --expose-gc app.js
let objs = [];
for (let i = 0; i < 3_000_000; i++) objs.push({ id: i });
objs = null;
gc();
console.log(process.memoryUsage());

Cheat sheet