TCP/IP & the OSI Model

The 30-second version

Here's one line of JavaScript:

const res = await fetch("https://example.com/");

Your code just sent a request and got an answer. But between the two machines there might be 15 boxes, two oceans' worth of cable, and a Wi-Fi radio. None of them knows what HTTP is. So how does it work?

What happens to that one line

Roughly, in order:

1. Find the address. The name example.com is looked up with DNSDNSThe Domain Name System: the internet's phone book that turns a name like example.com into an IP address your computer can connect to. and turned into an IP addressIP addressThe number that identifies a machine on a network, like a postal address: 93.184.216.34 (IPv4) or 2606:2800:220:1::248 (IPv6)., a number like 203.0.113.10. (The DNS topic covers this step.)
2. Open a connection. Your OS opens a TCPTCPTransmission Control Protocol: delivers a stream of bytes between two programs reliably and in order, resending anything lost. HTTP/1.1, HTTP/2 and most databases run on it. connection to that address on portPortA number from 0 to 65535 that picks which program on a machine should get the traffic — like an apartment number after the street address. HTTPS servers usually listen on port 443. 443, the "door" where web servers wait for HTTPSHTTPSHTTP sent through an encrypted TLS connection — the padlock in your browser. Same requests and responses as HTTP, but private and tamper-proof.. (The TCP & UDP topic covers how.)
3. Wrap and send. Your request is wrapped in a few headers and cut into packetsPacketOne small chunk of data sent across a network on its own, with a header saying where it's from and where it's going. Big messages are split into many packets.. Each packet travels on its own, hopping from routerRouterA device that forwards packets between networks, choosing the next hop toward each packet's destination address. Your home Wi-Fi box is one; the internet is thousands of them. to router until it reaches the server.
4. Unwrap and answer. The server's OS strips the headers off again, hands the request to the web server program, and the answer comes back the same way.

This topic is about steps 3 and 4: how bytes actually travel. We'll use plain http:// in most examples, because the encryption that https adds (TLSTLSTransport Layer Security: the encryption layer that makes HTTPS secure. It proves the server is who it claims to be and scrambles everything so nobody in between can read or change it., see the TLS topic) is one more wrapper and doesn't change the picture.

Letters, envelopes and mailbags

That nesting has a name: encapsulationEncapsulation (networking)Wrapping data in a header for the next layer down: your HTTP text goes inside a TCP segment, which goes inside an IP packet, which goes inside an Ethernet frame. Like putting a letter in an envelope, then in a mailbag. (Different from encapsulation in object-oriented code.). It's the single most important idea in networking, and §03 animates it byte by byte.

What you'll be able to do after this topic

• Explain what "L4" and "L7" mean when someone says "L7 load balancer" (§02).
• Read an address like 10.0.0.0/16 and know which machines are in it (§04).
• Debug the classic "works on my laptop, not in Docker" bug in under a minute (§06).
• Understand how one server port handles 10,000 clients at once (§07).
• Read a real packet capture (§09), and know where the clean layer picture breaks in production (§10).

Four layers that run, seven that we name

Why layers at all? Because one giant program that handles Wi-Fi radio and HTTP and lost packets would be impossible to change. With layers, Wi-Fi can be swapped for fibre without touching your browser, and HTTP/2 can replace HTTP/1.1 without touching a single router.

The four layers, bottom to top

Read the "job" column from the bottom row up and you have the whole story of §01: an application message, delivered to a program (transport), on a machine (internet), one hop at a time (link).

Same thing, two maps

The OSI model splits the same territory into seven layers. Two of its extra layers (Session, Presentation) never became separate pieces of software on the internet; in practice they're part of the application. The bottom one, Physical, is the actual cable or radio signal.

Where you'll hear the OSI numbers

You'll almost never need the OSI model to build something, but you'll hear its numbers every week:

• "L3" = the IP layer. "An L3 problem" means routing or addressing.
• "L4" = TCP/UDP. An "L4 load balancer" forwards connections by IP address and port, without reading the data inside.
• "L7" = the application. An "L7 load balancer" or "L7 firewall" reads the HTTP request (path, headers) and decides based on that. The load balancing topic covers the trade-off.
• "L2" = the link layer: MAC addresses, switchesSwitchA box that connects devices on the same local network and forwards each frame to the right port by its MAC address. Unlike a router, it never looks at IP addresses., Wi-Fi.

Nobody says "L5" or "L6" with a straight face.

The most important row: IP is "best effort"

Look again at what the internet layer promises. It's best effortBest effort (delivery)“I'll try, but I promise nothing.” IP forwards each packet as well as it can, but a packet may be dropped, duplicated, delayed or arrive out of order, and IP won't tell anyone.: each router does its best to forward each packet, but a packet may be:

• dropped, because a router's queue was full or a Wi-Fi frame was damaged,
• duplicated, which is rare, but it happens,
• reordered, because packet 2 took a faster path than packet 1,
• delayed by any amount.

And IP doesn't tell anyone when this happens. That's not a bug. It's the design choice that made the internet cheap and scalable: routers stay simple and forget each packet the moment they've forwarded it.

Encapsulation: envelopes inside envelopes

Your code only touches the top layer

Here's an HTTP request sent by hand, with no HTTP library (new to Go? see the Go primer). HTTP really is just text written into a TCP connection:

conn, err := net.Dial("tcp", "example.com:80") // open a TCP connection
if err != nil {
    log.Fatal(err)
}
defer conn.Close()

// The HTTP request is just text. We write it ourselves:
fmt.Fprint(conn, "GET / HTTP/1.1\r\nHost: example.com\r\nConnection: close\r\n\r\n")

io.Copy(os.Stdout, conn) // print whatever the server sends back

Notice what's not in that code: no IP addresses in headers, no packet sizes, no MAC addresses, no resending. You write bytes to a socketSocketThe OS's handle for one open network connection. Your program reads and writes bytes through it., and the operating system does every layer below. Now let's watch it do that.

Down, across, and up

Step through it. The colours match the layers from §02: purple application, cyan transport, amber internet, pink link.

The names change as the data goes down: the transport layer makes a segmentSegment (TCP)One chunk of a TCP byte stream, sent inside one IP packet: a small TCP header (ports, sequence number, ACK number, flags, window) followed by up to ~1460 bytes of your data., the internet layer makes a packet, the link layer makes a frameFrame (Ethernet / Wi-Fi)The outermost wrapper at the link layer: a header with the next device's MAC address, the IP packet inside, and a trailer with a checksum. A frame only travels one hop, then a new one is built.. People mix these words up all the time ("packet" is often used for all of them), but if someone says "frame" they mean the link layer.

The numbers

Two worked examples:

• A tiny request, the 74 bytes of HTTP that curl http://example.com/ sends: 74 + 20 + 20 + 14 + 4 = 132 bytes on the wire, counting a minimal 20-byte TCP header (with the usual 12 bytes of options it's 144, as in the real capture in §09). 58 of those are headers. That's 44% overhead.
• A full-size packet in a big download: Ethernet carries at most 1500 bytes of IP packet (that limit is the MTUMTUMaximum transmission unit: the largest packet a link will carry in one piece, typically 1500 bytes on Ethernet. Bigger data gets split up., more in §08). After 20 B of IP and 20 B of TCP, 1460 B are left for your data. The frame is 1518 B, so headers are only about 4%.

That's why many small messages cost far more than one big one, and why protocols try to batch.

IP addresses, subnets and /24

An IPv4 address is just a 32-bit number

192.168.1.37 looks like four numbers, but it's really one 32-bit number (32 binaryBinaryBase-2 numbers, written with only 0 and 1. Each digit is one bit; 8 bits make a byte, and 11111111 in binary is 255 — which is why each part of an IPv4 address goes from 0 to 255. digits, each 0 or 1) written in four pieces of 8 bits (one byteByte8 bits — enough to store a number from 0 to 255, or one ASCII character. Network speeds are usually quoted in bits per second, file sizes in bytes. each). Each piece goes from 0 to 255. That's 2³² ≈ 4.3 billion possible addresses, which turned out to be far too few for every phone and laptop on Earth. More on that in a moment.

What /24 means

A subnetSubnet (subnetwork)A block of IP addresses that share the same first bits — e.g. every address from 192.168.1.0 to 192.168.1.255. Machines in the same subnet can reach each other directly; anything else goes through a router. is a block of addresses that share their network part. We write it in CIDR notationCIDR notation (/24)Writing a subnet as address/number, e.g. 192.168.1.0/24. The number says how many leading bits are the fixed network part; the rest are free for hosts. /24 leaves 8 bits = 256 addresses. The same thing as a “subnet mask” like 255.255.255.0.: an address, a slash, and how many leading bits are the network part.

Worked example, 192.168.1.0/24:

• 24 bits are fixed: the first three numbers, 192.168.1.
• 8 bits are free: the last number, from 0 to 255. That's 2⁸ = 256 addresses.
• Two are reserved: .0 names the network itself, and .255 is the broadcast addressBroadcast addressThe last address of an IPv4 subnet (e.g. 192.168.1.255 in 192.168.1.0/24). A packet sent there goes to every machine in the subnet, so it can't be given to a single host. ("everyone on this subnet").
• So 254 machines can get an address: 192.168.1.1 to 192.168.1.254.

Every bit you take away from the prefix doubles the block: /23 = 512 addresses, /16 = 65,536, /8 = about 16.7 million. Every bit you add halves it: /30 = 4 addresses (2 usable), and /32 is exactly one address.

You'll also see the same split written as a subnet mask: 255.255.255.0 is the same as /24 (24 one-bits followed by 8 zero-bits). (255 in binary is 11111111: eight one-bits.) Older tools and Windows settings show masks; cloud consoles and Kubernetes use CIDR.

When the cut falls inside a byte

/8, /16 and /24 are easy because the split lands between the dotted numbers. Other prefixes split a number in the middle, so you have to look at its bits. Worked example, 10.0.5.0/26:

• 26 = 24 + 2: the first three numbers are fixed, plus the first 2 bits of the last number.
• That leaves 6 host bits: 2⁶ = 64 addresses, from .0 (bits 00|000000) to .63 (bits 00|111111).
• Network address 10.0.5.0, broadcast 10.0.5.63, so 62 usable hosts.
• The next address, .64, is 01|000000. A network bit changed, so it belongs to the next block, 10.0.5.64/26.

For any address and prefix, drag the slider and watch the purple network bits grow:

Why the split matters: direct, or through a router?

Every machine knows its own address and prefix. Before sending a packet, it asks one question: is the destination in my subnet?

• Yes → it can reach that machine directly on the local network (same Wi-Fi, same switch).
• No → it sends the packet to its router, which forwards it onward. That's §05.

This is why a wrong prefix causes weird bugs. If a machine thinks it's in a /16 but is really in a /24, it will try to reach "neighbours" directly that are actually behind a router, and those packets go nowhere.

Addresses with special meanings

Some ranges are reserved. You'll meet these constantly:

IPv6, briefly

IPv6IPv4 / IPv6Two versions of IP. IPv4 addresses are 32 bits, written as four numbers 0–255 (192.168.1.37) — only about 4.3 billion exist. IPv6 addresses are 128 bits, written in hex groups (2001:db8::1), so there are practically unlimited. fixes the shortage with 128-bit addresses, written as eight groups of four hexHexadecimal (hex)Base-16 numbers using the digits 0–9 and a–f. One hex digit is exactly 4 bits, so a byte is two hex digits: ff = 255. MAC and IPv6 addresses are written this way. digits: 2001:0db8:0000:0000:0000:0000:0000:0001. Leading zeros can be dropped, and one run of all-zero groups can be written as ::, so that's 2001:db8::1.

The ideas carry over unchanged: CIDR prefixes (a normal network is a /64), routers, subnets. A few equivalents to remember: ::1 is loopback, fe80::/10 is link-local, and :: as a listen address means "every interface". With so many addresses, IPv6 doesn't need NAT, but it still needs a firewallFirewallA filter that allows or blocks network traffic by rules — e.g. “only port 443 from the internet”..

Checking addresses in code

import (
    "fmt"
    "net/netip"
)

// inside a function:
subnet := netip.MustParsePrefix("192.168.1.0/24")
ip := netip.MustParseAddr("192.168.1.37")

fmt.Println(subnet.Contains(ip))                                  // true
fmt.Println(subnet.Contains(netip.MustParseAddr("192.168.2.10"))) // false
fmt.Println(ip.IsPrivate(), ip.IsLoopback())                      // true false

Routing: one hop at a time

Your laptop's routing table

Every machine has a routing tableRouting tableA list every host and router keeps: “addresses in this range → send to that next hop, out of that interface”. For each packet it picks the most specific matching row (longest prefix match)., even your laptop. On Linux, ip route shows it (macOS: netstat -rn). A typical one at home:

$ ip route
default via 192.168.1.1 dev wlan0
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.37

Read it line by line:

• 192.168.1.0/24 dev wlan0: "Addresses in my own subnet: send them directly, out of the Wi-Fi interfaceNetwork interfaceOne network connection point on a machine, as the OS sees it: eth0 (wired), wlan0 or en0 (Wi-Fi), lo (loopback). Each has its own IP address(es); a server can choose which ones to listen on. wlan0." This is the "same subnet → direct" rule from §04.
• default via 192.168.1.1: "Everything else: hand it to 192.168.1.1." That router is your default gatewayDefault gatewayThe router your machine sends a packet to when no more specific route matches — usually your Wi-Fi box. In a routing table it's the 0.0.0.0/0 (“everything else”) row., usually your Wi-Fi box. default is short for 0.0.0.0/0, a prefix with zero fixed bits, which matches every address.

Your laptop got both lines automatically when it joined the Wi-Fi. A service on the router called DHCPDHCPThe service (usually running on your router) that hands a device joining the network its IP address, prefix, default gateway and DNS server, so you never type them in by hand. handed it an address, a prefix and the gateway.

The rule: most specific match wins

A router on the internet has many more rows. When several rows match, it picks the one with the longest prefix, the most specific one. Worked example: a packet for 203.0.113.10 arrives at a router with this table:

The packet goes to router B. This is called longest-prefix match, and it's how roughly a million table rows (the approximate size of the full IPv4 internet table today) can cover billions of addresses: broad rows for "roughly that way", specific rows where it matters.

Who writes the tables of the internet's big routers? Networks (ISPs, clouds, universities) announce to their neighbours "I can reach 203.0.113.0/24" using a protocol called BGPBGPBorder Gateway Protocol: how the big networks of the internet (ISPs, clouds) tell each other which address ranges they can reach. It's what fills the routing tables of the internet's core routers., and routers build their tables from those announcements. When someone announces the wrong thing by mistake, large parts of the internet can become unreachable for hours. That really happens.

TTL: the packet's self-destruct counter

What if two routers' tables point at each other by mistake? A packet would bounce between them forever. To prevent that, every IP packet carries a TTLTTL (IP time to live)A counter in every IP packet (called hop limit in IPv6), usually starting at 64 or 128. Each router subtracts 1; at 0 the packet is thrown away, so a routing loop can't keep a packet alive forever. (DNS also uses “TTL”, for something different: how long to cache an answer.) counter. The sender sets it (Linux and macOS use 64, Windows 128), and every router subtracts 1. When it reaches 0, the router drops the packet and sends a small error message back to the sender using ICMPICMPInternet Control Message Protocol: small error and status messages that routers and hosts send each other at the IP layer — “TTL expired”, “destination unreachable”, “packet too big”. ping uses it too., the internet layer's "error and status" protocol.

That error message is what makes the most useful network tool possible. Watch both:

traceroute: seeing the path yourself

traceroutetraceroute / tracertA tool that lists the routers between you and a destination. It sends packets with TTL 1, 2, 3, … and each router where the TTL hits 0 replies with an ICMP “time exceeded” message, revealing itself. exploits TTL on purpose: it sends a probe with TTL 1 (the first router drops it and replies), then TTL 2, then 3, and so on, until the destination itself answers. Each reply reveals one router and the round-trip time (RTTRound-trip time (RTT)How long it takes for a message to go from one machine to another and for the reply to come back. Roughly 1 ms inside a data center, 70–150 ms across an ocean.) to it.

traceroute example.com     # Linux, macOS
tracert example.com        # Windows
mtr example.com            # Linux/macOS: a live, continuously updating traceroute

How to read the output:

• Each line is cumulative. "88 ms" at hop 5 means the round trip from you to hop 5, not the time between hops 4 and 5.
• A big jump between two hops usually means distance. Light in fibre covers about 200 km per millisecond, so crossing an ocean adds tens of milliseconds and no software can fix that.
• * * * in the middle is usually harmless: that router just doesn't reply to probes. Only worry if every line after a point is stars.
• One slow hop with normal hops after it isn't a problem either. Routers answer probes with low priority, but they forward real traffic at full speed.

NAT, localhost and 0.0.0.0

NAT: sharing one public address

§04 said private addresses like 192.168.1.37 are never routed on the internet. Yet your laptop browses the web fine. The trick is NATNATNetwork Address Translation: a router rewrites private addresses (like 192.168.1.20) to its one public address on the way out, and back on the way in. Why your laptop isn't directly reachable from the internet.: your home router keeps a table and rewrites addresses and ports on every packet that crosses it.

Three consequences you'll run into:

• Nobody can connect in by default. NAT only has table rows for conversations started from inside. To run a server at home you add a manual "port forwarding" row, or use a tunnel service (e.g. ngrok or Cloudflare Tunnel, which relay traffic in through a connection your machine opened outward).
• Idle connections get forgotten. NAT rows expire after a period of silence, often minutes, sometimes less. A database connection that sits idle behind NAT can be silently dead when you next use it. TCP keepalive probesTCP keepalive (probes)An OS feature: after a connection has been idle for a while, the kernel sends tiny empty probes and closes the connection if the other side never answers. Not the same as HTTP keep-alive (reusing a connection). and application heartbeat messages exist partly to keep those rows alive.
• You might be behind two NATs. Many ISPsISPInternet service provider: the company that connects your home or office to the internet (and runs the first few routers your packets cross)., especially mobile ones, put customers behind carrier-grade NATCGNAT (carrier-grade NAT)NAT run by your internet provider: many customers share one public IPv4 address. It uses the range 100.64.0.0/10, so your home router's “public” address may not be public at all., so even your router's "public" address is shared.

One machine, several addresses

A machine doesn't have an IP address. Each interface has its own. Your laptop has at least two:

• lo (loopback), address 127.0.0.1: a fake interface that only leads back to the same machine.
• wlan0 or en0 (Wi-Fi), address 192.168.1.37: the one other devices can reach.

When a server starts, it bindsBind (a socket)Telling the OS which local IP address and port a socket should use. A server binds before it listens; binding to 127.0.0.1 means only this machine can connect, 0.0.0.0 means “every interface”. its listening socket to an address and a port. The address decides which interfaces it hears:

// Pick ONE of these two lines.
// Option A: only this machine can connect.
ln, err := net.Listen("tcp", "127.0.0.1:3000")
// Option B: every interface (no host part = all addresses).
// ln, err := net.Listen("tcp", ":3000")

if err != nil {
    log.Fatal(err)
}
defer ln.Close()

// net/http follows the same rule:
//   http.ListenAndServe(":3000", handler)          → all interfaces
//   http.ListenAndServe("127.0.0.1:3000", handler) → this machine only

"Works on my laptop, not in Docker"

A containerContainerA normal Linux process (or a small group of them) that the kernel fences off: it sees only its own files, processes and network, and gets a cap on how much CPU and memory it may use. Docker and Kubernetes run containers. gets its own network stack, using a Linux kernel feature called namespacesNamespaces (Linux)A kernel feature that gives a process its own private view of something — its own list of PIDs, its own network interfaces, its own file system root. It controls what a container can see.. Inside, it has its own lo with its own 127.0.0.1, plus its own eth0 with an address like 172.17.0.2. When you run docker run -p 8080:3000, Docker's port publishingPort publishing (docker -p)docker run -p 8080:3000 tells Docker to forward connections that hit the host's port 8080 to port 3000 on the container's own network address. forwards host port 8080 to the container's eth0, port 3000.

If your server listens on 127.0.0.1 inside the container, it's listening on the container's private loopback, which nothing from outside can ever reach. Toggle it:

The fix is one word: listen on 0.0.0.0 (or leave the host out). Many dev tools default to localhost for safety. Vite, for example, needs --host to listen on all interfaces. So this bug shows up the first time you containerize something.

Ports, sockets and the 4-tuple

Ports: apartment numbers

A port is a 16-bit number, 0 to 65535, in the TCP or UDP header. If the IP address is the building, the port is the apartment. Some numbers are conventions everyone knows:

On Linux, ports below 1024 need extra privileges to listen on, which is why dev servers like to use 3000, 5173 or 8080.

Your side has a port too

When you connect to 203.0.113.10:443, your packets also need a source port, so the replies can find their way back to your program. You don't choose it. The OS picks a free ephemeral portEphemeral portA temporary source port the OS picks for an outgoing connection, from a fixed range (Linux default: 32768–60999). It's freed again shortly after the connection closes. from a range (on Linux, 32768–60999 by default; check with cat /proc/sys/net/ipv4/ip_local_port_range).

So every connection has four addresses, the 4-tuple4-tuple (connection identity)The four values that identify one TCP connection: source IP, source port, destination IP, destination port. No two open connections on a machine can have the same four.:

  src IP          src port     dst IP          dst port
  192.168.1.37    51234   →    203.0.113.10    443

Two connections can share three of the four values, but never all four. Open two browser tabs to the same site and the only difference is the source port. (Strictly, the protocol, TCP or UDP, is a fifth value, so you'll also hear "5-tuple".)

One listening socket, many connected sockets

A server has two kinds of socketsSocketThe OS's handle for one open network connection. Your program reads and writes bytes through it.:

• The listening socket, bound to port 443. It never carries any request data. Its only job is to accept new clients.
• One connected socket per client, created each time a client is accepted. Each is a separate file descriptorFile descriptor (fd)A small number the OS gives your program to identify an open file or socket — e.g. “fd 7”. with its own 4-tuple.

When a packet arrives, the kernel looks up its 4-tuple and hands it to the matching socket. Try it:

So what limits a server to N clients? Not port numbers: the server uses one port for all of them. The real limits are the number of open file descriptors the OS allows (see ulimitulimitPer-user or per-process limits the OS enforces, e.g. max open files (ulimit -n) or max processes/threads (ulimit -u). Hitting one makes calls fail with errors like EMFILE or EAGAIN.) and memory, which on a modern Linux box allows hundreds of thousands of connections.

A complete echo server and client

Here's the whole thing in a few lines. The server repeats back every line it receives; the client sends one line and prints the answer. You can mix them: the Go client works with the Node server and vice versa.

package main

import (
    "bufio"
    "fmt"
    "log"
    "net"
)

func main() {
    ln, err := net.Listen("tcp", "127.0.0.1:9000") // ← bind + listen
    if err != nil {
        log.Fatal(err)
    }
    for {
        conn, err := ln.Accept() // ← waits for a client; returns a NEW socket
        if err != nil {
            log.Println(err)
            continue
        }
        go handle(conn) // one goroutine per connection
    }
}

func handle(conn net.Conn) {
    defer conn.Close()
    log.Printf("new connection: %s → %s", conn.RemoteAddr(), conn.LocalAddr())
    sc := bufio.NewScanner(conn)
    for sc.Scan() { // one line at a time
        fmt.Fprintf(conn, "echo: %s\n", sc.Text())
    }
}

package main

import (
    "bufio"
    "fmt"
    "log"
    "net"
)

func main() {
    conn, err := net.Dial("tcp", "127.0.0.1:9000") // ← OS picks our source port
    if err != nil {
        log.Fatal(err)
    }
    defer conn.Close()
    fmt.Println("me:", conn.LocalAddr(), "server:", conn.RemoteAddr())

    fmt.Fprintf(conn, "hello\n")
    reply, err := bufio.NewReader(conn).ReadString('\n')
    if err != nil {
        log.Fatal(err)
    }
    fmt.Print(reply)
}

Run a client twice (here the Go one, then the Node one) and the output shows the ephemeral port changing:

$ go run client.go
me: 127.0.0.1:44884 server: 127.0.0.1:9000
echo: hello
$ node client.mjs
me: 127.0.0.1:44888 server: 127.0.0.1:9000
echo: hello

The errors you'll see

"Connection refused" is actually good news when debugging: the machine is reachable and answered "nobody's home on that port". A timeout means packets vanished somewhere: a firewall, a wrong route, or a machine that's down.

Down to the wire: MAC, ARP and MTU

MAC addresses: names for the next hop

Every network card has a MAC addressMAC addressA 48-bit hardware address burned into each network card, written in hex like 3c:22:fb:1a:7e:09. It's only meaningful on the local network — routers never forward it., 48 bits written in hexHexadecimal (hex)Base-16 numbers using the digits 0–9 and a–f. One hex digit is exactly 4 bits, so a byte is two hex digits: ff = 255. MAC and IPv6 addresses are written this way.: 3c:22:fb:1a:7e:09. The link layer (EthernetEthernetThe standard for wired local networks (and, by extension, the frame format Wi-Fi traffic is converted to). It moves frames between devices on the same network using MAC addresses., Wi-Fi) uses it to deliver a frame to one device on the same local network. A switchSwitchA box that connects devices on the same local network and forwards each frame to the right port by its MAC address. Unlike a router, it never looks at IP addresses. (or a Wi-Fi access point) learns which MAC sits behind which of its ports and forwards each frame there. It never looks at IP addresses.

MAC addresses never cross a router. As you saw in §03, each router throws the old frame away and builds a new one with its own MAC as the source and the next router's as the destination. IP addresses stay the same for the whole trip (unless a NAT rewrites them, §06); MAC addresses change at every hop.

ARP: "who has 192.168.1.1?"

Your laptop wants to send a packet to 203.0.113.10. The routing table (§05) says: "not local, send it to the gateway 192.168.1.1". But Ethernet and Wi-Fi need a MAC address to put on the frame. Where does that come from?

ARPARPAddress Resolution Protocol: how a machine finds the MAC address for an IP on its local network. It broadcasts “who has 192.168.1.1?” and the owner replies with its MAC. Answers are cached for a while. (IPv6 uses a similar thing called neighbor discovery.):

1. The laptop broadcasts to everyone on the local network: "Who has 192.168.1.1? Tell 192.168.1.37."
2. The router replies, to the laptop only: "192.168.1.1 is at a4:91:b1:0c:22:5e."
3. The laptop caches the answer for a while, so the next packets go straight out.

You can see the cache with ip neigh on Linux (arp -a on macOS and Windows):

$ ip neigh
192.168.1.1 dev wlan0 lladdr a4:91:b1:0c:22:5e REACHABLE

Note what the laptop does not do: it never asks for the MAC of 203.0.113.10. That server is on another network, and its MAC is meaningless here. You only ever ARP for machines in your own subnet, which most of the time means the gateway.

MTU: the biggest packet a link will carry

Every link has a maximum packet size, its MTUMTUMaximum transmission unit: the largest packet a link will carry in one piece, typically 1500 bytes on Ethernet. Bigger data gets split up.. Typical values:

That last row is where trouble starts. Your laptop's link is 1500, the server's link is 1500, but a VPN tunnel in the middle can only carry 1420. What happens to a 1500-byte packet when it reaches that tunnel?

There are two possible answers:

• Fragmentation. In IPv4, the router can cut the packet into smaller fragmentsIP fragmentationSplitting one IP packet into smaller pieces because the next link's MTU is too small. The receiver must reassemble all pieces; lose one and the whole packet is lost. IPv6 routers never fragment., each with its own IP header, and the receiver glues them back together. It works, but it's slow, and losing any one fragment means losing the whole packet. IPv6 routers never do this.
• Drop and tell. If the packet has the Don't Fragment (DF) bitDF bit (Don't Fragment)A flag in the IPv4 header that says “never split this packet”. A router that can't forward it whole drops it and should send back an ICMP “fragmentation needed” message with the smaller MTU. set, the router drops it and sends an ICMP message back: "too big, my limit is 1420". Linux and most other systems set DF on TCP traffic by default.

The second answer is the basis of path MTU discoveryPath MTU discovery (PMTUD)How a sender learns the smallest MTU along the whole path: it sends packets that may not be fragmented and shrinks them when a router replies “too big”. It breaks if those ICMP replies are blocked.: the sender learns the smallest MTU on the path and sends smaller packets from then on. TCP helps too. When a connection opens, each side announces its MSSMSS (maximum segment size)The most data bytes one TCP segment will carry, chosen so the whole packet fits the link's MTU. On normal Ethernet: 1500 − 20 (IP header) − 20 (TCP header) = 1460 bytes. (maximum segment size), normally the local MTU minus 40 bytes of headers: 1500 − 20 (IP) − 20 (TCP) = 1460. You'll see mss 1460 in packet captures (§09).

The PMTUD black hole

Path MTU discovery depends on one thing: the ICMP "too big" message making it back. Some networks block all ICMP "for security". When that happens:

• The connection opens fine. The handshake packets are tiny.
• Small requests and responses work.
• The first full-size packet disappears without a trace, and so does every retry. The download stalls forever.

The symptoms look like an application bug ("only big responses hang", "works on home Wi-Fi, not on the VPN"), but the cause is three layers down. Fixes, in order of preference: let ICMP "fragmentation needed" (IPv4) and "packet too big" (ICMPv6) through; have the tunnel rewrite the MSS in TCP handshakes to a smaller value ("MSS clamping"); or lower the MTU on the affected interface.

Seeing it: ip, ss and tcpdump

The commands below are Linux (they're also what you'll have inside most containers and servers). macOS equivalents are listed at the end of the section.

ip addr: my interfaces and addresses

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 3c:22:fb:1a:7e:09 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.37/24 brd 192.168.1.255 scope global dynamic wlan0
    inet6 fe80::3e22:fbff:fe1a:7e09/64 scope link

Everything from earlier sections is here: the loopback interface lo with 127.0.0.1/8 and ::1 (§04), the Wi-Fi interface with its MAC (link/ether, §08), its IPv4 address and prefix 192.168.1.37/24 with the broadcast address, an IPv6 link-local address, and each interface's MTU.

ip route: where packets go

You met this in §05. default via … is the gateway; the other lines are directly connected subnets. To ask "which route would a packet to X use?", there's a handy shortcut:

$ ip route get 203.0.113.10
203.0.113.10 via 192.168.1.1 dev wlan0 src 192.168.1.37

ss: sockets

ss lists sockets. Recv-Q/Send-Q are bytes waiting in the kernel; on LISTEN rows they show how many connections have finished the handshake but haven't been accepted by your program yet, and the maximum allowed (the accept backlog). The flags you'll use most: -t TCP, -u UDP, -l listening only, -n show numbers instead of names, -p show the owning process (needs root for other users' processes).

$ ss -tlnp
State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port  Process
LISTEN  0       4096         127.0.0.1:5432        0.0.0.0:*      users:(("postgres",pid=812,fd=6))
LISTEN  0       4096           0.0.0.0:22          0.0.0.0:*      users:(("sshd",pid=640,fd=3))
LISTEN  0       511          127.0.0.1:5173        0.0.0.0:*      users:(("node",pid=4120,fd=20))
LISTEN  0       4096                 *:3000              *:*      users:(("server",pid=5301,fd=3))

This answers "is my server listening, and on which address?" in one line, which is the §06 bug at a glance. Postgres and the Vite dev server (port 5173) are local-only; sshd listens on every IPv4 interface; the Go server on 3000 shows *, every address, IPv4 and IPv6. Drop the -l (ss -tnp) to see established connections with their full 4-tuples (§07).

tcpdump: the packets themselves

tcpdump prints every packet that crosses an interface. Here's a capture of curl http://example.com/ (plain HTTP so we can read it; with HTTPS the payload would be encrypted). -nn means "don't turn addresses and ports into names". The output is trimmed: long options […] fields are shortened, and your numbers will differ.

$ sudo tcpdump -i wlan0 -nn 'host 203.0.113.10 and port 80'
10:32:48.815086 IP 192.168.1.37.51234 > 203.0.113.10.80: Flags [S], seq 4135661311, win 64240, options [mss 1460,…], length 0
10:32:48.906211 IP 203.0.113.10.80 > 192.168.1.37.51234: Flags [S.], seq 3095193394, ack 4135661312, win 65160, options [mss 1460,…], length 0
10:32:48.906290 IP 192.168.1.37.51234 > 203.0.113.10.80: Flags [.], ack 1, win 502, length 0
10:32:48.906402 IP 192.168.1.37.51234 > 203.0.113.10.80: Flags [P.], seq 1:75, ack 1, win 502, length 74: HTTP: GET / HTTP/1.1
10:32:48.997815 IP 203.0.113.10.80 > 192.168.1.37.51234: Flags [.], ack 75, win 509, length 0
10:32:48.998127 IP 203.0.113.10.80 > 192.168.1.37.51234: Flags [P.], seq 1:1257, ack 75, win 509, length 1256: HTTP: HTTP/1.1 200 OK
10:32:48.998190 IP 192.168.1.37.51234 > 203.0.113.10.80: Flags [.], ack 1257, win 501, length 0
10:32:48.998542 IP 192.168.1.37.51234 > 203.0.113.10.80: Flags [F.], seq 75, ack 1257, win 501, length 0
10:32:49.089617 IP 203.0.113.10.80 > 192.168.1.37.51234: Flags [F.], seq 1257, ack 76, win 509, length 0
10:32:49.089680 IP 192.168.1.37.51234 > 203.0.113.10.80: Flags [.], ack 1258, win 501, length 0

How to read one line: time, IP, source IP.port > destination IP.port, then the TCP details. The Flags are TCP's: S = SYN (open), . = ACK (got it), P = push (here's data), F = FIN (I'm done). What the ten packets show:

1. Lines 1–3: TCP's opening handshake (SYN, SYN-ACK, ACK). Both sides announce mss 1460: a 1500-byte MTU minus 40 bytes of headers (§08). Notice the ~91 ms gap between line 1 and line 2: that's one round trip to the server.
2. Line 4: your HTTP request, 74 bytes. Port 51234 is curl's ephemeral port (§07).
3. Lines 5–7: the response, 1256 bytes, and the acknowledgements in both directions.
4. Lines 8–10: closing, a FIN from each side.

After the handshake, tcpdump shows sequence numbers relative to the start of the connection, which is why seq 4135661311 becomes seq 1:75. What the handshake, seq, ack and win numbers mean is the TCP & UDP topic's job. What matters here: you can see the 4-tuple, the ports, the round-trip time and the data, all in plain text.

Every layer in one packet

Add -e (show the link-layer header) and -v (verbose IP details) and look at just the request packet. Every layer from §03 is visible:

$ sudo tcpdump -i wlan0 -nn -e -v 'host 203.0.113.10 and port 80'
10:32:48.906402 3c:22:fb:1a:7e:09 > a4:91:b1:0c:22:5e, ethertype IPv4 (0x0800), length 140: (tos 0x0, ttl 64, id 23702, offset 0, flags [DF], proto TCP (6), length 126)
    192.168.1.37.51234 > 203.0.113.10.80: Flags [P.], cksum 0x1c3a (incorrect -> 0xc4fa), seq 1:75, ack 1, win 502, options […], length 74: HTTP, length: 74
	GET / HTTP/1.1
	Host: example.com
	User-Agent: curl/8.5.0
	Accept: */*

The TCP header here is 32 bytes, not 20: 12 bytes of options (timestamps) ride along on every packet.

macOS and Windows equivalents

Where the layers leak (staff-level)

1. MTU: tunnels, overlays and Kubernetes

Every tunnel wraps packets in more headers, which shrinks the room left for your data. VPNs do it, and so do cloud and Kubernetes overlay networksOverlay networkA virtual network built on top of a real one by wrapping each packet in another packet (e.g. VXLAN). It lets pods on different machines talk as if on one network — at the cost of extra headers (often 50 bytes), which shrinks the usable MTU.. VXLAN, a common overlay, adds 50 bytes, so pods get an MTU of 1450 on a 1500-byte network. If any part of the system still assumes 1500, and ICMP "too big" is filtered somewhere, you get the §08 black hole. It tends to show up under rare conditions: only in one cluster, only for large responses, only through one VPN.

What staff engineers do about it:

• Know the MTU of every hop you own: ip link on nodes, the CNI (Kubernetes network plugin) config, the cloud's documented values (some clouds support 9000-byte jumbo frames inside a VPCVPC (virtual private cloud)A private network you rent inside a cloud provider such as AWS or GCP, with its own private IP ranges, subnets and routing tables. but 1500 or less across VPNs and peering).
• Clamp the MSS at tunnel edges, so TCP never builds segments that won't fit, even if ICMP is lost.
• Allow ICMP "fragmentation needed" / "packet too big" in security groups and firewalls. Blocking it buys no real security and breaks PMTUD.
• Linux can also probe for the right size without relying on ICMP (the net.ipv4.tcp_mtu_probing setting), a useful safety net on hosts behind networks you don't control.

2. TCP over lossy Wi-Fi and mobile links

TCP was designed when lost packets almost always meant one thing: a router somewhere was overloaded. So when TCP sees loss, it slows down. On Wi-Fi and mobile links, packets are also lost to radio interference, which slowing down doesn't fix. The link layer partly hides this by retrying frames itself, but every retry adds delay and jitter (varying delay). The result is a familiar mix: low throughput even though the "bandwidth" is high, and latency spikes that look like server slowness. Link-layer behaviour leaks straight into transport-layer performance. How TCP's congestion control decides how fast to send is covered in the TCP & UDP topic.

3. Middleboxes: why protocols "ossify"

In theory only the two endpoints read the TCP header. In practice, middleboxesMiddleboxAny device in the path that does more than forward packets: firewalls, NAT boxes, load balancers, corporate proxies. They often look inside TCP or even HTTP, which is why new protocols are hard to deploy. (firewalls, NAT, corporate proxies, carrier equipment) read it, rewrite it, and sometimes drop packets with options they don't recognise. So adding new features to TCP became nearly impossible, because some box somewhere would break. Engineers call this ossification.

This is a big part of why QUICQUICA transport protocol built on UDP that does TCP's job (reliable, ordered delivery, congestion control) plus encryption, with independent streams so a lost packet only delays the stream it belongs to. HTTP/3 runs on it. (the transport under HTTP/3) runs over UDP and encrypts almost all of its own headers: if middleboxes can't read the headers, they can't depend on them, and the protocol can keep evolving. See the HTTP topic. NAT idle timeouts (§06) are another middlebox leak: long-lived idle connections die silently unless something sends keepalive probesTCP keepalive (probes)An OS feature: after a connection has been idle for a while, the kernel sends tiny empty probes and closes the connection if the other side never answers. Not the same as HTTP keep-alive (reusing a connection). or heartbeat messages.

4. Anycast: one address, many places

Normally one IP address means one machine. With anycastAnycastAnnouncing the same IP address from many places at once. Routers send each packet to the nearest copy, so 1.1.1.1 or a CDN address is answered by a server in your region., many sites announce the same prefix through BGPBGPBorder Gateway Protocol: how the big networks of the internet (ISPs, clouds) tell each other which address ranges they can reach. It's what fills the routing tables of the internet's core routers., and each router sends packets to the nearest one. That's how 1.1.1.1 and 8.8.8.8 answer from your own region, how a CDNCDNContent Delivery Network: servers spread around the world that keep copies of your files close to users, so they're served from a nearby city instead of your origin server. sends you to a nearby edge, and how DNS root servers survive attacks.

The leak: "nearest" means nearest by routing, which isn't always nearest geographically. Also, a routing change in the middle of a connection can move your packets to a different site that has never heard of your TCP connection. In practice this is rare, and anycast works best for short connections (DNS) or when the sites share connection state.

5. IPv6, dual stack and Happy Eyeballs

On a dual-stackDual stackA machine or network that runs IPv4 and IPv6 side by side. A hostname can then resolve to both kinds of address, and the client has to choose. network, example.com may resolve to both an IPv6 and an IPv4 address. If the client tries IPv6 first and IPv6 is broken somewhere on the path, a naive client waits for a full connect timeout, often many seconds, before falling back. Users experience this as "the internet is slow".

The standard fix is Happy EyeballsHappy EyeballsA connection trick for dual-stack clients: try IPv6 first, and if it hasn't connected within a short delay (~250 ms), start IPv4 in parallel and use whichever wins. It hides broken IPv6 from users.: start IPv6, and if it hasn't connected within a short delay, start IPv4 in parallel and keep whichever connects first.

• Go does this by default in net.Dialer. Its FallbackDelay defaults to 300 ms.
• Node does it by default since v20 (autoSelectFamily: true), trying each address for 250 ms (net.getDefaultAutoSelectFamilyAttemptTimeout()).

If you write your own dialing logic (custom DNS, connection pools that pin to one address), make sure you didn't throw this behaviour away.

6. Containers and Kubernetes, in one picture

§06 said a container has its own network stack. Here's what Docker builds to make that work:

The pieces: each container is a network namespaceNamespaces (Linux)A kernel feature that gives a process its own private view of something — its own list of PIDs, its own network interfaces, its own file system root. It controls what a container can see. with its own lo and eth0. A veth pairveth pairA virtual network cable on Linux: two interfaces where whatever goes in one comes out the other. Docker puts one end inside the container and the other on the host. acts as a virtual cable from the container to a bridgeBridge (Linux)A virtual switch inside the Linux kernel. Docker's docker0 is one: every container's veth end plugs into it, so containers on the same host can reach each other. on the host. Outgoing traffic is NAT-ed to the host's address, and published ports are forwarded in. Every concept from this topic appears again, just implemented in software inside one kernel.

Kubernetes changes one rule: every podPod (Kubernetes)The unit Kubernetes runs and restarts: one or a few containers that live and die together. Stopping a pod is how Kubernetes stops your process during a deploy. gets its own routable IP, and any pod can reach any other pod without NAT. The network plugin makes that true across machines, either with plain routes or with an overlay (hence the MTU issue in point 1). Kubernetes Service IPs are stranger: no interface owns them. They're virtual addresses that the kernel rewrites into a real pod's IP as connections are made (kube-proxy programs those rules). That's why, in kube-proxy's default iptables mode, pinging a Service IP usually fails while curl to it works: ICMP isn't covered by the rewrite rules.

7. Where latency hides, layer by layer

When a request is slow, the time went somewhere in the stack. A rough map (all numbers approximate):

The lesson: with a 90 ms round trip, a new HTTPS connection spends 2 round trips to the server (TCP handshake + TLS 1.3 handshake), about 180 ms, plus a DNS lookup if the name isn't cached, before the first byte of your request is even sent. That's why connection reuse (keep-aliveKeep-alive (connection reuse)Leaving a connection open after a request finishes so the next request can reuse it, skipping the cost of a new handshake.), CDNs close to users and fewer round trips matter more than raw bandwidth. Measure it instead of guessing:

curl -o /dev/null -s -w 'dns %{time_namelookup}  tcp %{time_connect}  tls %{time_appconnect}  first byte %{time_starttransfer}  total %{time_total}\n' https://example.com/

start := time.Now()
at := func(what string) { fmt.Printf("%-14s %v\n", what, time.Since(start).Round(time.Millisecond)) }

trace := &httptrace.ClientTrace{ // hooks the HTTP client calls at each stage
    DNSDone:              func(httptrace.DNSDoneInfo) { at("dns done") },
    ConnectDone:          func(network, addr string, err error) { at("tcp connected") },
    TLSHandshakeDone:     func(tls.ConnectionState, error) { at("tls done") },
    GotFirstResponseByte: func() { at("first byte") },
}
req, err := http.NewRequest("GET", "https://example.com/", nil)
if err != nil {
    log.Fatal(err)
}
req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
res, err := http.DefaultClient.Do(req)
if err != nil {
    log.Fatal(err)
}
defer res.Body.Close()
io.Copy(io.Discard, res.Body) // read to the end so the connection can be reused
at("done")

Pitfalls gallery

addr := net.JoinHostPort("::1", "8080") // "[::1]:8080"  ← adds brackets when needed
conn, err := net.Dial("tcp", addr)

Check yourself

First a warm-up: which layer does each thing belong to? (§02, §03)

Now predict the outcome before you click. Q1 is §06, Q2 is §04, Q3 is §03 and §05, Q4 is §07, Q5 is §08, and Q6 is §05.

// server.mjs, inside a container started with: docker run -p 3000:3000 my-app
import http from "node:http";
http.createServer((req, res) => res.end("ok"))
    .listen(3000, "127.0.0.1");

// on the host:  curl http://localhost:3000

You get the subnet 10.0.5.0/26 for a group of servers.
How many of them can get an address?

A packet goes laptop → router → router → server (no NAT involved).
Which of these is different when it leaves the first router
compared to when it arrived?

Your browser opens two connections to 203.0.113.10:443
from your laptop at 192.168.1.37. What differs between them?

After the company added a VPN, logging in to an internal site works,
but downloading a large file stalls forever at 0 bytes.
The network team "blocks all ICMP for security".

$ traceroute 203.0.113.10
 1  192.168.1.1     3.1 ms
 2  198.51.100.1    9.0 ms
 3  * * *
 4  198.51.100.130 88.2 ms
 5  203.0.113.10   91.0 ms

Cheat sheet