Text & Encodings

The 30-second version

Files, network packets and memory hold only bytesByte8 bits — enough to store a number from 0 to 255, or one ASCII character. Network speeds are usually quoted in bits per second, file sizes in bytes.: numbers from 0 to 255. A byte has no idea whether it's part of a photo, a price or a poem. Text appears only when a program reads the bytes using an agreed mapping from numbers to characters. That mapping is called an EncodingEncodingRewriting data in a different format so it fits somewhere: UTF-8 turns text into bytes, Base64 and hex turn bytes into text. There's no key; anyone can decode it. It hides nothing..

Modern text goes through up to four layers. Each one answers a different question, and most confusion comes from mixing them up:

What you'll be able to do

Bytes are just numbers: ASCII

Bits, bytes and the two ways to write them

A BitBitThe smallest unit of data: a single 0 or 1. Eight bits make a byte. is a single 0 or 1. Eight bits make a byte, and 8 bits can be arranged in 2⁸ = 256 different ways, so a byte holds a number from 0 to 255.

You'll see the same byte written three ways. They're all the same number:

Hex counts to 15 in one digit: 0–9, then a = 10 up to f = 15. So e9 = 14 × 16 + 9 = 233. Hex is popular because it's short and lines up with bits: 4 is 0100, 8 is 1000, so 48 is 0100 1000. Tools that dump raw bytes almost always print hex.

ASCII: 128 characters, 7 bits

In 1963 American engineers published ASCIIASCIIA 1960s character set of 128 characters (English letters, digits, punctuation and control codes), each stored in one byte. UTF-8 is identical to ASCII for those 128 characters.: a table of 128 codes (0–127), enough for English letters, digits, punctuation and some machine instructions. 128 values need only 7 bits, so every ASCII character fits in a byte with the top bit set to 0.

Notice two design tricks. The digit 7 is code 55, not 7: the character "7" and the number 7 are different things. And lowercase is always uppercase + 32 (A = 65, a = 97), a single bit's difference, which made case conversion cheap on 1960s hardware.

Seeing the bytes of a string

Here's "Hi!" as bytes in both languages, printed as decimal, hex and binary:

package main

import "fmt"

func main() {
    b := []byte("Hi!")       // the string's bytes
    fmt.Println(b)           // as decimal numbers
    fmt.Printf("% x\n", b)   // as hex, space-separated
    fmt.Printf("%08b\n", b)  // as 8-bit binary

    back := string([]byte{72, 105, 33}) // numbers → text
    fmt.Println(back)
}
// Output:
// [72 105 33]
// 48 69 21
// [01001000 01101001 00100001]
// Hi!

From a terminal you can dump any file's bytes with od (on most systems xxd or hexdump -C work too):

$ printf 'Hi!' | od -An -tx1
 48 69 21

ASCII covers English and nothing else: no é, no ñ, no ü, no 中. The other 128 values a byte can hold (128–255) were left unused, and that's where the trouble starts.

Code pages and mojibake

One byte, many meanings

ASCII used codes 0–127. Every country then filled 128–255 with its own letters, producing dozens of 8-bit tables called code pagesCode pageOne of many 8-bit character sets from the 1980s–90s. All of them agree with ASCII for bytes 0–127 but give bytes 128–255 different meanings: Windows-1252 for Western Europe, Windows-1251 for Cyrillic, CP437 for old IBM PCs, and so on.. They all agree with ASCII for the first 128 codes and disagree about the rest:

A file is just bytes. Nothing inside it says which table was used. So a French email with byte 0xE9 arrives in Moscow as "й", and nobody did anything "wrong" except assume.

The classic: café

That garbage has a name: MojibakeMojibakeGarbled text caused by reading bytes with a different encoding than the one used to write them, e.g. “café” showing up as “café”. The bytes are fine; the reader guessed the wrong table. The word is Japanese for “character transformation”..

Today almost everything writes UTF-8UTF-8The standard way to turn text into bytes. Plain English letters take 1 byte each; accented letters, other scripts and emoji take 2–4 bytes. (§05 explains it). In UTF-8, "é" takes two bytes: c3 a9. A program that still assumes Windows-1252 reads those two bytes as two separate characters: c3 → "Ã" and a9 → "©". That's why you see café.

It can get worse. If someone saves that garbage back as UTF-8 and it gets misread again, each wrong character grows again:

package main

import (
    "fmt"

    "golang.org/x/text/encoding/charmap" // go get golang.org/x/text
)

func main() {
    b := []byte("café")         // written as UTF-8
    fmt.Printf("% x\n", b)

    // The reader wrongly assumes Windows-1252:
    wrong, _ := charmap.Windows1252.NewDecoder().Bytes(b)
    fmt.Println(string(wrong))

    // The garbage is saved as UTF-8 again, then misread again:
    twice, _ := charmap.Windows1252.NewDecoder().Bytes(wrong)
    fmt.Println(string(twice))
}
// Output:
// 63 61 66 c3 a9
// café
// café

Try it yourself. Choose how the text was written and how it's read, then watch the same bytes turn into different text:

Recognizing mojibake in the wild

Unicode: one number for every character

Code pages failed because there were dozens of small tables. In the early 1990s the Unicode Consortium started building one table for all of them: Latin, Greek, Arabic, Chinese, Hindi, maths symbols, music notes, and later emoji.

Code points

Each entry's number is its Code pointCode pointThe number Unicode assigns to one character, written U+ followed by hex digits (é is U+00E9). An encoding such as UTF-8 decides how that number is stored as bytes., written U+ followed by at least four hex digits:

Two compatibility choices made adoption painless. The first 128 code points are ASCII (A is 65 in both), and the first 256 match Latin-1, so é is 233 in both too.

How big is the table?

Code points run from U+0000 to U+10FFFF: room for 1,114,112 values. About 173,000 are assigned so far (172,808 in Unicode 18.0, released September 2026; as of October 2026 that's the latest version); the rest is reserved for the future. The space is split into 17 planesPlane (Unicode)A block of 65,536 code points. Unicode has 17 planes (U+0000 to U+10FFFF). Plane 0, the Basic Multilingual Plane, holds almost all living scripts; emoji and rare scripts live in the higher planes. of 65,536 each:

The first plane, the Basic Multilingual PlaneBasic Multilingual Plane (BMP)Plane 0 of Unicode: code points U+0000 to U+FFFF. Latin, Greek, Cyrillic, Arabic, Chinese, Japanese, Korean and most symbols live here. In UTF-16 every BMP character takes exactly one code unit., holds nearly every character in everyday use. Emoji live mostly in plane 1. That detail looks like trivia now, but it's exactly why "😀".length is 2 in JavaScript (§06).

Asking for code points in code

Both languages can walk a string one code point at a time:

package main

import "fmt"

func main() {
    for _, r := range "Aé€😀" { // range decodes one code point (a rune) at a time
        fmt.Printf("%c  %U  decimal %d\n", r, r, r)
    }
}
// Output:
// A  U+0041  decimal 65
// é  U+00E9  decimal 233
// €  U+20AC  decimal 8364
// 😀  U+1F600  decimal 128512

Unicode is a table, not a file format

This is the distinction that clears up most confusion:

"This file is in Unicode."
(Which bytes? Unicode alone
doesn't say.)

character → code point   (Unicode: é → U+00E9)
code point → bytes       (an encoding: UTF-8 → c3 a9,
                          UTF-16BE → 00 e9)

Unicode decides which number. An encoding such as UTF-8 or UTF-16 decides which bytes store that number. The same code point becomes different bytes in different encodings, which is why "save as Unicode" in old Windows dialogs (it meant UTF-16) produced files other tools couldn't read. Here's one string stored three ways (§05 and §06 explain UTF-8 and UTF-16 in detail; for now just notice the code points stay the same):

UTF-8: from code point to bytes

The problem it solves

The biggest code point, U+10FFFF, needs 21 bits. The simplest encoding, UTF-32, spends a fixed 4 bytes on every character. That works, but English text becomes 4× bigger than ASCII, and every old ASCII file would need converting.

UTF-8 (designed in 1992 by Ken Thompson and Rob Pike, who later co-created Go) uses as few bytes as each character needs:

The fixed bits at the start of each byte are the whole trick. The first byte of a character, the lead byteLead byte / continuation byteIn UTF-8, the first byte of a character is the lead byte; its top bits say how many bytes the character has (0xxxxxxx = 1, 110xxxxx = 2, 1110xxxx = 3, 11110xxx = 4). Every following byte is a continuation byte starting with 10., announces the length: 0 = 1 byte, 110 = 2, 1110 = 3, 11110 = 4. Every following byte is a continuation byte and starts with 10.

Worked example: é

1. é is U+00E9 = 233. In binary that's 11101001 (8 bits).
2. 8 bits is too many for the 1-byte form (7 payload bits), so use the 2-byte form, which has 11 slots. Pad with zeros on the left: 000 1110 1001.
3. Cut from the right: the last 6 bits 101001 go in the continuation byte, the remaining 5 bits 00011 go in the lead byte.
4. Lead byte: 110 + 00011 = 11000011 = 0xC3. Continuation byte: 10 + 101001 = 10101001 = 0xA9.

So é is c3 a9, the two bytes that turned into "é" in §03. Step through it, then try € and 😀:

package main

import (
    "fmt"
    "unicode/utf8"
)

func main() {
    for _, s := range []string{"A", "é", "€", "😀"} {
        r, _ := utf8.DecodeRuneInString(s)
        fmt.Printf("%s  %U  %d byte(s): % x\n", s, r, len(s), s)
    }
}
// Output:
// A  U+0041  1 byte(s): 41
// é  U+00E9  2 byte(s): c3 a9
// €  U+20AC  3 byte(s): e2 82 ac
// 😀  U+1F600  4 byte(s): f0 9f 98 80

Why UTF-8 won

1. ASCII is unchanged. Every ASCII file is already a valid UTF-8 file, byte for byte, and English text costs nothing extra. Old tools that only look for /, , or newline keep working, because those bytes (all below 128) never appear inside a multi-byte character: lead and continuation bytes always have the top bit set.

2. It's Self-synchronizingSelf-synchronizingAn encoding where you can start reading at any byte and find the next character boundary on your own. In UTF-8 you just skip bytes that start with 10. A lost or damaged byte breaks one character, not the rest of the text.. Lead bytes and continuation bytes look different, so a program dropped anywhere in the middle of a stream can find the next character start by skipping bytes that begin with 10:

That's why one corrupted byte damages only one character, and why searching UTF-8 text for a UTF-8 word can never match half a character.

3. No byte-order question. UTF-8 works one byte at a time, so it has no EndiannessEndiannessThe order in which a multi-byte number's bytes are stored: big-endian puts the most significant byte first, little-endian (most CPUs today) puts it last. problem: the bytes are the same on every CPU. UTF-16 and UTF-32 store 2- and 4-byte numbers, so they come in big- and little-endian versions.

4. Sorting the bytes sorts the code points. Comparing two UTF-8 strings byte by byte gives the same order as comparing their code points. That isn't the alphabetical order a human expects (§12), but it's consistent everywhere.

Today about 98% of websites use UTF-8 (approximate, from web surveys in 2025), as do Go source files, JSON (required by RFC 8259), and, by convention, Linux and macOS file names.

UTF-16, JavaScript strings and Go strings

UTF-16 in one minute

Unicode started out as a fixed 16-bit code called UCS-2, back when 65,536 characters seemed enough for everyone. When that ran out, UTF-16UTF-16An encoding that stores each code point as one or two 16-bit numbers (code units). Characters in the BMP take one unit; everything above U+FFFF, such as most emoji, takes two (a surrogate pair). JavaScript, Java and Windows use it inside. (1996) became the patch: it keeps the 16-bit units and adds a trick for everything above. It stores text in 16-bit code unitsCode unitThe fixed-size chunk an encoding works in: 8 bits for UTF-8, 16 bits for UTF-16. One code point takes one or more code units. JavaScript's .length counts UTF-16 code units, not characters.:

• A code point in the Basic Multilingual PlaneBasic Multilingual Plane (BMP)Plane 0 of Unicode: code points U+0000 to U+FFFF. Latin, Greek, Cyrillic, Arabic, Chinese, Japanese, Korean and most symbols live here. In UTF-16 every BMP character takes exactly one code unit. (up to U+FFFF) is stored as one code unit with the same value: é → 00E9.
• A code point above U+FFFF, like most emoji, doesn't fit in 16 bits. It's split across two code units called a Surrogate pairSurrogate pairTwo UTF-16 code units that together stand for one code point above U+FFFF. The first is in the range D800–DBFF (high surrogate), the second in DC00–DFFF (low surrogate). Alone, either half is meaningless..

Unicode reserved the range D800–DFFF for exactly this: no character will ever have those code points, so a code unit in that range is always half of a pair. Here's how 😀 is split:

JavaScript (1995), Java and the Windows API all adopted 16-bit strings in the UCS-2 days, before emoji existed, and kept them for compatibility. That's why their strings are UTF-16 code units today.

JavaScript: length counts code units

const s = "hi😀";
console.log(s.length);                         // UTF-16 code units
console.log(s.charCodeAt(2).toString(16), s.charCodeAt(3).toString(16));
console.log(s.codePointAt(2).toString(16));    // the whole code point
console.log([...s].length);                    // code points
console.log(JSON.stringify(s.slice(0, 3)));    // cuts the emoji in half
// Output:
// 4
// d83d de00
// 1f600
// 3
// "hi\ud83d"

What the JavaScript lines show:

• length is 4: h, i, and the two halves of 😀.
• charCodeAt(i) returns one code unit; codePointAt(i) notices a pair starting at i and returns the full code point.
• [...s] (spread) and for...of iterate by code point, so they keep the emoji whole.
• slice, substring, s[i] and split("") work on code units and can leave a lone surrogate (Lone surrogateLone surrogateHalf of a UTF-16 surrogate pair with no partner, e.g. after slicing an emoji in JavaScript. It isn't valid text: encoders turn it into � (U+FFFD) and encodeURIComponent throws.): half an emoji that renders as � and isn't valid text. That's how a "truncate to 100 characters" helper corrupts emoji.

Go: a string is bytes

In Go, a string is a read-only sequence of bytes: it behaves like a []byte you can't change (no append, no assigning to s[i]). By convention, and always for string literals, those bytes are UTF-8.

package main

import (
    "fmt"
    "unicode/utf8"
)

func main() {
    s := "hé😀"
    fmt.Println(len(s))                    // bytes
    fmt.Println(s[1], s[2])                // indexing gives single bytes
    fmt.Println(utf8.RuneCountInString(s)) // code points
    fmt.Println(len([]rune(s)))            // same, by converting

    for i, r := range s { // i = byte offset, r = decoded rune
        fmt.Printf("%d:%c ", i, r)
    }
    fmt.Println()
    fmt.Printf("%q\n", s[:2]) // slicing by bytes can cut a character in half
}
// Output:
// 7
// 195 169
// 3
// 3
// 0:h 1:é 3:😀
// "h\xc3"

• len(s) counts bytes: h (1) + é (2) + 😀 (4) = 7.
• s[1] is a single byte (195 = 0xC3, the lead byte of é), not a character.
• for i, r := range s decodes UTF-8 as it goes: r is a RuneRune (Go)Go's type for one Unicode code point (an int32), which is usually but not always one visible character: an emoji with a skin tone is several runes. Ranging over a string with for _, ch := range s gives you runes, so “é” or “世” arrive as one value even though they take several bytes. (code point) and i is the byte offset where it starts. Offsets jump 0, 1, 3 because é took two bytes.
• []rune(s) converts to a slice of code points; utf8.RuneCountInString(s) counts them without allocating.
• s[:2] slices by bytes and here cuts é in half, leaving the lone byte \xc3.

See it all at once

Type anything below. Each card is one code point: its UTF-8 bytes in binary (the colored prefix bits from §05) and its UTF-16 code units. Try an emoji and watch it turn into a surrogate pair.

Graphemes: the length is a lie

Characters built from pieces

Unicode lets you combine code points. Three common cases:

• Accents. é can be the single code point U+00E9, or e followed by U+0301, a Combining markCombining markA code point that doesn't stand alone but modifies the character before it, like U+0301 COMBINING ACUTE ACCENT: e followed by U+0301 displays as é. that puts an acute accent on the letter before it. On screen they look identical.
• Flags. There's no "Vietnam flag" code point. 🇻🇳 is two special letters, 🇻 (U+1F1FB) and 🇳 (U+1F1F3), which fonts draw as a flag when they appear together.
• Emoji sequences. 👍🏽 is 👍 plus a skin-tone modifier (U+1F3FD). 👨‍👩‍👧 is man, woman and girl glued together by two invisible zero-width joinersZero-width joiner (ZWJ)The invisible code point U+200D. Placed between emoji, it asks the font to draw them as one combined picture: man + ZWJ + woman + ZWJ + girl draws a family. Fonts that don't know the combination show the pieces side by side. (U+200D): five code points.

The unit a human perceives as one character is a Grapheme clusterGrapheme clusterWhat a reader sees as one character: one “user-perceived character”. It can be one code point (a), several (e + accent, a flag made of two letters) or many (a family emoji made of 5–7 code points). Cursor movement and backspace work on graphemes.. It's what your cursor jumps over and what Backspace deletes (mostly; editors differ).

Four honest answers to "how long is it?"

None of these numbers is wrong. They answer different questions. Pick presets or type your own:

Counting graphemes in code

Unicode publishes the rules for where graphemes start and end (they change a little each year as new emoji arrive). JavaScript has them built in as Intl.SegmenterIntl.SegmenterA built-in JavaScript API (all modern browsers, Node 16+) that splits text into graphemes, words or sentences using Unicode's rules. new Intl.Segmenter("en", { granularity: "grapheme" }).segment(s) gives you the user-perceived characters.. Go's standard library doesn't, so use the widely used github.com/rivo/uniseg package:

package main

import (
    "fmt"
    "unicode/utf8"

    "github.com/rivo/uniseg" // go get github.com/rivo/uniseg
)

func main() {
    for _, s := range []string{"cafe\u0301", "🇻🇳", "👨‍👩‍👧"} {
        fmt.Printf("%s bytes=%d runes=%d graphemes=%d\n",
            s, len(s), utf8.RuneCountInString(s), uniseg.GraphemeClusterCount(s))
    }
}
// Output (each café is e + U+0301):
// café bytes=6 runes=5 graphemes=4
// 🇻🇳 bytes=8 runes=2 graphemes=1
// 👨‍👩‍👧 bytes=18 runes=5 graphemes=1

Which length should I use?

Code points are rarely the right answer on their own. They're the unit Unicode's rules are written in, but users don't see them and storage isn't measured in them.

Drag the cut point and compare the three ways of cutting:

Normalization and case: when é !== é

Same picture, different bytes

In §07, é could be one code point (U+00E9) or two (e + U+0301). Which one you get depends on where the text came from: macOS file names have historically used the two-code-point form, most keyboards on Windows and Linux produce the one-code-point form, and copy-paste carries whatever the source used.

To === and ==, those are simply different strings:

package main

import (
    "fmt"

    "golang.org/x/text/unicode/norm" // go get golang.org/x/text
)

func main() {
    a := "caf\u00e9"  // é as ONE code point (U+00E9)
    b := "cafe\u0301" // e + combining acute accent (U+0301)

    fmt.Println(a, b, a == b)
    fmt.Println(len(a), len(b)) // bytes
    fmt.Println(norm.NFC.String(a) == norm.NFC.String(b))
    fmt.Println(len(norm.NFD.String(a)))

    fmt.Println(norm.NFKC.String("file ① Ｈｉ"))
}
// Output (the second café is e + U+0301):
// café café false
// 5 6
// true
// 6
// file 1 Hi

The four normalization forms

Unicode normalizationUnicode normalization (NFC / NFD)Converting text to one standard form so that equal-looking strings have equal code points. NFC composes (e + ◌́ → é as one code point); NFD decomposes (é → e + ◌́). Compare and store text in one form, usually NFC. rewrites text into one standard form, so equal-looking text gets equal code points:

The K forms (compatibility normalizationCompatibility normalization (NFKC / NFKD)A stronger normalization that also folds look-alike “compatibility” characters into plain ones: fi → fi, ① → 1, full-width Ｈ → H. Useful for usernames and search; it loses formatting, so don't use it to store documents.) go further: the ligature fi becomes f + i, ① becomes 1, full-widthFull-width charactersWide versions of Latin letters and digits (Ｈ, １) that line up with Chinese and Japanese characters. They have their own code points; NFKC turns them into the normal ones. Ｈ (a wide Latin letter used when typing Chinese or Japanese) becomes H. That's great for matching what a user meant, but it changes meaning (x² becomes x2), so don't apply it to documents you store. Explore each form:

Case-insensitive comparison is harder than toLowerCase()

Lowercasing both sides works for English and breaks elsewhere:

• German ß uppercases to SS: two letters from one. JavaScript's "straße".toUpperCase() gives "STRASSE", but Go's strings.ToUpper keeps ß (it only does one-to-one mappings), giving "STRAßE".
• Turkish has a dotted İ/i and a dotless I/ı. "TITLE".toLocaleLowerCase("tr") is "tıtle", so the same code gives different results depending on the user's language setting.

The right tool is Case foldingCase foldingTurning text into a form for case-insensitive comparison. It's more than lowercasing: German ß folds to ss, and some languages (Turkish) have dotted and dotless i. Use a library's fold or compare function, not toLowerCase() on both sides., a mapping designed for comparison, not display:

package main

import (
    "fmt"
    "strings"

    "golang.org/x/text/cases"
)

func main() {
    fmt.Println(strings.EqualFold("Straße", "STRASSE")) // simple folding only

    fold := cases.Fold() // full Unicode case folding
    fmt.Println(fold.String("Straße") == fold.String("STRASSE"))
}
// Output:
// false
// true

Building a comparison key you can store

localeCompare compares two strings, but it can't give you a value to store in a unique column. For that you need a key function. Unicode's recipe (called NFKC_Casefold) is: normalize with NFKC, case-fold, then normalize with NFKC again, because folding can produce characters that need composing.

// Exact: full Unicode case folding from golang.org/x/text.
func Key(s string) string {
    s = norm.NFKC.String(s)
    s = cases.Fold().String(s)
    return norm.NFKC.String(s)
}

// Key("Straße") == Key("STRASSE") → true

If the key must be exact, build it in one place on the server (Go's cases.Fold(), or a library implementing NFKC_Casefold) and never recompute it differently in the browser.

Base64: bytes through text-only pipes

Why it exists

Many channels were built for text and mangle anything else. Email (SMTPSMTPThe protocol programs use to hand an email to a mail server for delivery. An “SMTP server” is the mail server your code connects to.) was designed for 7-bit ASCII lines. JSON strings can't hold raw bytes. HTTP headersHTTP headerA Name: value line sent before the body of an HTTP request or response, carrying metadata — Content-Type: application/json, Authorization: Bearer …, Cache-Control: max-age=60. and URLsURLA web address: scheme, host, optional port, path and query — https://api.example.com:443/users?id=7. have characters that mean something special. A raw image byte like 0x00 or 0x0A (newline) would be cut off, rewritten or misread.

So we re-encode the bytes using characters that survive everywhere: A–Z, a–z, 0–9, + and /. That's 64 characters, and 64 = 2⁶, so each character carries exactly 6 bits.

The mechanism: 3 bytes → 4 characters

Bytes have 8 bits and Base64 characters carry 6. The smallest chunk where both line up is 24 bits: 3 bytes in, 4 characters out. For each group of 3 bytes:

1. Write the 3 bytes as 24 bits in a row.
2. Cut those 24 bits into four 6-bit pieces.
3. Each piece is a number from 0 to 63. Look it up in the alphabet A…Z a…z 0…9 + / (0 = A, 26 = a, 52 = 0, 62 = +, 63 = /).

When the input doesn't divide into groups of 3, the last group is filled up with zero bits, and the characters that carry no real data are written as =. That's PaddingPadding (Base64)The = signs at the end of Base64 text. Base64 works in groups of 3 bytes; when the input length isn't a multiple of 3, one or two = fill the last 4-character group so the decoder knows how many real bytes it held.: one = means the last group had 2 real bytes, == means it had 1. Step through Man, then Ma and M:

package main

import (
    "encoding/base64"
    "fmt"
)

func main() {
    for _, s := range []string{"Man", "Ma", "M"} {
        fmt.Println(s, "→", base64.StdEncoding.EncodeToString([]byte(s)))
    }

    raw := []byte{0xfb, 0xff, 0xbf} // bytes that hit the + and / characters
    fmt.Println(base64.StdEncoding.EncodeToString(raw))    // standard
    fmt.Println(base64.RawURLEncoding.EncodeToString(raw)) // URL-safe, no padding

    back, err := base64.StdEncoding.DecodeString("Y2Fmw6k=")
    fmt.Println(string(back), err)
}
// Output:
// Man → TWFu
// Ma → TWE=
// M → TQ==
// +/+/
// -_-_
// café <nil>

base64url: the web-safe variant

+ and / cause trouble in URLs (/ separates path segments, + can be read as a space in query strings) and = separates keys from values. Base64urlBase64urlBase64 with two characters swapped so the result is safe in URLs and headers (- instead of +, _ instead of /) and usually without = padding. JWT parts are base64url text: anyone can decode them; it is not encryption. swaps + → - and / → _, and usually drops the = padding, since the decoder can work out the length. JWTs, URL tokens and file names use it. A decoder for one alphabet will reject or misread the other, so check which one an API expects.

The browser trap: btoa() is not UTF-8

Browsers have btoa() (bytes to ASCII) and atob(), but they're from the 1990s and treat each UTF-16 code unit as one byte:

btoa("Man");            // "TWFu"
btoa("café");           // "Y2Fm6Q==", the Latin-1 bytes, not UTF-8's "Y2Fmw6k="
btoa("😀");             // throws InvalidCharacterError: code unit 0xD83D is bigger than a byte

// Correct: make UTF-8 bytes first, then turn each byte into one character for btoa
const bytes = new TextEncoder().encode("😀");
btoa(String.fromCharCode(...bytes)); // "8J+YgA=="

Newer runtimes add Uint8Array.prototype.toBase64() and Uint8Array.fromBase64(), which work on bytes directly. As of October 2026 they're in all major browsers (since Chrome 140, September 2025; Firefox and Safari had them earlier) and in Node 25+, but not in Node 22 or 24. In every current Node version, Buffer works; its "base64url" option needs Node 15.7 / 14.18 or later.

The 33% tax

Every 3 bytes become 4 characters, so Base64 output is 4/3 of the input, rounded up to a multiple of 4:

Compression (gzipCompression (gzip, Brotli)The server shrinks text files (HTML, CSS, JS) before sending them and the browser unpacks them, announced with the Content-Encoding header. Brotli files are usually 15–20% smaller than gzip for JavaScript.) wins some of it back on the wire, because each character only uses 6 of its 8 bits. But memory and parsing time pay the full price, and a Base64 string inside JSON must be decoded again after the JSON is parsed.

JSON vs binary formats

When a program sends an object over the network or saves it to disk, it first has to turn it into a flat sequence of bytes. That's called SerializationSerializationTurning an in-memory object into a flat string of bytes (e.g. JSON) so it can be sent somewhere, and parsing it back on the other side. It costs CPU time proportional to the data size.. JSONJSONJavaScript Object Notation: a text format for data — {"id": 7, "tags": ["a"]}. Readable by humans and every language; the default body format for web APIs. is the default choice for web APIs, and it's a good one. But it's worth knowing what it costs.

What JSON costs

package main

import (
    "encoding/json"
    "fmt"

    "github.com/vmihailenco/msgpack/v5" // go get github.com/vmihailenco/msgpack/v5
)

type User struct {
    ID     int      `json:"id" msgpack:"id"`
    Name   string   `json:"name" msgpack:"name"`
    Age    int      `json:"age" msgpack:"age"`
    Active bool     `json:"active" msgpack:"active"`
    Tags   []string `json:"tags" msgpack:"tags"`
    Avatar []byte   `json:"avatar" msgpack:"avatar"` // raw bytes
}

func main() {
    u := User{ID: 42, Name: "Zoë", Age: 31, Active: true,
        Tags: []string{"admin", "beta"}, Avatar: []byte{0xde, 0xad, 0xbe, 0xef, 0x00, 0x01}}

    j, _ := json.Marshal(u)
    fmt.Println(len(j), string(j))

    m, _ := msgpack.Marshal(u)
    fmt.Printf("%d % x\n", len(m), m)
}
// Output:
// 90 {"id":42,"name":"Zoë","age":31,"active":true,"tags":["admin","beta"],"avatar":"3q2+7wAB"}
// 60 86 a2 69 64 2a a4 6e 61 6d 65 a4 5a 6f c3 ab a3 61 67 65 1f a6 61 63 74 69 76 65 c3 a4 74 61 67 73 92 a5 61 64 6d 69 6e a4 62 65 74 61 a6 61 76 61 74 61 72 c4 06 de ad be ef 00 01

Reading the JSON line, you can count where the 90 bytes go:

• Every key, every time. "id", "name", "active"… are spelled out in every record. In a list of 10,000 users, the key names are sent 10,000 times.
• Numbers as text. 42 is two characters. A 64-bit ID like 9007199254740993 is 16 characters, and the parser has to turn the digits back into a number.
• No bytes type. Bytes must become Base64 (§09), 33% bigger, and both sides must agree that this particular string is "really" bytes.
• Escaping. Quotes, backslashes and control characters inside strings must be escaped (\", \\, \n), so a parser has to scan every character.

The big-integer trap

JSON itself allows numbers of any size, but JavaScript's JSON.parse turns every number into a regular JS number, which is a Floating-point numberFloating-point numberThe way computers store fractional numbers: a sign, a fraction and an exponent, like scientific notation in binary. Fast and wide-ranging, but most decimal fractions (like 0.1) can only be stored approximately. value. Integers above 2⁵³ − 1 aren't all stored exactly; the ones that can't be are silently rounded:

JSON.parse('{"id": 9007199254740993}').id; // 9007199254740992  ← off by one, no error

Go is only safe if you ask for it. A typed int64 field decodes exactly, but decoding into any or map[string]any gives a float64, which rounds exactly like JavaScript:

data := `{"id": 9007199254740993}`

var typed struct{ ID int64 `json:"id"` }
json.Unmarshal([]byte(data), &typed)
fmt.Println(typed.ID) // typed field: exact

var loose map[string]any
json.Unmarshal([]byte(data), &loose)
fmt.Println(int64(loose["id"].(float64))) // any → float64: rounded

dec := json.NewDecoder(strings.NewReader(data))
dec.UseNumber() // keep numbers as text (json.Number)
var safe map[string]any
dec.Decode(&safe)
fmt.Println(safe["id"]) // exact
// Output:
// 9007199254740993
// 9007199254740992
// 9007199254740993

loose["id"].(float64) is a type assertion: "I know this any holds a float64". Use a typed struct or UseNumber() for IDs, and remember that the JS client can still round them. So a Go server and a JS client can disagree about the same ID. That's why APIs like Twitter's send large IDs as strings ("id_str"). Why floats work this way is covered in Numbers crossing boundaries; for now, remember the safe integerSafe integer (JavaScript)A whole number JavaScript's normal number type can store exactly: up to 2^53 − 1 = 9,007,199,254,740,991. Bigger integers get rounded to a nearby value, silently. Use BigInt or a string for larger IDs. limit: 9,007,199,254,740,991.

Binary formats

Binary formats write values as compact bytes instead of text. Two families are worth knowing:

The comparison below encodes the same record for real, in your browser, with small encoders that produce byte-for-byte the same output as the @msgpack/msgpack and protobufjs libraries. Pick a format to see its bytes; key names and field tags are highlighted:

{"id":42,"name":"Zoë","age":31,"active":true,"tags":["admin","beta"]}

Two things to notice. Protobuf is smallest because it sends 08 2a ("field 1, value 42") instead of "id":42. And the more bytes a record holds (switch on the avatar), the worse JSON does, because of Base64. For how Protobuf lays out field numbers and varintsVarintA variable-length way to write an integer: small numbers take 1 byte, bigger ones take more. Each byte carries 7 bits of the number; its top bit says whether another byte follows., see gRPC: On the wire.

Schema evolution: changing the shape later

Your data's shape will change: fields get added, renamed and retired. Each format handles that differently:

The rules for Protobuf are in gRPC: Changing the contract safely.

When to pick what

Staff-level: BOMs, invalid bytes and truncation

Invalid UTF-8 and the replacement character

Not every byte sequence is valid UTF-8. A continuation byte with no lead byte, a lead byte followed by too few continuation bytes, or the Latin-1 byte 0xE9 sitting alone are all errors. Decoders have two choices:

• Replace: put the Replacement characterReplacement character (U+FFFD)The � symbol. Decoders put it in place of bytes that aren't valid in the expected encoding, instead of crashing. Seeing � means bytes were damaged or decoded with the wrong encoding somewhere earlier. � (U+FFFD) where the bad bytes were, and carry on. This is the default almost everywhere.
• Reject: stop with an error.

package main

import (
    "fmt"
    "strings"
    "unicode/utf8"
)

func main() {
    b := []byte{'c', 'a', 'f', 0xe9} // "café" saved as Latin-1, read as UTF-8
    s := string(b)                   // no error: a Go string can hold any bytes

    fmt.Println(utf8.ValidString(s))              // check before trusting
    fmt.Printf("%q\n", s)                         // the bad byte, visible
    fmt.Println(strings.ToValidUTF8(s, "\uFFFD")) // replace it with �
    for _, r := range s {
        fmt.Printf("%U ", r) // range yields U+FFFD for the bad byte
    }
    fmt.Println()
}
// Output:
// false
// "caf\xe9"
// caf�
// U+0063 U+0061 U+0066 U+FFFD

Notice that Go doesn't check anything when you convert bytes to a string. A Go string can hold any bytes; only range, the utf8 package and some encoders look at whether they're valid. encoding/json, for example, silently replaces invalid bytes with U+FFFD when it marshals a string.

JavaScript's version of the problem: lone surrogates

JS strings can contain a half surrogate pair (§06), for example after "😀".slice(0, 1). That isn't valid Unicode text. TextEncoder turns each lone half into U+FFFD (bytes ef bf bd), encodeURIComponent throws URIError, and JSON.stringify writes it as the escape "\ud83d" (since ES2019) so the output stays valid UTF-8. s.isWellFormed() and s.toWellFormed() (in Node 20+ and current browsers) check for and repair them.

The byte order mark

The Byte order markByte order mark (BOM)The invisible character U+FEFF placed at the very start of a file to announce its encoding (EF BB BF in UTF-8). Required by nothing in UTF-8, added by some Windows tools, and it breaks parsers that don't expect it. is the code point U+FEFF at the very start of a file. In UTF-16 it tells the reader the byte order (FE FF = big-endian, FF FE = little-endian). UTF-8 has no byte order, but some Windows tools (older Notepad, Excel's "CSV UTF-8" export, PowerShell 5's -Encoding UTF8) still write the three bytes EF BB BF at the start as a "this is UTF-8" marker.

Parsers that don't expect it see a strange first character:

$ printf '\xef\xbb\xbf{"a":1}' | od -An -c
 357 273 277   {   "   a   "   :   1   }

• Go: json.Unmarshal fails with invalid character 'ï' looking for beginning of value. The ï is 0xEF shown as Latin-1, mojibake inside an error message.
• JS: JSON.parse("\uFEFF{}") throws SyntaxError: Unexpected token. But new TextDecoder().decode(bytes) strips a leading BOM by default, so reading files that way avoids it.
• Shell scripts: a BOM before #!/bin/sh stops the kernel recognizing the shebang lineShebang lineThe first line of a script starting with #!, like #!/bin/sh. It tells the operating system which interpreter should run the file. It only works if #! are the very first two bytes., so the script won't run.
• CSV: the first column header becomes "\uFEFFid", so row["id"] is undefined and nobody can see why.

Truncating to a byte limit

Database columns, message queues, push notifications and HTTP headers often limit bytes. Cutting at byte N can land in the middle of a multi-byte character and produce invalid UTF-8, which the next system may reject, replace with �, or choke on.

// TruncateBytes returns the longest prefix of s that is at most max bytes
// and doesn't split a UTF-8 character.
func TruncateBytes(s string, max int) string {
    if len(s) <= max {
        return s
    }
    // Step back while the byte at the cut is a continuation byte (10xxxxxx).
    for max > 0 && !utf8.RuneStart(s[max]) {
        max--
    }
    return s[:max]
}

// TruncateBytes("héllo", 2) → "h"   (cutting at 2 would split é)

These keep code points whole. To keep graphemes whole too (not splitting 👨‍👩‍👧 into its parts), step through graphemes with Intl.Segmenter or uniseg and stop before the one that would cross the limit.

Staff-level: security, HTTP charsets and databases

Overlong encodings: the bug that made UTF-8 strict

§05's table says / (U+002F) is one byte, 2F. But the 2-byte pattern 110xxxxx 10xxxxxx could also hold the value 0x2F with leading zeros: C0 AF. That's an Overlong encodingOverlong encodingWriting a code point in UTF-8 with more bytes than needed, e.g. “/” as C0 AF instead of 2F. The UTF-8 standard forbids it because filters that check for “/” could miss it. Every modern decoder rejects it..

Around 2000, Microsoft's IIS web server checked URLs for ../ (to block Path traversalPath traversalEscaping a folder by putting ../ (or an absolute path, or a symlink) in a file name the server opens, e.g. ?name=../../etc/passwd. Also called directory traversal.) before decoding UTF-8, and a lenient decoder later turned %c0%af into /. Requests like /scripts/..%c0%af../winnt/system32/cmd.exe walked out of the web root (the folder the server is supposed to serve files from) and ran commands. Since then, the UTF-8 standard (RFC 3629) requires decoders to reject overlong forms, and Go and every browser do:

r, size := utf8.DecodeRune([]byte{0xc0, 0xaf})
fmt.Printf("%U %d\n", r, size) // U+FFFD 1  ← an error, never "/"

The general lesson outlives this bug: decode and normalize first, then validate. Checking the raw bytes and then transforming them lets the transformation create what the check was looking for.

Look-alike characters

Unicode has many homoglyphsHomoglyphA character that looks the same as a different one: Latin a (U+0061) and Cyrillic а (U+0430), or the digit 0 and letter O. Attackers use them to make fake domain names and usernames that look real.: Latin a (U+0061) and Cyrillic а (U+0430) are drawn identically in most fonts. So аpple.com with a Cyrillic first letter is a different domain that looks like the real one.

Domain names with non-ASCII letters travel through DNSDNSThe Domain Name System: the internet's phone book that turns a name like example.com into an IP address your computer can connect to. as PunycodePunycodeThe ASCII form of a domain name that contains non-ASCII letters, starting with xn--: münchen.de is xn--mnchen-3ya.de. DNS only carries ASCII, and browsers show the xn-- form when a name looks like a spoof.. Browsers display that ASCII form when a name mixes scripts suspiciously, which exposes the fake:

new URL("https://\u0430pple.com/").hostname; // "xn--pple-43d.com"  ← not apple.com
new URL("https://münchen.de/").hostname;      // "xn--mnchen-3ya.de" (a real German city)

The same trick works on usernames, package names, and display names in chat apps. Related problems:

• Invisible characters: the zero-width space (U+200B) and joiner make admin and ad + U+200B + min different strings that look identical on screen.
• Bidirectional controls: U+202E (right-to-left override) makes text display in reverse order. In 2021 "Trojan Source" (CVE-2021-42574; a CVE is the public ID number given to a known vulnerability) showed source code that looked harmless in review but compiled differently. GitHub and most editors now warn about these characters.

Usernames: a recipe

If two accounts can have names that look the same, someone will impersonate someone. A practical pipeline when a user picks a name:

1. Reject invalid UTF-8 (§11) and control characters.
2. Normalize with Compatibility normalizationCompatibility normalization (NFKC / NFKD)A stronger normalization that also folds look-alike “compatibility” characters into plain ones: fi → fi, ① → 1, full-width Ｈ → H. Useful for usernames and search; it loses formatting, so don't use it to store documents., so ｊｏｈｎ (full-width) and john become the same (§08).
3. Case-fold (§08) to build a comparison key; keep the user's original spelling for display. Do this in one server-side function (NFKC → fold → NFKC, as in §08's Key): JavaScript has no exact fold.
4. Restrict the allowed characters: for example, letters and digits from a single script plus _ and -. Mixing Latin and Cyrillic in one name is almost never legitimate.
5. Check uniqueness on the key, with a unique database index on that column.
6. For high-value names, also compare a "skeleton" from Unicode's confusables list (UTS #39), which maps 0/O and а/a to the same thing.

The IETF's PRECIS framework (RFC 8264/8265) and libraries implementing it (golang.org/x/text/secure/precis in Go) package steps 1–3.

HTTP: say the charset out loud

The charsetCharacter set (charset)An agreed table that says which number stands for which character. ASCII, Windows-1252 and Unicode are character sets. Software can only show text correctly if it uses the same table the writer used. is the name of the table the bytes were written with.

Bytes on the wire carry no encoding, so HTTP says it in the Content-Type header (see what an HTTP message looks like). The value is a media typeMedia type (Content-Type)A label saying what format a body is in, sent in the Content-Type header: application/json, text/html, application/problem+json. The client uses it to decide how to parse the body. plus an optional charset:

Content-Type: text/html; charset=utf-8
Content-Type: application/json

• For text/ types, always add charset=utf-8. Without it, a browser may guess, and old rules default some types to Windows-1252: exactly the mojibake from §03.
• JSON is defined as UTF-8 (RFC 8259) and application/json has no charset parameter. Adding one is harmless, and some old clients want it.
• In HTML, also put <meta charset="utf-8"> in the first 1024 bytes of the page, so a file opened from disk or served without the header still decodes right.
• Go's net/http sniffs the body and sets text/html; charset=utf-8 or text/plain; charset=utf-8 if you set no Content-Type. Set it yourself; sniffing is a guess.

Databases: encodings and collations

The encoding. In MySQLMySQLA widely used open-source relational SQL database (its default storage engine is InnoDB)., the type called utf8 (now utf8mb3) stores at most 3 bytes per character, so it can't hold 😀 or any 4-byte code point. Inserts fail with Incorrect string value: '\xF0\x9F\x98\x80' or, with strict SQL mode turned off (MySQL then warns instead of rejecting bad data), the text is cut off. Use utf8mb4utf8mb4 (MySQL)MySQL's name for real UTF-8. MySQL's older “utf8” (now utf8mb3) stores at most 3 bytes per character, so emoji and other 4-byte characters fail or get cut off. Always use utf8mb4. for the table and the connection (it's the default since MySQL 8.0). In PostgreSQLPostgreSQL (Postgres)A popular open-source relational SQL database, known for correctness and features. Used throughout this site's database examples., the encoding is chosen once per database at CREATE DATABASE; make it UTF8.

Lengths. VARCHAR(50) counts characters (code points) in both MySQL and Postgres, but index size limits are in bytes. That's why old MySQL schemas are full of VARCHAR(191): 191 × 4 bytes fits the 767-byte index limit of older InnoDB (MySQL's storage engine).

The collation. A CollationCollationThe rules a database uses to sort and compare text (is “a” before “B”? where do accents go?). It depends on language settings, and it affects whether LIKE 'abc%' can use a normal index. decides how text is compared and sorted. Byte order puts every uppercase letter before every lowercase one and accented letters last, which isn't what people expect:

package main

import (
    "fmt"
    "slices"

    "golang.org/x/text/collate"
    "golang.org/x/text/language"
)

func main() {
    words := []string{"zebra", "Éclair", "apple", "éclair"}

    slices.Sort(words) // compares bytes
    fmt.Println(words)

    collate.New(language.French).SortStrings(words) // language-aware
    fmt.Println(words)
}
// Output:
// [apple zebra Éclair éclair]
// [apple éclair Éclair zebra]

Collations also decide equality. MySQL's default utf8mb4_0900_ai_ci is accent-insensitive (ai) and case-insensitive (ci), so a unique index on username treats José and jose as duplicates. That's often what you want, but it surprises people who expect byte equality. In Postgres, the default collation is case-sensitive; use a lower-cased or folded column, or a nondeterministic ICU collation (Postgres 12+), for case-insensitive uniqueness.

Check yourself

Predict the answer before you click. Each explanation names the section that covers it.

s := "héllo"
fmt.Println(len(s))

console.log("👍🏽".length);

const a = "caf\u00e9";
const b = "cafe\u0301";
console.log(a === b, a.normalize("NFC") === b.normalize("NFC"));

const order = JSON.parse('{"id": 9007199254740993}');
console.log(order.id);

Cheat sheet