WRWEB ENGINEERING
REFORGING01 · 1/7
ALL LESSONS→WEEK 01→MONDAY

UNDERSTAND · COMPUTER + TERMINAL

Bytes, text, and encodings

Files are the first database30–60 MINUTESCORE + PRACTICAL
01

GROUND

Problem

What becomes confusing, fragile, or impossible without understanding bytes, text, and encodings? This lesson answers that through explanation, a worked example, two runnable exercises, and a reference solution. No teacher-supplied worksheet is required.

Before integrated development environments, the operating system, terminal, files, and text streams were the environment. Their small composable interfaces still underpin modern tooling.
02

LEARN

Concept explanation

Files contain bytes, not characters. Text appears only after software interprets those bytes with an encoding.

Unicode assigns code points to abstract characters. UTF-8 encodes each code point as one to four bytes, so character count and byte count are different measurements.

A reader using wrong encoding can produce mojibake or reject input even when stored bytes are unchanged. Always record encoding at system boundaries.

Byte

Eight bits. Files, sockets, and memory ultimately move byte sequences.

Code point

Unicode number identifying an abstract character, such as U+00E9 for é.

Encoding

Rule mapping code points to bytes and bytes back to code points.

Example bank

Compare normal, boundary, failure, and cross-layer cases. Predict each observation before revealing the explanation.

ASCII stays one byte

SETUPEncode A (U+0041) as UTF-8.

OBSERVEod prints 41; wc -c reports 1.

WHY IT MATTERSUTF-8 preserves ASCII byte values, which is why ASCII protocols remain readable inside UTF-8.

One character, two bytes

SETUPEncode composed é (U+00E9).

OBSERVEod prints c3 a9; wc -c reports 2 although the screen shows one grapheme.

WHY IT MATTERSVisible characters, Unicode code points, and storage bytes are different measurement layers.

Same appearance, different bytes

SETUPCompare composed é with e (U+0065) plus combining acute (U+0301).

OBSERVEThe forms can look identical; bytes are c3 a9 versus 65 cc 81.

WHY IT MATTERSVisual equality does not imply byte equality. Normalize only where the product contract requires it.

Emoji crosses UTF-16 too

SETUPCompare 🙂 in UTF-8 and JavaScript string length.

OBSERVEUTF-8 uses f0 9f 99 82 (4 bytes); JavaScript .length is 2 UTF-16 code units; [...text].length is 1 code point.

WHY IT MATTERSAlways name the unit being counted: bytes, code units, code points, or grapheme clusters.

Invalid input is still evidence

SETUPRead lone ff as UTF-8.

OBSERVEff can never start a valid UTF-8 sequence; tools reject it or emit a replacement character.

WHY IT MATTERSDecoder policy—reject, replace, or preserve bytes—is an explicit system-boundary decision.

03

SEE

Worked example

Encode three characters as UTF-8 and compare their bytes.

  1. ASCII A is U+0041 → hexadecimal 41 → 1 byte.
  2. é is U+00E9 → hexadecimal c3 a9 in UTF-8 → 2 bytes.
  3. 🙂 is U+1F642 → hexadecimal f0 9f 99 82 in UTF-8 → 4 bytes.
RESULT
Same visible-character count (one) can require 1, 2, or 4 bytes. JavaScript string length may differ again because it counts UTF-16 code units.
04

START HERE

Starter material

PREREQUISITESA POSIX shell with wc, od, chmod, printf, and file (macOS, Linux, or WSL). iconv is optional for the decoder check.

ONE-TIME SETUPmkdir bytes-encoding-lab && cd bytes-encoding-lab

Create lab.sh, paste this exact content, then run the command below.

#!/usr/bin/env sh
set -eu
printf 'A\né\n🙂\n' > samples.txt
printf '\377' > invalid.bin
printf 'sample bytes: ' && wc -c < samples.txt
od -An -tx1 -c samples.txt
printf 'invalid bytes: ' && od -An -tx1 invalid.bin
RUNchmod +x lab.sh && ./lab.sh

STOP / CLEANUPNo background process. Keep samples.txt, invalid.bin, and lab.sh as lesson evidence.

05

DO WITH GUIDANCE

Guided exercise

Measure characters as bytes

  1. Create samples.txt using supplied command.
  2. Predict byte count for A, é, 🙂, and three newline bytes before running wc.
  3. Run wc -c and od. Label which hex bytes belong to each character and newline.
  4. Replace é with decomposed e + combining acute accent; compare visible text and bytes.
Concrete guided solution
  1. Create lab.sh from Starter material, then run chmod +x lab.sh && ./lab.sh; this command creates samples.txt and invalid.bin.
  2. Predict 10 bytes: A=1, é=2, 🙂=4, and three newline bytes=3.
  3. Annotate od output as 41 | 0a | c3 a9 | 0a | f0 9f 99 82 | 0a.
  4. Create decomposed text with printf 'e\314\201\n' > decomposed.txt; verify 65 cc 81 0a.
  5. Compare with printf 'é\n' > composed.txt; explain why similar rendering does not mean equal bytes.
06

DO ALONE

Independent exercise

Create and diagnose invalid UTF-8

  1. Create invalid.bin containing lone byte ff.
  2. Inspect it with od -An -tx1 and file invalid.bin.
  3. Try decoding it with a UTF-8-aware tool available on your machine; capture rejection or replacement behavior.
  4. Repair file by replacing invalid byte with UTF-8 bytes for é, then prove decoder accepts it.
Concrete independent solution
  1. Create the invalid byte reproducibly with printf '\377' > invalid.bin.
  2. Run od -An -tx1 invalid.bin; the only byte must be ff.
  3. Run file invalid.bin and, where available, iconv -f UTF-8 -t UTF-8 invalid.bin; record rejection, exit status, and stderr.
  4. Repair with printf '\303\251' > invalid.bin; od must now show c3 a9.
  5. Rerun the decoder and prove it exits successfully and displays é. Keep before/after bytes in the notebook.
07

COMPARE

Expected result

  • wc -c reports 10: A=1, é=2, 🙂=4, plus three newline bytes.
  • Hex sequence is 41 0a c3 a9 0a f0 9f 99 82 0a.
  • invalid.bin contains ff, which cannot begin a valid UTF-8 code point.
08

PROVE

Acceptance checks

Lesson is complete only when every check is true. Each check is stored locally and travels with your JSON backup.

0/5 complete · saved on this device

09

UNSTICK

Hints

Reveal hints
  1. Newline is byte 0a and counts even though it is mostly invisible.
  2. Composed é is c3 a9; decomposed e plus U+0301 begins 65 cc 81.
  3. Byte ff is never valid in UTF-8.
10

VERIFY

Solution

Attempt both exercises before opening reference approach.

Reveal reference solution
  1. Run starter: it creates valid samples and one invalid byte without requiring an editor encoding setting.
  2. Annotate 41 | 0a | c3 a9 | 0a | f0 9f 99 82 | 0a, totaling 10 bytes.
  3. Create decomposed sample with printf 'e\314\201\n'; od shows 65 cc 81 0a while screen still resembles é.
  4. Replace invalid.bin with printf '\303\251' > invalid.bin; od shows c3 a9 and UTF-8 decoding succeeds.
11

PREDICT · INSPECT · BREAK · DEBUG · MEASURE

Interrogate reality

Prediction: write expected output, state transition, ordering, and failure evidence before running either exercise.

Inspection: Use the shell, process table, filesystem metadata, curl, and Git plumbing. Never trust a command you cannot observe.

Measurement: Count bytes, processes, descriptors, syscalls, round trips, and elapsed time before explaining performance.

INSPECT

Capture raw evidence before explaining.

BREAK

Change one assumption and force controlled failure.

DEBUG

Find cause with Ghostty · tmux · hx · shell · curl · Git before editing fix.

TOOL DRILL · keyboard only · record one retrievable command or shortcut
12

MASTERY + FRONTIER + BOUNDARY

Own the knowledge

TEACH

Explain bytes, text, and encodings at beginner, intermediate, and senior depth.

REBUILD

Recreate smallest useful example from blank file without notes or AI.

RETRIEVE

Schedule recall for day 1, 7, 30, and 90.

Creative frontier lab

Try first without opening the solutions. The constraints invite invention; the reference gives one concrete direction, never the only valid answer.

Build a count microscope

Create a tiny report comparing bytes, UTF-16 code units, code points, and grapheme clusters for five strings.

CONSTRAINTInclude A, é, decomposed é, 🙂, and a family emoji joined by zero-width joiners.

ORIGINAL IDEARender every measurement beside an escaped code-point sequence so the tool explains surprising counts instead of hiding them.

Reveal frontier solution
  1. Use TextEncoder().encode(text).length for UTF-8 bytes.
  2. Use text.length for UTF-16 code units and [...text].length for code points.
  3. Use Intl.Segmenter with granularity 'grapheme' for user-perceived clusters.
  4. Print code points with [...text].map(c => `U+${c.codePointAt(0).toString(16).toUpperCase()}`). The family emoji demonstrates many code points but one grapheme.
Invent a reversible byte postcard

Design a text format that lets another student reconstruct exact bytes even when the text is invalid UTF-8.

CONSTRAINTIt must be human-inspectable, diffable, and round-trip 00, newline, ff, and valid multibyte UTF-8.

ORIGINAL IDEAPair a hexadecimal payload with a deliberately non-authoritative preview; the preview can fail without corrupting truth.

Reveal frontier solution
  1. Store one lowercase two-digit hex byte per token, with an optional # preview comment.
  2. Encode with od -An -v -tx1 and normalize whitespace; decode with xxd -r -p where available.
  3. Round-trip 00 0a ff c3 a9, then compare original and reconstructed files with cmp.
  4. Document that the hex payload is authoritative and the preview is replaceable UI.
Normalization trap lab

Make a lookup that appears correct visually but fails because composed and decomposed keys differ.

CONSTRAINTFirst demonstrate the bug; then offer both a byte-preserving policy and a normalized-search policy.

ORIGINAL IDEAStore original bytes for fidelity while deriving a normalized comparison key—two representations with explicitly different jobs.

Reveal frontier solution
  1. Create a Map keyed by composed é; show lookup with decomposed e + combining acute returns undefined.
  2. Log both UTF-8 byte sequences to prove cause.
  3. For search, derive key with text.normalize('NFC') before insertion and lookup.
  4. Keep original input separately when byte fidelity matters; explain why unconditional normalization may be data loss.

Capability frontier

Push bytes, text, and encodings until another layer becomes justified. Record one robust technique, one contextual trade-off, and one labeled hack or historical curiosity.

CORE · PRACTICAL · CONTEXTUAL · HACK · FRAGILE · HISTORICAL · GOLF

Boundary

Shell composition becomes costly when state, error recovery, or data structure complexity dominates the pipeline.

If this vanished tomorrow…

Reproduce the useful behavior with files, text streams, process primitives, and a minimal compiled or interpreted program.

Why next layer is earned

Networking and HTTP are earned when local processes need to exchange representations across a boundary.