Cracking the Systems Programming Interview

Rust · Linux · Kubernetes · Distributed Systems

Interview preparation for senior cloud and systems engineering roles. Five parts, each one runnable on its own.

Part	Contents	Start with
rust-interview-lab/	31 Rust library modules and 22 standalone programs: graphs, trees, caches, concurrency primitives built from scratch, a hand-rolled async runtime, an epoll server, TCP and HTTP servers. Every module carries its own unit tests.	cd rust-interview-lab && cargo test
rust-interview-lab/benchmarking_examples/	Two caches and four linked lists, the programs that measure them, captured output in BENCHMARKS.md, and an 8-page report as a PDF.	cd rust-interview-lab && cargo run --release --bin benchmark
operator/	Kubebuilder v4 layout on controller-runtime. The GpuWorkload API type and the reconciler are hand-written; the CRD and deepcopy code is generated by controller-gen.	cd operator && go test ./...
docs/	14 study guides: Linux internals, Kubernetes, controller-runtime, CLI/SDK/REST design, concurrency and async, testing and CI, behavioural rounds, and a three-part Rust guide.	read, or rg them
cracking-the-systems-programming-interview/	The book: 19 chapters in six parts (tools, whiteboard algorithms, memory and the machine, concurrency, the network, interview day) and four appendices. Every lab program is quoted in full with {{#include}}, versions of the same problem are shown in the order they were written, and each chapter ends with the questions an interviewer will ask.	make serve, make pdf
gpu-accelerated-kubernetes/	A book on running GPU workloads on Kubernetes, with every program in both Go and Rust: an informer model built from scratch, a GPU job controller on controller-runtime and kube-rs, what one GPU unit contains and how GPUs are shared (time-slicing, MPS, MIG, vGPU, DRA), and a kind cluster whose pods use a real GPU. Read it online.	cd gpu-accelerated-kubernetes && mdbook serve
democratizing-linux/	Democratizing Linux: A Pipe Dream. A book that rebuilds the kernel's mechanisms in user space, with every program in both Go and Rust: pipes, file descriptors, processes, a shell, signals, CPU scheduling, virtual memory, the page cache, epoll, TCP sockets, namespaces, control groups and a container. 13 chapters with kernel structure definitions, measurements from a real machine, diagrams and animations. Read it online.	cd democratizing-linux && mdbook serve
distributed-systems-from-the-ground-up/	Distributed Systems from the Ground Up. A system design book in 28 chapters. Part 1 designs a replicated, sharded key-value store and builds it in Rust: a write-ahead log, replication with a leader and a majority, Raft leader election, a router and group commit, each measured and tested against the failures it claims to survive. Part 2 is 22 system designs with no printed code: rate limiter, unique ids, cache, lock, message queue, job scheduler, notifications, URL shortener, news feed, object store, video streaming, chat, payments, ticket booking, ride hailing, autocomplete, web crawler, metrics, file sync, collaborative editing, stock exchange and click aggregation. Every algorithm has a picture that advances step by step, and every design has a runnable simulation. Read it online.	cd distributed-systems-from-the-ground-up && mdbook serve
systems_design/	A GPU and cloud systems design reference with Mermaid diagrams: the GPU execution model, cluster scheduling and sharing, the control plane and API design, failure handling, inference serving, model distribution, metering, and diagnosis.	open the index
scripts/	Kind cluster setup, plus a live end-to-end check of the operator.	scripts/verify-kind.sh

Quick start

Rust lab

cd rust-interview-lab
cargo test                       # library and program unit tests
cargo run --bin count_islands    # or ll, shadowing, mutex_poisoning, ...
cargo run --bin os_paging        # or os_scheduler, net_ipv4, cs_fib, ...
cargo run --release --bin syscall_overhead

Benchmarks and the report

cd rust-interview-lab
cargo run --release --bin benchmark            # cache, then all four list variants
cargo run --release --bin benchmark cache      # index arena against Rc<RefCell> list
cargo run --release --bin benchmark list       # the four linked-list variants
cargo run --release --bin list_drop            # 5,000,000 nodes, freed iteratively
python3 benchmarking_examples/build_report_pdf.py   # rebuild the report

Recorded output is in BENCHMARKS.md, and the report is at benchmarking-report.pdf.

Go operator

cd operator
go test ./...

Live Kubernetes check

scripts/verify-kind.sh creates a Kind cluster, installs the CRD, starts the manager, applies a sample GpuWorkload, and waits for the Deployment to roll out. The rollout is the assertion: if reconciliation is broken, the wait times out.

scripts/verify-kind.sh
scripts/cleanup-kind.sh          # optional

The Kind sample sets gpuCount: 0, because Kind nodes have no GPU and no device plugin, so the pod would stay Pending forever. On a real GPU cluster set gpuCount: 1 and runtimeClassName: nvidia, as in operator/config/samples/gpucloud_v1_gpuworkload_gpu.yaml.

Book

cd cracking-the-systems-programming-interview
make serve                       # browser edition at http://localhost:3000
make pdf                         # -> build/cracking-the-systems-programming-interview.pdf
make art figures animations      # redraw the plates, the diagrams, and the animations
make lint                        # check the figures for colliding labels, and the prose

The print edition is at cracking-the-systems-programming-interview/build/cracking-the-systems-programming-interview.pdf.

Animations

Each animation follows one run of the code in the listing beside it. The drawing moves: values travel between cells, pointers slide, threads fall asleep and wake. A caption under the drawing names the step, and the line of code that runs is highlighted in a panel below it. Code lines appear only when the run reaches them. Most animations end on a failing case, such as a missing wake or a wrong insert order, drawn in rust.

The HTML book plays each animation as a video with pause, three speeds, a button for each step, and a mode that stops after each step. The GIF is the fallback.

python3 tools/animations.py                  # every animation
python3 tools/animations.py two-sum          # one of them
python3 tools/preview.py out.png two-sum:12  # a still of one moment, for checking layout
python3 tools/anim_markup.py                 # point the chapters at the new videos

tools/ANIMATIONS.md describes the engine, the drawing rules, and the pacing rules.

Repository layout

.
├── docs/                         # Interview docs and study guides
│   ├── 01-role-map-and-study-plan.md
│   ├── 02-linux-concepts.md
│   ├── 03-kubernetes-concepts.md
│   ├── 04-kubernetes-operator.md
│   ├── 05-kubernetes-operator-lab.md
│   ├── 06-system-design-cloud-gpu.md
│   ├── 07-cli-sdk-rest-concurrency.md
│   ├── 08-testing-cicd-observability.md
│   ├── 09-behavioral-interview.md
│   ├── 10-gpu-operator-kubebuilder-deep-dive.md
│   ├── 11-rust-guide-part1-collections-adt-smart-pointers.md
│   ├── 12-rust-guide-part2-errors-concurrency-sdk.md
│   ├── 13-rust-guide-part3-gotchas-linkedlist-interview.md
│   └── 14-concurrency-async-networking.md
│
├── rust-interview-lab/           # LeetCode-style DSA + systems, concurrency and async labs
│   ├── Cargo.toml
│   ├── BENCHMARKS.md             # captured benchmark output
│   ├── src/lib.rs                # crate root, re-exports, and the benchmarking modules
│   ├── src/problems/             # 31 library modules, each with its own #[cfg(test)] block
│   ├── src/bin/                  # 22 standalone programs (os_paging, net_ipv4, http_server, ...)
│   └── benchmarking_examples/    # the measured structures and the programs that measure them
│       ├── benchmark.rs          # the single benchmark main
│       ├── list_box.rs           # demos of the four list variants
│       ├── list_enum.rs
│       ├── list_drop.rs
│       ├── lists/                # boxed, enum_node, boxed_drop, enum_drop
│       ├── cache/                # arena (Vec + indices), rc_list (Rc<RefCell>)
│       ├── benchmarking-report.md # report source
│       ├── report.css             # print stylesheet
│       ├── build_report_pdf.py    # Markdown + WeasyPrint -> PDF
│       └── benchmarking-report.pdf
│
├── cracking-the-systems-programming-interview/   # the book: mdBook + WeasyPrint
│   ├── src/                      # chapters, art/ plates, figures/ diagrams and GIFs
│   ├── diagrams/                 # Graphviz sources for the figures
│   ├── tools/                    # art.py, figures.py, anim*.py, build_pdf.py, print.css
│   └── build/                    # generated PDF
│
├── democratizing-linux/          # the Linux book: mdBook, programs in Go and Rust
│   ├── src/                      # chapters, art/ plates, figures/ diagrams and animations
│   ├── code/go/, code/rust/      # 25 programs in each language
│   ├── code/output/              # the output both versions must print
│   ├── tools/                    # chapter sources, figure and art scripts, checks
│   └── Makefile                  # make check, make html
│
├── operator/                     # Idiomatic Go Kubernetes operator
│   ├── api/v1/                   # GpuWorkload CRD types
│   ├── cmd/main.go               # manager entrypoint
│   ├── internal/controller/      # reconciler + unit tests
│   ├── config/                   # CRDs, RBAC, samples
│   ├── PROJECT                   # Kubebuilder-compatible layout
│   └── Makefile
│
└── scripts/
    ├── setup-tools.sh            # install Go/kind/kubectl/controller-gen
    ├── verify-kind.sh            # full live verification on Kind
    └── cleanup-kind.sh           # delete the Kind cluster

What the Rust lab covers

Topic	File	Idea it tests
Two sum	two_sum.rs	Precompute the complement instead of searching for it
Valid parentheses	valid_parentheses.rs	Unmatched openers need order, not a count, so a stack
Min stack	min_stack.rs	Amortise the minimum over pushes instead of rescanning
Sliding window	sliding_window.rs	The window start only moves forward, so the scan is linear
Merge intervals	merge_intervals.rs	Sort by start, then extend or append against the last interval
Top k frequent	top_k_frequent.rs	A bounded heap beats sorting the whole input
Binary tree	binary_tree.rs	enum Tree makes the empty case a variant, not a null
Binary search tree	bst_clean.rs	The ordering invariant decides which subtree to recurse into
Binary search	binary_search.rs	A half-open range and a sorted-half test
Trie	trie.rs	Share prefixes as edges, so lookup costs the key length
Topological sort	graph_topology.rs	Kahn's algorithm, and a short output means a cycle
Coin change	dp.rs	Option<i32> instead of a -1 sentinel
Subsets	backtracking.rs	One push and one pop around each recursive call
Reverse linked list	linked_list.rs	Three bindings and no allocation
LRU cache	lru_cache.rs	HashMap for lookup, VecDeque for recency
Rate limiter	rate_limiter.rs	Refill from elapsed time, so the bucket needs no timer
Worker pool	worker_pool.rs	Bounded workers, results collected by index to keep order
Workload state machine	state_machine.rs	Legal transitions as one exhaustive match
Typed API errors	state_machine.rs	is_retryable() without parsing a message

What the systems lab adds

The second half of the lab moves from puzzles to the machinery the role is about: concurrency primitives written out by hand, async without a runtime, and the network stack from raw TCP to HTTP request framing.

Topic	File	Idea it tests
Lifetime of a spawned thread	threads.rs	thread::scope borrows instead of requiring 'static
Atomic state	threads.rs	Ordering is part of the contract, not a detail
Spin lock from scratch	spin_lock.rs	compare_exchange_weak, acquire/release, RAII guard
Counting semaphore	semaphore.rs	Mutex plus Condvar, and a guard that cannot leak a permit
Bounded MPMC queue	bounded_queue.rs	Backpressure, and a close() that wakes every waiter
Lock-free SPSC ring	ring_buffer.rs	A release store publishes, an acquire load consumes
Sharded cache	sharded_cache.rs	Per-shard locks trade memory for less contention
Retry policy	retry.rs	Exponential backoff with full jitter, and a Retry-After floor
Consistent hashing	consistent_hash.rs	Virtual nodes keep remapping small when a node leaves
Bump allocator	bump_allocator.rs	Alignment arithmetic, and why reset costs nothing
HTTP/1.1 parsing	http_request.rs	Content-Length against chunked, and the smuggling checks
Async from first principles	async_mini.rs	Future, Waker, Pin, and a task queue that runs them
Threading and async in practice	docs/14	The same material as a study guide, with the failure modes
Server implementations	src/bin/tcp_echo_server.rs, http_server.rs, epoll_echo.rs	Thread per connection, keep-alive, and readiness-based I/O

Benchmarks

benchmarking_examples/ holds the structures and the programs that measure them. Two caches and four linked lists are compared on the same workload, and every number in the report was produced by these programs.

Question	Where
Index arena against Rc<RefCell> nodes for an LRU	cache/arena.rs, cache/rc_list.rs
Four ways to store and free a singly linked list	lists/boxed.rs, enum_node.rs, boxed_drop.rs, enum_drop.rs
Where the recursive free stops working	about 265,000 nodes on an 8 MiB stack
How the measurements are taken	warm-up, then the fastest of three passes
Captured output and the written report	BENCHMARKS.md, benchmarking-report.pdf

Rust in production

Same skill set, these employers:

• NVIDIA, cuda-oxide (Rust → PTX GPU kernels), cutile-rs (tile-based kernel DSL), nvrc (microVM init); Rust Foundation Platinum member
• Cloudflare, Pingora, Workers runtime, Wrangler
• AWS, Firecracker, Bottlerocket, Lambda
• Microsoft, Windows kernel components, Azure, windows-rs
• Google, Android low-level services, Fuchsia
• Meta, source control backend, developer tooling
• Discord, Read States
• Dropbox, sync engine
• Shopify, checkout and search
• Figma, multiplayer server-side rendering
• 1Password, core password-manager engine
• Sentry, Symbolicator, Relay
• Fly.io, edge proxy
• Deno / Astral / Tauri, runtimes and tooling

Coverage map

Interview topic	Where it is
CLI/SDK tools in Python/Go/Rust	docs/07, docs/11, docs/12, docs/13, rust-interview-lab
RESTful web services	docs/07, src/bin/http_server.rs, src/problems/http_request.rs
Concurrency and synchronisation	src/problems/{threads,spin_lock,semaphore,bounded_queue}.rs, docs/14
Async Rust and executors	src/problems/async_mini.rs, src/bin/async_demo.rs, docs/14
Lock-free and sharded data structures	src/problems/{ring_buffer,sharded_cache}.rs
Operating system concepts	src/bin/{os_paging,os_scheduler}.rs, docs/02
TCP/IP and sockets	src/bin/{tcp_echo_server,epoll_echo,net_ipv4,net_window}.rs, docs/14
Performance measurement	benchmarking_examples/, BENCHMARKS.md
Linux and containers	docs/02
Kubernetes	docs/03
Operators and controller-runtime	operator/, docs/04, docs/05, docs/10
Unit and integration testing	Rust tests, Go tests, docs/08
CI/CD	docs/08, .github/workflows/ci.yaml
GPU ecosystem	docs/06, docs/10, operator samples
Behavioural rounds	docs/09

License

MIT, see LICENSE.