Session 5: Containers — Docker Deep Dive

Why Containers?

Imagine this: a developer writes an app on her MacBook with Node 21 and pnpm. She hands it off. The QA engineer runs it on Ubuntu — but his Node is version 18 and he uses npm. It crashes. The Ops engineer deploys to a production server running CentOS — different glibc, different OpenSSL, different everything. It crashes differently. All three are running “the same code.” None of them can reproduce each other’s bugs.

That’s the “works on my machine” problem. Containers solve it by packaging the app plus its entire runtime — Node version, system libraries, environment variables, file layout — into a single artifact that runs identically anywhere a container engine exists.

Containers vs VMs

You may have used VMs (VirtualBox, VMware) — those are heavyweight: each VM ships its own Linux kernel and boots like a full OS. Containers don’t. They share the host’s kernel and just isolate the userland (filesystem, processes, network).

┌─────────────────────────────┐        ┌─────────────────────────────┐
│        VIRTUAL MACHINES     │        │         CONTAINERS          │
├─────────────────────────────┤        ├─────────────────────────────┤
│  App A    │  App B          │        │  App A  │  App B  │  App C  │
│  Bins/    │  Bins/Libs      │        │  Bins/  │  Bins/  │  Bins/  │
│  Libs     │                 │        │  Libs   │  Libs   │  Libs   │
├───────────┼─────────────────┤        ├─────────┴─────────┴─────────┤
│ Guest OS  │  Guest OS       │        │       Docker Engine         │
│ (kernel)  │  (kernel)       │        ├─────────────────────────────┤
├───────────┴─────────────────┤        │     Host OS (1 kernel)      │
│        Hypervisor           │        ├─────────────────────────────┤
├─────────────────────────────┤        │         Hardware            │
│         Host OS             │        └─────────────────────────────┘
├─────────────────────────────┤
│         Hardware            │
└─────────────────────────────┘
   Heavy (GB), slow boot                  Lightweight (MB), fast start

The shared-kernel insight is everything. It’s why containers boot in milliseconds, why an image is 100 MB instead of 10 GB, and why you can run 50 containers on a laptop that would choke under 5 VMs.

Trade-off: because containers share the host kernel, you can’t run a Windows container on a Linux host (different kernel APIs). Different Linux distros are fine — they all speak the same kernel.

Images vs Containers

This trips everyone up at first. Memorize this distinction:

One image can produce many containers. The image of chugli-ai:1.0.10 is one fixed artifact; you can docker run it 50 times and get 50 separate containers, each with its own state.

Layers — the magic that makes everything fast

An image is a stack of read-only layers. Each line in a Dockerfile that changes the filesystem creates a new layer. When you docker run, Docker adds a thin writable layer on top:

┌─────────────────────────────┐  ← writable layer (this container's writes)
├─────────────────────────────┤  ← image layer: CMD ["node", "server.js"]
├─────────────────────────────┤  ← image layer: COPY . .
├─────────────────────────────┤  ← image layer: RUN pnpm install
├─────────────────────────────┤  ← image layer: COPY package.json
└─────────────────────────────┘  ← image layer: FROM node:21.7.2-alpine

Three consequences you should internalize:

1. Layers are cached. If package.json hasn’t changed, pnpm install is reused from cache — that’s why your second build is fast.
2. Layer order matters. Put rarely-changing things (deps) before frequently-changing things (source code). Get this wrong and your cache is useless.
3. Layers are shared between images. If two images both start with node:21.7.2-alpine, that 50 MB base layer is stored once on disk.

How Docker Actually Works (Under the Hood)

Up to this point I’ve been calling containers “isolated” and saying they “share the kernel.” Time to be specific. There is no Docker magic — just three Linux kernel features stitched together by a stack of programs. If you understand what’s actually happening, you’ll debug container weirdness 10× faster.

A container is just a Linux process

Run this on a Linux host (or inside Docker Desktop’s VM):

docker run -d --name demo nginx
ps -ef | grep nginx

You’ll see the nginx process on the host’s process table. Same top, same kill -9, same PID space (from the host’s view). There is no separate “container kernel” or “container OS.” It is an ordinary Linux process that the kernel has been told to pretend is alone in the universe.

That pretending is done with three kernel features.

Primitive 1 — Namespaces (what the process can see)

A Linux namespace isolates one kind of system resource. There are seven types, and a “container” is just a process with all seven namespaces set:

You can poke at namespaces by hand, without Docker:

sudo unshare --pid --uts --mount --fork bash
hostname new-hostname     # changes only in this shell
ps -ef                    # sees only its own tree

That’s what “containerization” fundamentally is. Docker just sets all seven for you and adds tooling around it.

Primitive 2 — cgroups (how much the process can use)

cgroups (control groups) put hard limits on CPU, memory, I/O, and network. They’re how docker run --memory=512m --cpus=1 actually enforces limits.

docker run -d --memory=128m --cpus=0.5 --name limited nginx

# Find the cgroup file (cgroup v2 layout)
CID=$(docker inspect -f '{{.Id}}' limited)
cat /sys/fs/cgroup/system.slice/docker-${CID}.scope/memory.max
# → 134217728   (128 MB in bytes)

If the container tries to use more than 128 MB, the kernel kills it with an OOM signal. Docker doesn’t enforce this — the kernel does, because Docker wrote the limit into a cgroup file.

Primitive 3 — Union/overlay filesystems (how layers stack)

Image layers are made real by the overlay2 filesystem driver. Multiple read-only layers are stacked, and a writable layer sits on top. When the container writes a file, it’s copied up to the writable layer (copy-on-write). Other containers using the same base layers don’t see the change.

docker info | grep "Storage Driver"
# → Storage Driver: overlay2

ls /var/lib/docker/overlay2/
# → directories named by layer hash

This is why pulling a second image that shares a base layer is “instant” — Docker just references the existing layer on disk.

The Docker stack — CLI down to the kernel

What looks like docker run is actually a chain of programs:

┌──────────────┐
│  docker CLI  │   → speaks HTTP to the daemon
└──────┬───────┘
       │ /var/run/docker.sock
┌──────▼───────┐
│   dockerd    │   → manages images, networks, volumes, build cache
└──────┬───────┘
       │ gRPC
┌──────▼───────┐
│  containerd  │   → manages container lifecycle (CRI-compatible)
└──────┬───────┘
       │
┌──────▼───────┐
│  containerd  │   → one shim process per container
│    -shim     │      keeps stdio open, decouples from containerd
└──────┬───────┘
       │ exec
┌──────▼───────┐
│     runc     │   → low-level container creator (OCI Runtime Spec)
└──────┬───────┘
       │ syscalls
┌──────▼───────┐
│ Linux kernel │   → clone(), unshare(), pivot_root(), exec()
└──────────────┘

What each layer does:

This separation is why Kubernetes dropped Docker as a runtime in 2020. K8s calls containerd directly and skips dockerd entirely. The containers themselves are unchanged — Docker the daemon was just a management layer on top.

What actually happens when you docker run

1. The CLI parses your command, sends POST /containers/create to dockerd.
2. dockerd validates, makes sure the image exists locally (pulls if not), and assembles an OCI bundle (config + rootfs).
3. dockerd asks containerd to create a container from that bundle.
4. containerd forks a shim process.
5. The shim invokes runc create, then runc start.
6. runc calls clone() with the namespace flags: CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWIPC | CLONE_NEWUSER | CLONE_NEWCGROUP.
7. In the new namespaces, runc sets cgroup limits, applies seccomp + AppArmor profiles, drops Linux capabilities, pivot_roots into the container’s filesystem, and finally execs your process.
8. Your process is now running. From here, the kernel enforces all the isolation.

All of that, every time, in roughly 50 milliseconds.

OCI — the standards that hold it together

Two OCI (Open Container Initiative) specs make the whole ecosystem interoperable:

• OCI Runtime Spec — what runc implements. Defines how a container is configured and launched.
• OCI Image Spec — defines what an image looks like on disk: a manifest, a config blob, and a stack of layer tarballs, all content-addressed by SHA-256.

Because of these specs, you can swap parts of the stack:

The image you build with Docker runs anywhere that speaks the OCI spec.

Docker Desktop is a polite lie

Your Mac or Windows laptop’s kernel cannot run Linux containers — different syscall ABIs. So Docker Desktop runs a tiny Linux VM in the background (HyperKit on Mac, WSL2 on Windows). The docker CLI on your laptop talks over a socket to dockerd running inside that VM. The containers run on the VM’s kernel, not your laptop’s.

Alternatives — OrbStack, Colima, Rancher Desktop — do the same trick with different VM technology. The shared-kernel rule still holds; it’s the VM’s kernel being shared, not your host’s.

# On a Mac, prove there's a Linux kernel underneath:
docker run --rm alpine uname -a
# → Linux 6.6.31-linuxkit ... x86_64 Linux
# Not Darwin. Not your Mac's kernel.

This also explains why Docker Desktop uses RAM and CPU even when no containers are running — the VM is always on.

Why this all matters in practice

Knowing the implementation pays off in five real situations:

You don’t need to memorize syscall names. You do need to know that containers are processes, isolation is namespaces + cgroups, the rootfs is overlay layers, and there’s a daemon stack between your CLI and the kernel. That model will save you hours of debugging.

The 7 Dockerfile Instructions You’ll Actually Use

There are ~17 Dockerfile instructions. You’ll use these seven 95% of the time:

Two more you’ll see often:

| EXPOSE | Document which port the container listens on (informational, not enforced) | EXPOSE 3000 | | ARG | Build-time variable (not available at runtime) | ARG NODE_VERSION=21 |

RUN vs CMD — this is the most common confusion. RUN happens at build time and bakes its result into a layer. CMD runs when the container starts. RUN apt-get install installs a package into the image; CMD node server.js starts the server when the container launches.

A Real Dockerfile — chugli.ai Walkthrough

Open chugli.ai/Dockerfile. It’s 74 lines, three stages, and ships our production Next.js app. We’re going to read every meaningful line.

The full file (annotated)

# syntax=docker/dockerfile:1.7                                       # ① BuildKit features
############################
# Stage 1: Build
############################
FROM --platform=linux/amd64 node:21.7.2-alpine AS builder           # ② pinned, named stage
WORKDIR /app

RUN apk add --no-cache python3 make g++ libc6-compat \              # ③ native build deps
  && corepack enable \
  && corepack prepare [email protected] --activate                        # ④ pinned pnpm

COPY package.json pnpm-lock.yaml ./                                 # ⑤ deps first → cache
RUN pnpm install                                                    # ⑥ install (incl. dev)

COPY . .                                                            # ⑦ source last

ARG NEXT_PUBLIC_APPWRITE_URL                                        # ⑧ build-time args
ARG NEXT_PUBLIC_APPWRITE_PROJECT_ID
# ...more ARGs...

ENV NEXT_TELEMETRY_DISABLED=1
RUN pnpm run build && rm -rf .next/cache                            # ⑨ build + trim

############################
# Stage 3: Runtime
############################
FROM --platform=linux/amd64 node:21.7.2-alpine AS runner            # ⑩ fresh, slim stage
WORKDIR /app

ENV NODE_ENV=production \
    NEXT_TELEMETRY_DISABLED=1

RUN addgroup -S nodejs && adduser -S nextjs -G nodejs               # ⑪ non-root user

COPY --from=builder /app/.next/standalone ./                        # ⑫ copy only output
COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public

RUN mkdir -p .next/cache && chown -R nextjs:nodejs /app

USER nextjs                                                         # ⑬ switch to non-root
EXPOSE 3000
CMD ["node", "server.js"]

Line-by-line

① # syntax=... — enables BuildKit Dockerfile features (cache mounts, secret mounts). Always put this at the very top.

② FROM --platform=linux/amd64 node:21.7.2-alpine AS builder

• node:21.7.2-alpine — pinned to a specific patch version, on Alpine (~5 MB base vs ~200 MB for Debian). Never use node:latest.
• --platform=linux/amd64 — forces x86-64 even when building on an M-series Mac. Otherwise the resulting image would be ARM and wouldn’t run on our DigitalOcean x86 nodes.
• AS builder — names this stage so later stages can copy from it.

③ Native build deps — Next.js builds need a C toolchain (some npm packages compile native code). --no-cache skips Alpine’s package index cache to keep the layer small.

④ corepack — Node’s official way to pin package managers. We’re saying “use pnpm 8.15.2 exactly” — no surprise upgrades.

⑤–⑦ The cache trick. Copy package.json and the lockfile first, run install, then copy source. Why? Because source changes constantly but deps don’t. If only source changes, the pnpm install layer is reused from cache and the build is 10× faster.

⑧ ARG NEXT_PUBLIC_* — Next.js bakes NEXT_PUBLIC_* env vars into the JS bundle at build time. We accept them as build args so they end up in the static output. Server-only secrets are never passed as ARG — they’d be visible in image layers.

⑨ rm -rf .next/cache — trims dev-only cache from the build output. Saves ~50 MB.

⑩ Fresh FROM for the runtime stage. This is the heart of multi-stage builds: the runtime stage starts clean, with none of the build tools, dev deps, or source code from stage 1. We only copy built artifacts across.

⑪ Non-root user. Two lines (addgroup, adduser) to create a dedicated nextjs user. If a container ever gets compromised, the attacker is nextjs, not root — that’s a huge security delta.

⑫ COPY --from=builder — copy specific paths from the build stage. Only the standalone Next.js output, static assets, and public files. No source code, no node_modules, no pnpm.

⑬ USER nextjs — switch users for the runtime. Everything from this line onward runs as nextjs. The CMD will run as nextjs when the container starts.

What the multi-stage trick saves us

Same code, 10× smaller image. That’s bandwidth savings, faster pulls in CI, smaller attack surface.

Best Practices (pulled from what we just read + the gaps)

What chugli.ai’s Dockerfile does well

Gaps in chugli.ai’s Dockerfile (your homework)

The .dockerignore that ships with chugli.ai

node_modules
.next
.git
.env*
__tests__
coverage
*.log

Why .dockerignore matters: every file in your build context is sent to the Docker daemon, hashed for cache lookups, and risks being COPYed into the image. A missing .dockerignore means node_modules (300 MB) gets uploaded, gets COPYed in, and you waste minutes per build.

Universal rules

1. Pin everything. Base image, language tools, build args. latest is a time bomb.
2. One process per container. No supervisord running Node and nginx — split them.
3. Read-only filesystem where possible. docker run --read-only if your app doesn’t write to disk.
4. Never bake secrets into the image. Use --env-file, mount secrets, or use BuildKit secret mounts.
5. Add a HEALTHCHECK. Without it, your container can be “running” but completely broken and no one knows.
6. Minimize layer count for RUN. Chain commands with &&, clean up apt caches in the same RUN so they don’t persist in the layer.

Build → Run → Push — The Daily Workflow

These three commands are 80% of what you’ll type day-to-day.

1. Build

docker build -t chugli-ai:1.0.10 \
  --build-arg NEXT_PUBLIC_APPWRITE_URL="$NEXT_PUBLIC_APPWRITE_URL" \
  --platform linux/amd64 \
  .

• -t name:tag — name and tag the image
• --build-arg — pass build-time variables
• --platform — force target architecture (essential on Apple Silicon)
• . — the build context (current directory). Everything in here gets sent to the Docker daemon.

2. Run locally

docker run -p 3000:3000 --env-file .env.local chugli-ai:1.0.10

• -p host:container — port forwarding (host port 3000 → container port 3000)
• --env-file — read env vars from a file (never put secrets on the CLI)
• -d — detached (background)
• --name chugli-ai — friendly container name (so you can docker logs chugli-ai instead of docker logs abc123def)

3. Tag + push to registry

docker tag chugli-ai:1.0.10 \
  registry.digitalocean.com/loadsnap-registry/chugli-ai:1.0.10

docker push registry.digitalocean.com/loadsnap-registry/chugli-ai:1.0.10

The full pipeline is automated in chugli.ai/docker-build.sh — it builds, tags both :VERSION and :latest, pushes both with retries, restarts the dev K8s deployment, and creates a git tag. Read it later — it’s a tour of the whole release flow.

Other commands you’ll use weekly

Docker Compose — Local Dev Only

Compose lets you describe a multi-container setup (app + database + cache) in one YAML file. Useful for local development. Not used in production — that’s Kubernetes’s job.

Minimal example

# docker-compose.yml
services:
  app:
    build: .
    ports:
      - "3000:3000"
    env_file: .env.local
    depends_on:
      - mongo
      - redis

  mongo:
    image: mongo:7
    volumes:
      - mongo-data:/data/db
    ports:
      - "27017:27017"

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

volumes:
  mongo-data:

Then:

docker compose up        # start all services
docker compose down      # stop and remove
docker compose logs -f app

Why this is local-only: Compose has no node scheduling, no rolling updates, no auto-healing, no service mesh. K8s does. Use Compose to spin up dependencies for local dev; deploy with K8s.

Container Security Basics

Six rules. Memorize these.

1. Don’t run as root. USER nextjs (we already do this in chugli.ai).
2. Never embed secrets in the image. Anyone who pulls the image can docker history it and recover layers. Use --env-file at runtime or BuildKit secrets at build time.
3. Use minimal base images. alpine (5 MB) over ubuntu (80 MB) over node (Debian-based, 200 MB). Smaller = smaller attack surface + faster pulls.
4. Pin base images by digest, not just tag.

   FROM node:21.7.2-alpine@sha256:abc123...

   Tags can be re-tagged; digests can’t.
5. Scan images for vulnerabilities.

   docker scout cves chugli-ai:1.0.10
   trivy image chugli-ai:1.0.10

   Run this in CI. Block merges if HIGH/CRITICAL CVEs are found.
6. Don’t mount the Docker socket into containers unless you fully understand the consequences. A container with /var/run/docker.sock mounted is effectively root on the host.

What’s in .dockerignore matters as much as what’s in Dockerfile

If .env isn’t in .dockerignore and you write COPY . ., you’ve shipped your production credentials inside the image. Forever. To anyone with pull access.

Our chugli.ai/.dockerignore correctly excludes .env*, .git, and test files. Verify this on every Dockerfile you write.

Try This on Your Own

Three exercises against the actual chugli.ai repo. Don’t push the resulting image — keep everything local.

Exercise 1 — Build it, measure it (10 min)

1. cd chugli.ai
2. Read .env.local (it exists locally; you don’t need real values for this exercise)
3. Run ./docker-build.sh 1.0.99 --no-push — builds without pushing
4. Run docker images chugli-ai — note the final image size
5. Run docker history chugli-ai:1.0.99 — look at each layer’s size. Which layer is the biggest? Why?

Goal: see a real multi-stage build run, understand which layers cost what.

Exercise 2 — Add a HEALTHCHECK (15 min)

1. Edit chugli.ai/Dockerfile
2. Add this above the CMD line:

   HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
     CMD wget -q --spider http://localhost:3000/api/health || exit 1

3. Rebuild
4. docker run -d --name chugli-test -p 3000:3000 chugli-ai:1.0.99
5. Wait 30 seconds, then run docker ps — note the STATUS column now shows (healthy) or (unhealthy)

Goal: make Docker actually know whether the app is responding.

Exercise 3 — Scan it (10 min)

1. docker scout cves chugli-ai:1.0.99 (or trivy image chugli-ai:1.0.99 if you have Trivy installed)
2. Read the output. How many HIGH/CRITICAL CVEs are there?
3. Pick one CVE. Identify which dependency it’s in. Could you upgrade past it?

Goal: experience the “wait, every image has dozens of CVEs?” moment, and learn what to do about it.

Going Deeper — deep-dive-handout.md

Once you’ve internalized the foundations above, the companion handout deep-dive-handout.md in this same folder goes deep on the topics this one only touches:

• Full docker command reference (lifecycle, inspection, exec, images, system)
• Every docker run flag worth knowing, grouped by purpose
• Volumes & persistent storage — named volumes vs bind mounts vs tmpfs, the Mac perf gotchas, backup/restore patterns
• Networking — bridge vs host vs custom networks, DNS resolution, what -p actually does at the iptables level, service-to-service patterns
• BuildKit & buildx — cache mounts, secret mounts, multi-platform builds
• Debugging running containers — including distroless/scratch images where exec doesn’t work
• 5 hands-on exercises against the chugli.ai + Mongo + Redis local stack

Plan ~90 minutes to read it end-to-end, or use it as a lookup when you’re stuck.

Where this leaves us

You should now be able to:

1. Explain why containers exist and how they differ from VMs
2. Write a Dockerfile using the 7 core instructions
3. Read and audit chugli.ai’s Dockerfile, calling out what’s good and what’s missing
4. Build, run, tag, and push an image — and know what each command does
5. Spin up a multi-service stack locally with Docker Compose
6. Spot the most common container security mistakes
7. Add HEALTHCHECK and scan an image for vulnerabilities

Next session takes the image you just built and shows how Kubernetes runs it at scale — deployments, services, probes, resource limits, the lot.