Linux

What a network namespace actually is

When you run ip netns add net1, the kernel allocates a brand-new network namespace — an isolated copy of the entire network stack. That's not a metaphor: the new namespace gets its own interfaces, its own routing tables, its own ARP/neighbour tables, its own netfilter (iptables) rule set, its own conntrack table, and its own /proc/net and /sys/class/net. Two namespaces on one host are as isolated, network-wise, as two physical machines.

A namespace is a kernel object that lives only as long as something references it. A running process references its namespaces through its nsproxy; when the last process exits, the namespace is destroyed. ip netns keeps one alive without a process by bind-mounting its handle at /var/run/netns/<name> — which is why you can create it, walk away, and it's still there. nsenter and ip netns exec simply run a command with its namespace pointers swapped to that handle.

This explains the two things beginners trip over. First, a fresh namespace has only a loopback interface, and it starts DOWN — the kernel gives you the bare minimum, so even 127.0.0.1 fails until you bring lo up. Second, isolation is total: the namespace cannot see the host's eth0 or its routes at all. To give it connectivity you must explicitly build a link into it — which is exactly what a veth pair is for.

veth pairs: the virtual cable

A veth pair is the kernel's virtual patch cable. You never create a single veth; they come in twos and behave like the two ends of one Ethernet cable: every frame transmitted on one end appears on the receive path of the other. There is no socket and no user-space copy — the kernel hands the packet straight from one device's TX to the other's RX in a softirq.

  host netns                  net1 netns
 ┌──────────────┐           ┌──────────────┐
 │  veth1-br    │ ◄═══════► │  veth1       │
 │  (on bridge) │   cable   │  10.10.0.2   │
 └──────────────┘           └──────────────┘
       TX on one end  ==  RX on the other end

Move one end into a namespace with ip link set veth1 netns net1 and you've joined two otherwise-isolated stacks: a packet the container sends on veth1 pops out on veth1-br in the host. Both ends must be UP or nothing flows — the single most common fault is bringing up the container end and forgetting the host end, leaving the cable dangling. Every Docker container and every Kubernetes pod gets exactly this: one veth end inside the pod namespace, the other in the host, plugged into a bridge or handed to a routing daemon.

The Linux bridge is a real software switch

Wiring two namespaces together is useful; to let many of them talk you need a switch. The Linux bridge is an L2 switch built into the kernel, and it behaves exactly like the physical switch under your desk. Enslave a port with ip link set veth1-br master br0 and the bridge takes over that interface's receive path.

The bridge keeps a forwarding database (the FDB — see it with bridge fdb show): a table of MAC address → port. On each frame it learns the source MAC's port, then decides where to send: known destination MAC → forward to just that port; unknown or broadcast → flood to every port. That learn-and-forward loop is the entire job of a switch, done in software over your veth ports.

Give the bridge an IP (ip addr add 10.10.0.1/24 dev br0) and it becomes an L3 gateway too: the host now has a foot in the container subnet, which is why each namespace's default route points at the bridge IP. The bridge is simultaneously the switch the containers share and the router that gets them off-subnet. This object, named docker0, is exactly what Docker creates on install; Kubernetes CNI plugins build the same thing per node (often cni0).

How the kernel makes the forwarding decision

Every time the kernel sends a packet it answers one question: which interface and next-hop reach this destination? It consults the routing table of the current namespace and applies longest-prefix match — the most specific route containing the destination wins; if nothing else matches, the default route (0.0.0.0/0) catches it.

You never have to guess what it will pick. ip route get 10.10.0.3 asks the kernel to run the real forwarding decision and print the result — the most useful routing command there is, because it shows reality instead of your mental model. A gateway must be on-link (directly reachable on an attached subnet); that's why default via 10.10.0.1 only works once veth1 holds an address in 10.10.0.0/24. Otherwise the kernel has no interface on which the gateway is reachable and you get the classic Network is unreachable.

netfilter: the five hooks and how NAT really works

So far packets only move at L2/L3. netfilter is where the kernel lets you inspect and rewrite them — the machinery behind port publishing, container internet access, and Kubernetes Services. It exposes five hook points along a packet's path; iptables/nftables rules attach there, grouped into tables (raw, mangle, nat, filter).

                    ┌──────────┐
 in ─► PREROUTING ─► │ routing  │ ─► INPUT ─► local process
        (DNAT)       │ decision │
                     └────┬─────┘
                          ▼
                       FORWARD              local process
                          │                     │
 out ◄─ POSTROUTING ◄─────┴─────── OUTPUT ◄─────┘
         (SNAT / MASQUERADE)

DNAT runs at PREROUTING, rewriting the destination before routing — that's what docker run -p 8080:80 installs: a rule turning host:8080 into container-ip:80. SNAT/MASQUERADE runs at POSTROUTING, rewriting the source on the way out — that's how a private 10.10.0.0/24 container reaches the internet: its source is masqueraded to the host's real address so replies have a way back.

Critically, NAT is stateful. The first packet of a flow is translated and the kernel records the mapping in the conntrack table; every later packet, replies included, is translated automatically from that entry — which is why you write a rule for only one direction. Watch it live with conntrack -L. This also explains a classic outage: at very high connection rates the conntrack table fills (nf_conntrack: table full, dropping packet) and new connections are silently dropped until entries expire. And none of this forwarding happens unless net.ipv4.ip_forward=1 — without it the kernel refuses to route between interfaces and your namespaces stay islands.

A debugging method that always works

When container networking breaks, don't guess — bisect the path and find the last hop that works. Every layer has a command answering a yes/no question; walk them in order until one says no.

1. ip addr / ip link       → is the interface UP and does it have an IP?
2. ip route get DST        → what route will the kernel actually use?
3. ping GATEWAY            → is the next hop reachable at L3?
4. bridge link / bridge fdb → is the veth really enslaved to the bridge?
5. iptables -t nat -L -nv  → are NAT rules present and counting hits?
6. conntrack -L            → is the flow being tracked and translated?
7. tcpdump -i DEV -n       → where in the path do packets actually stop?

The discipline matters more than the commands. Nine of ten container-networking incidents are one of: an interface that's DOWN, a veth never enslaved to the bridge, a missing or wrong default route, ip_forward off, or a NAT rule that isn't matching. tcpdump is the tiebreaker — run it on each device along the path and you can see exactly where a packet vanishes, turning "the network is broken" into "the packet reaches veth1-br but never leaves the bridge, so the host end is down." Producing that sentence in the first five minutes is what separates a senior engineer — and it comes straight from walking this ladder.