Building a TCP/IP Stack from Scratch in Rust

Every HTTP server I have shipped sits on a TCP stack that I did not write. That is fine professionally and unsatisfying personally, so I built one: a TCP/IP implementation in Rust over a Linux TUN device. It implements RFC 793, the three-way handshake, connection tracking, raw packet parsing and TCP state management.

It is a learning project, not a production stack, and it never carried production traffic. This is a write-up of the implementation and of what changed in my thinking once I had written the thing I had been taking for granted.

01Why a TUN device and not a userland socket

A TUN device is a virtual layer-3 interface. The kernel routes packets to it, you read raw IP frames from it, and you write frames back. You never get a file descriptor from a socket() call, so there is nothing between your code and the bytes.

use tokio::io::{AsyncReadExt, AsyncWriteExt};
 
pub struct TunDevice {
    file: tokio::fs::File,
    mtu: usize,
}
 
impl TunDevice {
    pub async fn open(name: &str, mtu: usize) -> std::io::Result<Self> {
        let file = tokio::fs::OpenOptions::new()
            .read(true)
            .write(true)
            .custom_flags(libc::O_NONBLOCK)
            .open("/dev/net/tun")?;
        // TUNSETIFF with IFF_TUN | IFF_NO_PI: no 4-byte packet-info prefix.
        Ok(Self { file, mtu })
    }
 
    pub async fn read_frame(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
        let n = self.file.read(buf).await?;
        Ok(n)
    }
}

IFF_NO_PI matters more than it looks. Without it, every read starts with a 4-byte struct tun_pi telling you which protocol the frame belongs to. With it, the buffer starts at byte zero of the IP header, and every offset in the rest of the code is four bytes smaller than you expected.

02The checksum is one's complement, and that is not a detail

IP and TCP checksums are the 16-bit one's complement of the one's complement sum of the data, with the checksum field zeroed during calculation. It is six lines, and it is very easy to get subtly wrong, because the field is byte-swapped relative to everything else in the header.

pub fn internet_checksum(data: &[u8]) -> u16 {
    let mut sum: u32 = 0;
    let mut chunks = data.chunks_exact(2);
 
    for chunk in &mut chunks {
        sum += u32::from(u16::from_be_bytes([chunk[0], chunk[1]]));
    }
 
    // Odd trailing byte is padded on the right.
    if let [last] = chunks.remainder() {
        sum += u32::from(u16::from_be_bytes([*last, 0]));
    }
 
    while sum >> 16 != 0 {
        sum = (sum & 0xffff) + (sum >> 16);
    }
 
    !(sum as u16)
}

The fold loop at the end is the part that is easy to forget. Sums above 16 bits carry back into the low 16, and skipping the fold gives you a checksum that looks fine for short headers and is wrong the moment a segment carries options or payload.

03The handshake is a state machine

RFC 793 defines the states a connection can be in. You do not need all of them to accept a connection. You need four, and you need them to be a state machine rather than a chain of boolean flags.

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TcpState {
    Closed,
    Listen,
    SynReceived,
    Established,
}
 
impl Connection {
    pub fn on_segment(&mut self, seg: &Segment) -> Result<Transition, TcpError> {
        match (self.state, seg.flags) {
            (TcpState::Listen, Flags::SYN) => self.accept_syn(seg),
            (TcpState::SynReceived, Flags::SYN | Flags::ACK) if seg.ack == self.seq + 1 => {
                self.state = TcpState::Established;
                Ok(Transition::Established)
            }
            (TcpState::Established, _) => Ok(Transition::Data),
            _ => Err(TcpError::UnexpectedSegment { state: self.state, seg }),
        }
    }
}

The transition that matters most is SYN-RECEIVED plus SYN|ACK. Accepting it without checking that ack == seq + 1 means completing a handshake with a client that never sent a SYN. That is the whole SYN-flood story, and writing the state machine by hand is the fastest way I know to understand why SYN cookies and rate limits exist at all.

04Connection tracking is a hash of the four-tuple

A connection is identified by source address, source port, destination address and destination port. Two clients can share a source IP behind NAT and must not share a connection, so the key has to be all four.

#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct FlowKey {
    pub src_addr: Ipv4Addr,
    pub src_port: u16,
    pub dst_addr: Ipv4Addr,
    pub dst_port: u16,
}
 
pub struct ConnectionTable {
    flows: HashMap<FlowKey, Connection>,
}

Hashing a four-field struct and using the ports directly means no manual packing, no endianness bugs in the table, and HashMap gives the average constant-time lookup the data path needs.

05Segment parsing is where correctness actually lives

The fixed part of a TCP header is 20 bytes, and the data offset field in the first byte tells you how much of the header is options. Everything else is flags and two sequence numbers.

impl<'a> Segment<'a> {
    pub fn parse(mut buf: &'a [u8]) -> Result<Self, TcpError> {
        let data_offset = ((buf[12] >> 4) as usize) * 4;
        if data_offset < 20 || data_offset > buf.len() {
            return Err(TcpError::MalformedHeader);
        }
 
        let flags = buf[13];
        let sequence = u32::from_be_bytes(buf[4..8].try_into().unwrap());
        let acknowledgment = u32::from_be_bytes(buf[8..12].try_into().unwrap());
 
        let payload = &buf[data_offset..];
 
        Ok(Self { flags, sequence, acknowledgment, payload })
    }
}

A data_offset below 20 is a malformed header and must be rejected rather than clamped. This is a small rule with an outsized consequence: if you trust the header length, whoever sent the packet controls how much of your buffer you treat as payload.

06What changed in how I write application code

Nothing about how I write a payment service changed. That was the least satisfying answer I could give. Three things changed in how I reason about it.

Sequence numbers are not free. Once you have implemented wraparound, you never again assume seq_a < seq_b means a arrived first. Every ordering question in a real service, ordering results, comparing timestamps, deciding whether this webhook is newer than that one, becomes a question with wraparound semantics. This is part of why I use explicit monotonic version columns rather than relying on comparison order.

Head-of-line blocking is a design choice. One slow write blocks every segment behind it in a naive implementation. Real stacks use per-segment acknowledgement and out-of-order queues. When a service has a slow downstream call inside a request handler, it has effectively chosen the naive implementation.

The network lies, so a service must not trust it. Duplicate segments, out-of-order delivery, retransmissions, a checksum that passes because the hardware offloaded the sum: none of these are exceptional, they are ordinary. Idempotency keys, deduplication on a unique constraint and monotonic state transitions are not paranoia. They are the only reason a payment can be retried safely.

That last point is the real reason I built this. The stack is a curiosity. The instinct it gives me, that every message from the outside world is a maybe, is what I use every week.

The code is here (opens in a new tab). Read it as the notes above and not as something to run in anger.