VOOZH about

URL: https://dev.to/hajirufai/building-an-http-server-framework-from-raw-tcp-sockets-in-python-82e

⇱ Building an HTTP Server Framework from Raw TCP Sockets in Python - DEV Community


Most web developers never look under the hood of their framework. Flask, Django, Express — they all abstract away the same thing: raw TCP bytes being transformed into structured HTTP messages. I wanted to understand that transformation, so I built ServeKit — a complete HTTP/1.1 server framework from nothing but Python's socket module.

No Flask. No http.server. No dependencies. Just socket.socket(AF_INET, SOCK_STREAM) and hand-written protocol parsing.

The TCP Foundation

Every HTTP server starts the same way: bind a socket, listen for connections, accept them, read bytes.

import socket
import selectors

class TCPServer:
 def __init__(self, host="0.0.0.0", port=8080, workers=4):
 self.host = host
 self.port = port
 self._selector = selectors.DefaultSelector()
 self._executor = ThreadPoolExecutor(max_workers=workers)

 def start(self):
 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
 sock.bind((self.host, self.port))
 sock.listen(128)
 sock.setblocking(False)

 self._selector.register(sock, selectors.EVENT_READ)
 while self._running:
 events = self._selector.select(timeout=1)
 for key, mask in events:
 if key.data is None:
 self._accept(key.fileobj)
 else:
 self._executor.submit(self._handle, key.fileobj)

The selectors module gives us non-blocking I/O — we can handle thousands of connections without one blocking the rest. When a connection is ready to read, we hand it to a thread pool worker.

Parsing HTTP by Hand

When raw bytes arrive on a TCP connection, they look like this:

GET /users/42?format=json HTTP/1.1\r\n
Host: localhost:8080\r\n
Accept: application/json\r\n
Connection: keep-alive\r\n
\r\n

That's it. Text separated by \r\n, with a blank line marking the end of headers. My parser reads this byte-by-byte:

class HTTPParser:
 def parse(self, data: bytes, client_addr=("0.0.0.0", 0)):
 # Find the header/body boundary
 head_end = data.find(b"\r\n\r\n")
 head_str = data[:head_end].decode("latin-1")
 body_bytes = data[head_end + 4:]

 lines = head_str.split("\r\n")
 method, path, query, version = self._parse_request_line(lines[0])
 headers = self._parse_headers(lines[1:])
 body = self._read_body(headers, body_bytes, data, head_end + 4)

 return Request(method=method, path=path, headers=headers,
 body=body, query_string=query)

The parser handles everything from HTTP/1.0 to chunked transfer encoding. It validates methods, enforces size limits (no 10GB request lines), and builds a clean Request object.

URL Routing with Priority

Route matching seems simple until you have overlapping patterns. ServeKit supports three match types with clear priority:

  1. Exact match: /users/me
  2. Parameterized: /users/{id}
  3. Wildcard: /files/*path
class Router:
 def resolve(self, method, path):
 # Exact match first
 for route in self._routes:
 if route.pattern == path and route.method in (method, "ANY"):
 return route.handler, {}

 # Then parameterized
 for route in self._routes:
 if route.has_params:
 params = route.match(path)
 if params is not None:
 return route.handler, params

 # Finally wildcards
 for route in self._routes:
 if route.is_wildcard:
 params = route.match(path)
 if params is not None:
 return route.handler, params

 raise NotFound(f"No route for {method}{path}")

This means /users/me always wins over /users/{id}, which always wins over /users/*path. Route groups add prefix support for API versioning:

api = app.group("/api/v1")

@api.get("/users/{id}")
def get_user(req, res):
 res.json({"id": req.params["id"]})
# Matches: GET /api/v1/users/42

Middleware: The Onion Model

Middleware wraps handlers in layers. Each middleware calls next_handler to continue the chain, or skips it to short-circuit:

class MiddlewareChain:
 def execute(self, req, res, handler):
 def build_chain(index):
 if index >= len(self._middleware):
 return handler
 mw = self._middleware[index]
 def next_fn(req, res):
 mw(req, res, build_chain(index + 1))
 return next_fn

 build_chain(0)(req, res)

ServeKit ships with five built-in middleware:

  • CORS — handles preflight OPTIONS and sets Access-Control headers
  • Compression — gzip responses above a size threshold
  • Rate limiting — sliding-window limiter with proper 429 responses
  • Basic auth — HTTP Basic with challenge headers
  • Logger — colored request/response timing

WebSocket: Upgrading the Protocol

The most interesting part was WebSocket support. The handshake is a standard HTTP request with magic headers:

def build_upgrade_response(request):
 ws_key = request.headers["Sec-WebSocket-Key"]
 # Concatenate with the magic GUID from RFC 6455
 combined = ws_key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
 accept = base64.b64encode(hashlib.sha1(combined.encode()).digest())

 return (
 b"HTTP/1.1 101 Switching Protocols\r\n"
 b"Upgrade: websocket\r\n"
 b"Connection: Upgrade\r\n"
 b"Sec-WebSocket-Accept: " + accept + b"\r\n\r\n"
 )

After the handshake, the connection switches from HTTP to WebSocket frames — a binary format with opcode, length, optional mask, and payload. Parsing these frames is surprisingly tricky because of variable-length encoding: payloads under 126 bytes use 1 byte for length, 126-65535 use 2 bytes, and larger use 8 bytes.

Response Serialization

The response builder mirrors the parser — turning a structured object back into HTTP bytes:

class Response:
 def serialize(self):
 status_line = f"HTTP/1.1 {self.status_code}{phrase}\r\n"
 header_lines = "".join(f"{k}: {v}\r\n" for k, v in self.headers.items())
 return (status_line + header_lines + "\r\n").encode() + self.body

The chainable API makes handlers clean:

@app.post("/api/items")
def create_item(req, res):
 data = req.json()
 res.status(201).header("X-Custom", "value").json({"created": data})

What I Learned

Building an HTTP server from scratch taught me things I never would have learned from using frameworks:

  1. HTTP is just text over TCP. The entire protocol is human-readable until you get to WebSocket frames.

  2. Keep-alive is why HTTP/1.1 matters. The difference between opening a new TCP connection per request vs. reusing one is massive for performance.

  3. Headers are case-insensitive. The spec says so, but most developers assume lowercase. A CaseInsensitiveDict solves this cleanly.

  4. Chunked transfer encoding exists because servers don't always know Content-Length upfront. Streaming responses need it.

  5. WebSocket masking is XOR. Client-to-server frames must be masked; server-to-client frames must not. The mask is 4 random bytes, and unmasking is just payload[i] ^= mask[i % 4].

The Result

ServeKit is ~4,500 lines of Python with 222 passing tests. It handles:

  • HTTP/1.1 with keep-alive
  • Routing with params and wildcards
  • 5 built-in middleware
  • Static file serving with caching
  • WebSocket support
  • Graceful shutdown

All from raw TCP sockets.

# Try it
git clone https://github.com/hajirufai/servekit
cd servekit
python examples/hello.py
# → http://localhost:8080

GitHub: github.com/hajirufai/servekit
Live page: hajirufai.github.io/servekit