The other day I was interested in how many steps occur between sending a POST or GET request from a website to the actual processing that happens on the CPU of the server. I figured that I knew bits and pieces of the puzzle but I wanted to see the complete path from the highest levels of abstraction all the way to the lowest without missing anything too big in-between. It turns out that in a modern web system there are a lot of steps. I have been really fascinated by this much like the explorer that wants to find a path from one known place to another. If you are interested in better understanding how your computer works you might find walking along this path with your tech stack helpful. The general idea is very good in many contexts, this could be anything from looking at how printf really works through to wreaking havoc in an interview by understanding how clojure really works.

One thing that's always been very obvious to systems programmers is the amount of abstraction that exists between applications written in high level languages and the processing that happens on the metal. We have these fantastic frameworks and abstractions that save us from details we are not directly concerned with. The fact that these frameworks have good API's for their abstractions is a large factor in why they make us so productive. However if like me you have ever wondered what the abstractions are doing behind the scenes there's always this sense of wonder and a burning desire to know how it all really works. This post is an attempt at going from the highest level of abstraction of seeing a website with our own eyes all the way down to the lowest level, the electrons. This whole process is long so if areas don't interest you feel free to skip over sections. If you do skip take a moment to enjoy the wonders of the abstractions that modern software offers, you could still be productive without having to know all the details you just skipped.

To make this tangible this article uses real code! You may wish to look at the source code used for this article if you would like to interact with the code directly.

Overview

Here are all the steps from the web browser down to the electrons that are covered in this article. This index is arranged in the way that the abstractions layer in the real code, follow the links to see more details on each step:

• Frontend
  • prelude: GET request
  • Browser page rendering
  • POST request
  • sidenote: CSRF token
• Network stack
  • sidenote: The Internet
  • TCP
  • sidenote: more comprehensive treatment of network stack
• Backend
  • Handling web request
  • WSGI
  • Django
    • Django URL routing
    • Django views
  • Python implementations
    • CPython
    • CPython bytecode
    • CPython bytecode execution details
  • Machine Code
    • CPython to machine code
    • Machine code execution
  • Hardware implementation details
    • Microcode
    • Processor pipeline
    • Silicon implementation of addition
    • Silicon adder unit
    • AND gate
    • Transistor

The overall "product" - the highest level of abstraction

Specification: The user is supplied with a webpage where they can enter in 2 numbers and find the result of adding those numbers. The addition itself must be performed on the server and the results will be presented to the user via a results page.

To do this we have a web server running the Django web framework and the simplest code possible that lets this all happen (hopefully without hurting the educational value of this series of articles). For the purposes of keeping this manageable we are aiming for a minimal amount of conceptual complexity in order to focus on the nature of the layers of abstraction, in particular giving some insight into less commonly thought about layers. We are avoiding some abstractions that we would use in real projects as these would increase the number of layers in this article and increase the conceptual loading on the reader. (The reason we use these abstractions in our projects is because they reduce overall complexity of implementation and make us more productive). So infrastructure as a service, containers, microservices or any other potentially useful technique that introduces complexities will not be covered in this. The code for this article is a minimal set that can provide a window into the nature of the deep stack of abstractions we encounter with modern web apps.

See the next article in the series for the start of the journey, the web front end.