The other day I was interested in how many steps occur between sending a
POST or GET request from a website to the actual processing that happens
on the CPU of the server. I figured that I knew bits and pieces of the
puzzle but I wanted to see the complete path from the highest levels of
abstraction all the way to the lowest without missing anything too big
in-between. It turns out that in a modern web system there are a lot of
steps. I have been really fascinated by this much like the explorer
that wants to find a path from one known place to another. If you are
interested in better understanding how your computer works you might
find walking along this path with your tech stack helpful.
The general idea is very good in many contexts, this could be anything
from looking at how
printf really works
through to wreaking havoc in an interview by understanding how clojure really works.
One thing that's always been very obvious to systems programmers is the amount of abstraction that exists between applications written in high level languages and the processing that happens on the metal. We have these fantastic frameworks and abstractions that save us from details we are not directly concerned with. The fact that these frameworks have good API's for their abstractions is a large factor in why they make us so productive. However if like me you have ever wondered what the abstractions are doing behind the scenes there's always this sense of wonder and a burning desire to know how it all really works. This post is an attempt at going from the highest level of abstraction of seeing a website with our own eyes all the way down to the lowest level, the electrons. This whole process is long so if areas don't interest you feel free to skip over sections. If you do skip take a moment to enjoy the wonders of the abstractions that modern software offers, you could still be productive without having to know all the details you just skipped.
To make this tangible this article uses real code! This can be found JaggedVerge's GitHub page, you may wish to look at the source code used for this article if you would like to interact with the code directly.
Here are all the steps from the web browser down to the electrons that are covered in this article. This index is arranged in the way that the abstractions layer in the real code, follow the links to see more details on each step:
- Network stack
- Handling web request
- Python implementations
- Machine Code
- Hardware implementation details
The overall "product" - the highest level of abstraction
Specification: The user is supplied with a webpage where they can enter in 2 numbers and find the result of adding those numbers. The addition itself must be performed on the server and the results will be presented to the user via a results page.
To do this we have a web server running the Django web framework and the simplest code possible that lets this all happen (hopefully without hurting the educational value of this series of articles). For the purposes of keeping this manageable we are aiming for a minimal amount of conceptual complexity in order to focus on the nature of the layers of abstraction, in particular giving some insight into less commonly thought about layers. We are avoiding some abstractions that we would use in real projects as these would increase the number of layers in this article and increase the conceptual loading on the reader. (The reason we use these abstractions in our projects is because they reduce overall complexity of implementation and make us more productive). So infrastructure as a service, containers, microservices or any other potentially useful technique that introduces complexities will not be covered in this. The code for this article is a minimal set that can provide a window into the nature of the deep stack of abstractions we encounter with modern web apps.
See the next article in the series for the start of the journey, the web front end.
This post is part 1 of the "PageRequestsToTheMetal" series:
- Page requests to the metal - introduction *
- Page requests to the metal - frontend
- Page requests to the metal - Network stack - From frontend to backend
- Page requests to the metal - Backend - What happens on the server
- Page requests to the metal - Backend web framework
- Page requests to the metal - Backend implementation
- Page requests to the metal - hardware level