K8s Deep Dive Ep 0: Introduction

Kubernetes Deep Dive Series Intro

When I moved from a SWE (Software Engineer) role working with backend applications into devops/infrastructure, I struggled a lot to wrap my head around the various components of Kubernetes and how it works internally. After reading this article written by Swyx on learning in public, I was inspired to share my learnings in public.

This series is my best attempt to distill my learning journey over the last year, learning various internal Kubernetes components and how they each work and interact. If you are an engineer who has some exposure to Kubernetes but never really understood how it works, or what its components are and how they interact with each other, this series is for you.

I have found that I learn best by playing around on my own rather than just absorbing passively. Therefore, each post in the series includes a hands-on lab so you can follow along and play around in your local environment. We are going to run a cluster locally and slowly understand what the various components in Kubernetes are and how they interact with each other.

What this series covers

To begin the series, I thought it would be fun to see what goes on beneath the surface when we provision a pod using kubectl run my-pod. We will trace the main components of Kubernetes by dissecting what happens behind the scenes . The series will be split into 7 articles.

Ep 1 - What happens after kubectl run returns
Ep 2 - Inside the API Server
Ep 3 - etcd and watch
Ep 4 - The scheduler
Ep 5 - The kubelet runtime
Ep 6 - Exposing pod via service
Ep 7 - Why we need Deployment

Glossary

Each post has a glossary of the Kubernetes terms it uses. I find it especially helpful as we will throw around a lot of technical terms like workload or node which can be confusing if it's the first time you're hearing about them.

Let's introduce a few terms you will encounter when setting up the cluster.

Terms	Definition
Workload	an application running on Kubernetes
Cluster	a group of machines, called nodes, that Kubernetes manages together to run your applications
Node	a single machine in the cluster, either a physical server or a virtual machine. In `kind`, each node is a Docker container
Control plane	the components that manage the cluster. They store the cluster's state and decide which node each pod runs on
Worker node	a node that runs your workloads. The control-plane node mostly runs the control plane components, plus a few system pods we'll see later
Container	an application packaged with everything it needs to run, started from an image such as `nginx`
Container runtime	the software on each node that starts and stops containers. `kind` uses containerd
Pod	the smallest unit Kubernetes runs. A pod holds one or more containers that share a network address
Service	an object that gives a group of pods one stable network address, so other applications can reach them

Local Setup

As mentioned, this series will be hands on, so if you would like to follow along, here is a short guide of what you need. If you just want to read, feel free to skip this section!

We are going to run in a local cluster using kind (Kubernetes in Docker), where each Kubernetes node runs as a Docker container. In a production cluster, typically each node will run as a virtual machine (VM) instance. For this series, I will be using kind version 0.33 which will generate a Kubernetes cluster version 1.37.

Pre-requisite

This setup guide assumes you already have the following installed on your machine

Create the cluster

First, create the config file to provision the cluster. For this series we will create 1 control-plane node and 2 workers, so that the scheduling can be more visible. Save the following as kind-lab.yaml.

# kind-lab.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

Then start up the cluster

❯ kind create cluster --name lab --config kind-lab.yaml --wait 5m

kind also switches your kubectl context to kind-lab. If you have other clusters configured, check that you're pointing at the lab with kubectl config current-context before running anything.

You should see three nodes, all Ready. To interact with the Kubernetes cluster, we use kubectl

❯ kubectl get nodes -o wide
NAME                STATUS   ROLES           VERSION   INTERNAL-IP   OS-IMAGE                       CONTAINER-RUNTIME
lab-control-plane   Ready    control-plane   v1.37.0   172.18.0.2    Debian GNU/Linux 13 (trixie)   containerd://2.3.4
lab-worker          Ready    <none>          v1.37.0   172.18.0.4    Debian GNU/Linux 13 (trixie)   containerd://2.3.4
lab-worker2         Ready    <none>          v1.37.0   172.18.0.3    Debian GNU/Linux 13 (trixie)   containerd://2.3.4

The container runtime is containerd. On kind 0.33 it is containerd 2.3.4.

Provision a Pod

To test that your cluster is healthy, we can try running a pod using the following command

❯ kubectl run test-pod --image=nginx --port=80 --expose
service/test-pod created
pod/test-pod created

We can get a response from the pod by port-forwarding and calling the pod via curl

❯ kubectl port-forward svc/test-pod 8080:80

# open a new terminal and run
❯ curl localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

You should get back the HTML of the nginx welcome page. Once you have seen it, stop the port-forward and delete the test pod and its service.

❯ kubectl delete pod/test-pod svc/test-pod

Congratulations! You have a running cluster ready for the labs. If you would like to start fresh between posts, delete the cluster and create it again.

❯ kind delete cluster --name lab
❯ kind create cluster --name lab --config kind-lab.yaml --wait 5m

Bonus: Inspecting the Control Plane Components

Most of the next few posts focus on the components in the Kubernetes control plane. In kind, each node is a container, so we can go inside the node and look at them.

❯ docker ps --format '{{.Names}}'                         # lab-control-plane, lab-worker, lab-worker2
❯ docker exec -it lab-control-plane bash                  # start a bash shell on the node
root@lab-control-plane:/#

Once inside, list the containers on the control-plane node with crictl, a command-line tool that talks to the container runtime over the CRI (Container Runtime Interface). Here that runtime is containerd. It's basically the equivalent of docker ps -a if you're familiar with docker. The -a flag also includes containers that have exited. I trimmed a few columns from the output to fit the page.

root@lab-control-plane:/# crictl ps -a
CONTAINER       STATE     NAME                      POD                                         NAMESPACE
df245b0c7b93d   Running   coredns                   coredns-559f6c778d-lrbw8                    kube-system
892258006a2c7   Running   coredns                   coredns-559f6c778d-k9rdd                    kube-system
69063f919fed0   Running   local-path-provisioner    local-path-provisioner-75f7fc7dc5-hdjcj     local-path-storage
c63ff7a1b9bee   Running   kindnet-cni               kindnet-jdgfk                               kube-system
17b69404acdac   Running   kube-proxy                kube-proxy-kpnt2                            kube-system
829fefddca7e6   Running   etcd                      etcd-lab-control-plane                      kube-system
2c14950ebfb78   Running   kube-scheduler            kube-scheduler-lab-control-plane            kube-system
1d86c4854821e   Running   kube-apiserver            kube-apiserver-lab-control-plane            kube-system
bf5bf3065c803   Running   kube-controller-manager   kube-controller-manager-lab-control-plane   kube-system

If you don't understand half the commands you typed, or what all of these components do, don't worry! We will slowly unpack each of them over the next few posts. Next, let's see what kubectl run does behind the scenes and map out the Kubernetes components that helped us deploy the application in Ep 1.