K8s Deep Dive Ep 1: What happens after kubectl run returns

Designing kubectl run command

Let's imagine you are the engineer tasked with implementing what the kubectl run command does. You need to develop a script that, given some parameters, downloads a docker image onto a local machine and runs the application as a container. What would you do?

You might think of developing a CLI application that connects to a machine, sends a command to pull the image, and once it's done pulling, starts the container. Perhaps something like the diagram below.

However, you'll quickly realise there are some problems with the design, as indicated in the diagram. For one, the connection to lab-worker could get cut halfway while the image is still being pulled. Then the command will fail and you need to manually re-run the command.

Secondly, if for some reason the lab-worker node crashes a few hours later, someone will need to re-run the kubectl run command so the application can be hosted on another node.

So how does kubectl run tackle these problems?

Inspecting kubectl run

Let's start inspecting what the kubectl run command does. To do that, we will use the -v flag. The -v flag sets the verbosity level of the logs, and it goes from 0-9 (official documentation). Let's start with level 6 first to get a rough idea of what the command does.

Who kubectl talks to

❯ kubectl run nginx-demo --image=nginx --port=80 --expose -v=6

You will see a response similar to the following:

I1031 22:28:04.917560   39285 loader.go:407] Config loaded from file:  /Users/danipurwadi/.kube/config
I1031 22:28:04.917896   39285 envvar.go:195] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
...
I1031 22:28:04.941691   39285 round_trippers.go:632] "Response" verb="POST" url="https://127.0.0.1:58827/api/v1/namespaces/default/pods?fieldManager=kubectl-run" status="201 Created" milliseconds=15
I1031 22:28:04.947809   39285 round_trippers.go:632] "Response" verb="POST" url="https://127.0.0.1:58827/api/v1/namespaces/default/services?fieldManager=kubectl-run" status="201 Created" milliseconds=4
service/nginx-demo created
pod/nginx-demo created

Let's go through them line by line.

You'll see that it begins with first detecting the kubernetes config file stored locally. I1031 22:28:04.917560 39285 loader.go:407] Config loaded from file: /Users/danipurwadi/.kube/config

Then, it loads a bunch of feature gate settings (I cut most of them from the output above). For our purposes, we can ignore these for now, but just think of them as configurations that the command line tool has in place.

The interesting bit is the next 2 lines:

I1031 22:28:04.941691   39285 round_trippers.go:632] "Response" verb="POST" url="https://127.0.0.1:58827/api/v1/namespaces/default/pods?fieldManager=kubectl-run" status="201 Created" milliseconds=15
I1031 22:28:04.947809   39285 round_trippers.go:632] "Response" verb="POST" url="https://127.0.0.1:58827/api/v1/namespaces/default/services?fieldManager=kubectl-run" status="201 Created" milliseconds=4

Here we see that what kubectl run does is basically execute one POST call to create the pod, and another POST call to create the service! We'll come back to the service in Part 6. For now, let's follow the pod.

This gives us the first hint that what kubectl does is just sending the command as an HTTP request to https://127.0.0.1:58827. But what is listening on that address? 127.0.0.1 is our own machine, and in kind each node is a Docker container, so let's check which container publishes port 58827.

❯ docker ps --format '{{.Names}}\t{{.Ports}}'
lab-worker
lab-worker2
lab-control-plane	127.0.0.1:58827->6443/tcp

Docker forwards port 58827 on our machine to port 6443 on lab-control-plane. Now let's see which process listens on port 6443 inside that node.

❯ docker exec lab-control-plane ss -ltnp | grep 6443
LISTEN 0      4096               *:6443             *:*    users:(("kube-apiserver",pid=523,fd=4))

So the server behind that address is the kube-apiserver. kubectl knew where to find it because kind wrote the address into ~/.kube/config, the file from the first log line, when it created the cluster.

What it sends

To see the request itself, raise the verbosity to 8 (JSON is formatted in the example). The nginx-demo pod from the last run still exists, so delete it first.

❯ kubectl delete pod,svc nginx-demo
❯ kubectl run nginx-demo --image=nginx --port=80 --expose -v=8
...
I1031 22:28:09.775821   39289 helper.go:246] "Request Body" body=<
 {
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "nginx-demo",
    "labels": {
      "run": "nginx-demo"
    }
  },
  "spec": {
    "containers": [
      {
        "name": "nginx-demo",
        "image": "nginx",
        "ports": [
          {
            "containerPort": 80
          }
        ],
        "resources": {}
      }
    ],
    "restartPolicy": "Always",
    "dnsPolicy": "ClusterFirst"
  },
  "status": {}
}>
...

We can see kubectl injects the image, port and pod name into the request and adds a few more default parameters like the restartPolicy or dnsPolicy. Notice what's missing. There's no request to a node, the scheduler, or a container runtime, because kubectl never talks to any of them.

Who does the work

So how does this request provision the pod and eventually run a container?

Let's take a closer look at what happens in the cluster after the kubectl command was sent. Let's use the kubectl get events command to do that.

❯ kubectl get events --field-selector involvedObject.name=nginx-demo --sort-by=.lastTimestamp \
  -o custom-columns=REASON:.reason,FROM:.source.component,NODE:.source.host,MESSAGE:.message
REASON      FROM                NODE          MESSAGE
Scheduled   default-scheduler   <none>        Successfully assigned default/nginx-demo to lab-worker
Pulling     kubelet             lab-worker    Pulling image "nginx"
Pulled      kubelet             lab-worker    Successfully pulled image "nginx" in 1.399s (1.399s including waiting). Image size: 62240088 bytes.
Created     kubelet             lab-worker    Container created
Started     kubelet             lab-worker    Container started

You'll see the events are emitted by someone other than kubectl:

• default-scheduler writes Scheduled.
• kubelet on the chosen node writes Pulling, Pulled, Created, Started.

From here we can hypothesise that it's probably kube-scheduler, which reports itself as default-scheduler in events, that assigns the workload to one of the nodes (in this case lab-worker). The kubelet on lab-worker is then the one who pulls the image and starts the container.

Where the pod lives

But kubectl had already exited by the time they did any of this. So how did they know about our pod? Let's read the pod back and compare it with the request body we sent.

❯ kubectl get pod nginx-demo -o yaml
metadata:
  creationTimestamp: "2026-10-31T14:28:09Z"
  name: nginx-demo
  ...
spec:
  containers:
  - image: nginx
    name: nginx-demo
    ...
  nodeName: lab-worker
  ...
status:
  containerStatuses:
  - name: nginx-demo
    ...
    state:
      running:
        startedAt: "2026-10-31T14:28:11Z"
  ...
  phase: Running
  podIP: 10.244.1.6

Our request body had no nodeName and an empty status. The scheduler added nodeName: lab-worker, and the kubelet on lab-worker filled in status. The timestamps show when (they're in UTC, which is why they look 8 hours off from the logs above). The API server answers kubectl's POST in milliseconds, 15 in our first run, and kubectl exited right after. The container started 2 seconds later. The gap is short here because lab-worker already had the nginx image. If a fresh node is selected, it might take 10 seconds or more, depending on the network speed.

So the pod is a record that several components read and update, and they all do it through the kube-apiserver we found behind https://127.0.0.1:58827. They don't poll it for new pods. The scheduler and every kubelet keep a watch open on the API server, which sends them each change to the pods they care about. The scheduler watches for pods with no node, and each kubelet watches for pods assigned to its own node. We'll see how watches work in Part 3.

The API server stores the record in etcd, a key-value database on the control plane node, and we can look up the pod's key there directly. etcd only accepts clients that present a certificate, so we use the ones already on the control plane node.

❯ kubectl -n kube-system exec etcd-lab-control-plane -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  get /registry/pods/default/nginx-demo --keys-only
/registry/pods/default/nginx-demo

Putting it together

Let's put everything we know together. We will build up this map slowly as we go through the various components.

We see several components working together around one pod record:

• kube-apiserver is the process on port 6443 that answered our POST. Every client and component reads and writes cluster state through it.
• etcd is the database where the API server stored our pod, under /registry/pods/default/nginx-demo. It's the source of truth for the cluster.
• kube-scheduler picks a node for each pod that doesn't have one yet, and writes that node's name onto the pod. It wrote nodeName: lab-worker onto ours.
• kubelet watches for pods assigned to its node, pulls the image, starts the container and writes the pod's status. The kubelet on lab-worker wrote the four events we saw.

Let's go back to the two problems with our first design.

• Q: What happens when the connection is cut halfway while the image is being pulled?

  • A: Nothing, because kubectl isn't connected by then. It only waits for the API server to store the pod, which took 15 milliseconds in our first run. The scheduler and the kubelet read the pod from the API server and do the slow work afterwards.

• Q: What happens if one of the nodes dies?

  • A: Our pod is lost. I tested this by stopping the lab-worker container, which is the node our pod runs on. If you follow along, this deletes every pod on lab-worker, not just nginx-demo.

❯ docker stop lab-worker
❯ kubectl get nodes lab-worker          # about 1 minute later
NAME         STATUS     ROLES    AGE    VERSION
lab-worker   NotReady   <none>   6d7h   v1.37.0
❯ kubectl get pod nginx-demo -o wide    # about 6 minutes later
NAME         READY   STATUS        RESTARTS   AGE     IP           NODE         NOMINATED NODE   READINESS GATES
nginx-demo   1/1     Terminating   0          6m47s   10.244.1.6   lab-worker   <none>           <none>

The pod never moved to lab-worker2. After the node had been NotReady for 5 minutes, Kubernetes marked the pod for deletion, and nothing else happened. To bring the node back, start its container again:

❯ docker start lab-worker

Once lab-worker was back, its kubelet finished the deletion and nginx-demo was gone for good.

The scheduler only picks a node for a pod that doesn't have one yet. Once a pod has a nodeName, it stays on that node for its whole life. The Pod lifecycle docs say this directly: "Pods do not, by themselves, self-heal." restartPolicy: Always doesn't help either. It only tells the kubelet to restart a crashed container on the same node.

So storing the pod in etcd fixes the first problem but not the second. Remember the note in our first diagram, that nothing still says "there should be an nginx running". Our pod record says "run this container on lab-worker", which stops being useful when lab-worker dies. To survive a node failure, something has to keep saying "there should be one nginx running" and create a new pod when the old one disappears. Kubernetes does this with controllers. A Deployment, for example, uses a ReplicaSet controller that notices a missing pod and creates a replacement, and the scheduler then places the replacement on a healthy node.

The next four articles take the map one component at a time. In Part 2 we'll follow our POST into the API server and see what it does before the pod lands in etcd. Controllers wait until Part 7, where we'll run nginx as a Deployment and stop lab-worker again.

Glossary