K8s Deep Dive Ep 2: Kube API Server

Introduction

In Ep 1 we found that kubectl run sends the pod as one POST request and exits. We also learned that this request is sent to a component in Kubernetes called the api server. Think of api-server as the gateway that all clients will go through if they would like to communicate to the cluster.

In this article, we will take a look on what does the api server actually does.

Responsibilities of API Server

If you think about a gateway, what would be one of the most important feature it needs to have? If you answer Authentication or Authorization, then you're absolutely correct! The Kube API Server is the main compoennt that deals with those, but it does more than that. To be precise, here are the list of responsibilities that an api-server does:

1. Authentication
2. Authorisation
3. Rate Limiting (API Priority & Fairness)
4. Mutating Admission
5. Validating Admission
6. Writing to ETCD

Let's look at them one by one

Authentication (Who are you?)

The first thing the server needs to know is who is sending the request and whether they have access. To see how this work, we can try just sending the HTTP command using curl directly instead of kubectl using a random bearer token. The -k flag tells curl not to check the server's certificate, because curl doesn't trust the certificate authority kind created. Use the port from your own ~/.kube/config.

❯ curl -sk https://127.0.0.1:58827/api/v1/namespaces/default/pods
{
  ...
  "message": "pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  ...
  "code": 403
}

The server didn't turn us away for having no credentials. It treated us as a user called system:anonymous and passed the request on to the next stage, which checks what that user is allowed to do. system:anonymous can't list pods, so that stage refused us with a 403. Anonymous access is on by default in kind (this may vary depending on what implementation of Kubernetes are you using).

Let's try again with a made up token.

❯ curl -sk -H 'Authorization: Bearer random-token-123' https://127.0.0.1:58827/api/v1/namespaces/default/pods
{
  ...
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

This request didn't get as far. The API server tried each of its authentication methods, none accepted the token, and it stopped the request with a 401 instead of falling back to the anonymous user. A wrong token gets rejected, and no token at all gets you the anonymous user.

So how does kubectl get through? kind put a client certificate and its private key in ~/.kube/config, and kubectl presents the certificate on every request. curl can do the same if we give it those files. The kubeconfig stores them base64 encoded, so let's decode them. --raw stops kubectl from hiding the certificate data, and --minify keeps only the current context, so [0] picks our cluster and user.

❯ kubectl config view --raw --minify -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d > client.crt
❯ kubectl config view --raw --minify -o jsonpath='{.users[0].user.client-key-data}' | base64 -d > client.key
❯ kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt

client.crt and client.key prove who we are. ca.crt is the certificate authority kind created. With it, curl can check the server's certificate, so we can drop -k.

❯ curl -s --cert client.crt --key client.key --cacert ca.crt https://127.0.0.1:58827/api/v1/namespaces/default/pods
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "196262"
  },
  "items": [
    {
      "metadata": {
        "name": "nginx",
        "namespace": "default",
        ...
      },
      "spec": { ... },
      "status": { ... }
    }
  ]
}

Success! We managed to pass through the authentication gate. We can actually check who did we authenticate us by asking the server

❯ kubectl auth whoami
ATTRIBUTE                                           VALUE
Username                                            kubernetes-admin
Groups                                              [kubeadm:cluster-admins system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [X509SHA256=e4d8c91a...]

You might be wondering where does the username and groups come from? It reads the username and the group from the certificate.

Let's now delete the key and certs because anyone with the files has full control of the cluster. This doesn't really matter if you're running a local kind, but if you're using a real cluster you should delete these file.

❯ rm client.crt client.key ca.crt

Rate Limiting

After authentication, the request is put in a queue called API Priority and Fairness before the authorisation check stage. Think of this stage as checking a rate limit to the client's request. The goal is to prevent one client overwhelming the server so much that it prevents other clients from sending requests to the api-server. Clients that exceed the permitted rate limit can experience increased latency or be dropped with an HTTP 429 (Too Many Requests) error.

Authorization (What are you allowed to do?)

After it passes the API Priority check, the request goes through an authorisation check. Knowing who we are doesn't say what we can do and authorisation ensures the user only execute commands they are allowed to do. On this cluster it's mostly handled by RBAC (role-based access control), where rules grant a user or group permission to do certain things. As you can see from the earlier kubectl auth whoami command, our group is bound to the cluster-admin role, which can do practically anything.

❯ kubectl get clusterrolebinding kubeadm:cluster-admins -o wide
NAME                     ROLE                        AGE    USERS   GROUPS                   SERVICEACCOUNTS
kubeadm:cluster-admins   ClusterRole/cluster-admin   6d9h           kubeadm:cluster-admins

What would it look like if we have a more limited user? We can try using a service account called default. Service account is an identity for a software running on the cluster, think of them as a user account assigned to your application. Let's check if the account has access to create pods

❯ kubectl auth can-i create pods --as=system:serviceaccount:default:default
no

Perfect. So we should see some failure when we try to create a pod using this account

❯ kubectl run y --image=nginx --as=system:serviceaccount:default:default -v=6
...
"Response" verb="POST" url="https://127.0.0.1:58827/api/v1/namespaces/default/pods?fieldManager=kubectl-run" status="403 Forbidden" milliseconds=5
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:default" cannot create resource "pods" in API group "" in the namespace "default"

Glad to see our authorization check working. Notice what the error talks about. It mentions the user, the verb create, the resource pods and the namespace. It says nothing about our pod, because authorization doesn't look at the pod or any of the pod configuration parameters.

Mutating Admission

When you run the kubectl run there is a feature for dry-run that lets you preview what the request will look, and there are 2 input options, a --dry-run=client or --dry-run=server. What's the difference? Let's compare them

❯ kubectl run dry-demo --image=nginx --port=80 --dry-run=client -o yaml > dry-client.yaml
❯ kubectl run dry-demo --image=nginx --port=80 --dry-run=server -o yaml > dry-server.yaml
❯ diff dry-client.yaml dry-server.yaml
>   creationTimestamp: "2026-10-31T16:15:15Z"
>   generation: 1
>   namespace: default
>   uid: e998dbcd-64f7-4ab0-b54f-ac7c4422b47b
>     imagePullPolicy: Always
>       protocol: TCP
>     terminationMessagePath: /dev/termination-log
>     terminationMessagePolicy: File
>     volumeMounts:
>     - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
>       name: kube-api-access-88ghc
>       readOnly: true
>   enableServiceLinks: true
>   preemptionPolicy: PreemptLowerPriority
>   priority: 0
>   schedulerName: default-scheduler
>   securityContext: {}
>   serviceAccount: default
>   serviceAccountName: default
>   terminationGracePeriodSeconds: 30
>   tolerations:
>   - effect: NoExecute
>     key: node.kubernetes.io/not-ready
>     operator: Exists
>     tolerationSeconds: 300
>   - effect: NoExecute
>     key: node.kubernetes.io/unreachable
>     operator: Exists
>     tolerationSeconds: 300
>   volumes:
>   - name: kube-api-access-88ghc
>     projected:
...

We see how the API server added a lot of new fields to our request. There are actually 2 sources where these values come from:

The first is plain defaults. When the API server decodes the request body, it fills in the fields we left out, such as protocol: TCP and terminationGracePeriodSeconds: 30. They're listed in defaults.go. imagePullPolicy: Always is one of them, and it's set to Always because we didn't give the image a tag.

The second is admission control, a set of plugins that run inside the API server after authorization and before the write. The phase that runs first is mutating admission, which means these plugins are allowed to change the pod. Three of them affected our request:

• ServiceAccount set serviceAccountName: default and added the kube-api-access volume, which gives the pod a token for talking to the API server.
• Priority set priority: 0 and preemptionPolicy.
• DefaultTolerationSeconds added the two tolerations.

The plugins are compiled into the kube-apiserver binary, and about two dozen are on by default. We can see that kind adds one more, along with the authorization modes it uses.

❯ docker exec lab-control-plane grep -E 'admission|authorization-mode' /etc/kubernetes/manifests/kube-apiserver.yaml
    - --authorization-mode=Node,RBAC
    - --enable-admission-plugins=NodeRestriction

You can add your own mutating step with a webhook, which is an HTTP call from the API server to a service you run.

Built-in validation

After modifying the pod request, the API server then checks whether the request is valid. Let's try giving it a restartPolicy that doesn't exist. The --overrides flag merges JSON into the pod kubectl builds.

❯ kubectl run bad-pod --image=nginx --overrides='{"apiVersion":"v1","spec":{"restartPolicy":"Sometimes"}}' -v=6
...
"Response" verb="POST" url="https://127.0.0.1:58827/api/v1/namespaces/default/pods?fieldManager=kubectl-run" status="422 Unprocessable Entity" milliseconds=4
The Pod "bad-pod" is invalid: spec.restartPolicy: Unsupported value: "Sometimes": supported values: "Always", "OnFailure", "Never"

This time the server read the pod. 422 means the request was understood, but the object is invalid. This check is the same on every cluster because it comes with the pod's definition. But what if we want to have customised rules for our cluster?

Validating Admission

That's where validating admission plugins come into play. These plugins can't change the pod. They can only accept or reject it. They lets us write rules without running any service, using a ValidatingAdmissionPolicy. The rule is written in CEL (Common Expression Language), a small language for expressions.

Let's try to add a custom rule: that every image in the default namespace must have a tag. Save this as require-tag.yaml. While it's applied, any new pod in default with an untagged image will be refused, so we'll remove it at the end.

# require-tag.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-image-tag
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
  validations:
    - expression: "object.spec.containers.all(c, c.image.contains(':'))"
      message: "pin an image tag"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-image-tag
spec:
  policyName: require-image-tag
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: default

Apply it and wait a few seconds for the API server to pick it up.

❯ kubectl apply -f require-tag.yaml
❯ kubectl run test-pod --image=nginx -v=6
...
"Response" verb="POST" url="https://127.0.0.1:58827/api/v1/namespaces/default/pods?fieldManager=kubectl-run" status="422 Unprocessable Entity" milliseconds=11
The pods "test-pod" is invalid: : ValidatingAdmissionPolicy 'require-image-tag' with binding 'require-image-tag' denied request: pin an image tag
❯ kubectl run with-tag --image=nginx:1.27
pod/with-tag created

Our rule works! Notice that it returned a 422, not a 403. We were allowed to create pods, but this particular pod broke a rule.

Writing to etcd

Only after the request pass through all the previous steps, then it will enter the final stage, writing it to the etcd database. Let's try a valid request this time:

❯ kubectl run nginx-demo --image=nginx:1.27 --port=80
pod/nginx-demo created

Once it's written to the ETCD database, then other components in kubernetes will then process it further, which we will cover in the next article. However, this does not mean there can be no error at this stage. Let's try sending the same request again.

❯ kubectl run nginx-demo --image=nginx:1.27 --port=80 -v=6
...
"Response" verb="POST" url="https://127.0.0.1:58827/api/v1/namespaces/default/pods?fieldManager=kubectl-run" status="409 Conflict" milliseconds=6
Error from server (AlreadyExists): pods "nginx-demo" already exists

Here we see etcd rejecting the second write because pod names needs to be unique in a given namespace.

Putting it together

Let's zoom into the kube-apiserver box from Ep 1's map. Here is what a create request goes through.

Here's what stopped each of our requests:

• A made up token stopped at authentication with 401.
• No token became system:anonymous, and authorization refused it with 403.
• The default service account stopped at authorization with 403.
• restartPolicy: Sometimes stopped at schema validation with 422.
• An image with no tag stopped at our policy with 422.
• A name already in use stopped at the write with 409.

There's one thing this still doesn't solve. All we have now is a record in etcd. Nothing has told the scheduler, and no container is running.

In Ep 3 we'll look at that record in etcd, and at how the scheduler finds out about it without anyone calling it.

Glossary