A smarter way to use imagePullSecrets in Kubernetes Cluster using ServiceAccounts

[Lomanu4]

Kubernetes is everywhere now a days, so does the container images and fetching the images from a private registry is a norm because of N number of reasons including security, that being the topmost.

Recap


Just to give you a heads up before we dive further, let's assume I fire kubectl create deployment nginx --image nginx what do you think happens?

1. Using kubeconfig, kubectl first authenticates with the API server.
2. It then constructs a deployment object using the values which we provided like deployment name as nginx, image to be used as nginx and some default values and then sends a request to the API server.
3. API server then validates the request and the object definition.
4. This is further stored in etcd datastore.
5. Then the controller manager specifically deployment controller comes into the picture which creates a replica set.
6. Then the replica set creates a pod (this is where our nginx image will be used).
7. Then the schedular assigns the pod to a node, where it will run based on multiple conditions. If interested please check out my previous article on what all things happens in the background:
   
   

   Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

   
   
8. Once the node is identified, the kubelet running on the node is responsible for the management of the pod.
9. Once the pod is scheduled on a specific node, the kubelet looks for the image to be used and if it is already present.

if that is present locally, it will use it.

If not, it pulls the image from the registry.

If the registry is public, there are no issues, it just pulls the image, runs the pods, and we are done.

When imagePullSecrets are used?

• Assume the registry is private, this is where imagePullSecrets come into play.
• Kubernetes will need the secrets to authenticate the registry, so that it can pull the image.

How it works?

• Firstly, we need to create a secret which contains the credentials and some details of our docker registry like server-name, username, password, email, etc.
• It can be created both imperative as well as declarative way.

$ k get secrets -o yaml dockersec
apiVersion: v1
data:
.dockerconfigjson: ENCODED JSON OBJECT
kind: Secret
metadata:
name: dockersec
namespace: default
type: kubernetes.io/dockerconfigjson

$ k create secret docker-registry dockersec --docker-server=DOCKER_REGISTRY --docker-username=USERNAME --docker-password=PASSWORD

• Quick note: The secret must be of a type kubernetes.io/dockerconfigjson or docker-registry based on the way you choose to create the secret.
• Once the secret is created, it must be attached to a pod so that it can use it.
• Just add below content in your pod/deployment/statefulset yaml files, the name of the secret for me is dockersec but you can change based on your requirements.

spec:
imagePullSecrets:
- name: dockersec

• Before pulling the image, the kubelet checks if the image is from a private registry. If so, it looks for imagePullSecrets in the spec.
• Without proper imagePullSecrets, the image pull will fail with ImagePullBackOff.

Problem

• Now assume you have hundreds of YAML files and hundreds of Kubernetes Object where we have to use this imagePullSecrets.
• Just for the sake of doing once, you thought to do it manually but just imagine there was a change in the name of the Kubernetes secret.
• Maybe because the organization now decides to streamline the naming convention of all the Kubernetes objects based on environments like prod/preprod/staging/etc.
  • Or your org is now moving to a different registry or a different provider.
  • Or the application is getting re-factored based on different namespaces or functions.
  • There can be N number of reasons and doing all these changes again is a tricky and tiresome task.

One way to look at it is, use of helm charts which can help us to manage all the Kubernetes objects and this kind of a change can be pretty straight forward provided the helm charts are structured correctly and using values file we can just update the name of the secret and then can upgrade the chart, and we are done.

But if in case we are not using and package manager like helm and we are just working with vanilla YAML files then for that as well there is a neat way to do it.

The NEAT way!


We can attach the imagePullSecrets with the serviceaccounts and where-ever the serviceaccount is used, it will automatically populate the imagePullSecrets in the respective object like Pods.

• Below is the current status of my Kubernetes objects, no secrets, a default serviceaccount and no imagePullSecrets updated in the pod.

$ k get secrets
No resources found in default namespace.
$ k get sa
NAME SECRETS AGE
default 0 33m
$ k get pods
NAME READY STATUS RESTARTS AGE
nginx-65fd88fc88-8xjc9 0/1 ImagePullBackOff 0 32m

• Create the docker-registry secret

$ k create secret docker-registry dockersec --docker-server=DOCKER_REGISTRY --docker-username=USERNAME --docker-password=PASSWORD
secret/dockersec created
$ k get secret
NAME TYPE DATA AGE
dockersec kubernetes.io/dockerconfigjson 1 3s

• Patch the default serviceaccount to have the imagePullSecrets. It can be any serviceaccount based on your requirements.

$ kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "dockersec"}]}'
serviceaccount/default patched
$ k get sa
NAME SECRETS AGE
default 0 35m

• Or you can manually edit the serviceaccount as well and add below content.

$ k get sa default -o yaml
apiVersion: v1
imagePullSecrets: # ADD
- name: dockersec # ADD
kind: ServiceAccount
metadata:
name: default
namespace: default

• To see this in action now, delete the pod which is in failed state.

$ k delete pod nginx-65fd88fc88-8xjc9 --force

• Check the events of the pod, now it was successfully able to pull the image from the private registry.

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 65s default-scheduler Successfully assigned default/nginx-65fd88fc88-w8mph to minikube
Normal Pulling 64s kubelet Pulling image "sunnybhambhani/test:v1"
Normal Pulled 37s kubelet Successfully pulled image "sunnybhambhani/test:v1" in 30.288s (30.288s including waiting). Image size: 567185619 bytes.
Normal Created 21s (x3 over 36s) kubelet Created container: test
Normal Started 21s (x3 over 36s) kubelet Started container test

• Interesting bit is imagePullSecrets are automatically added to the object.

$ k get pods nginx-65fd88fc88-w8mph -o yaml | grep -i serviceaccountname
serviceAccountName: default
$ k get pods nginx-65fd88fc88-w8mph -o yaml | grep -i imagepullse -A 1
imagePullSecrets:
- name: dockersec
Bonus Tip

• In case the pod is not in running status, it can be due to multiple reasons.
• Just describe the pod to see what is happening in the background.
• kubectl describe pod POD_NAME | awk '/^Events:/ {found=1; next} found', it looks for the line starting with Events, once found prints the remaining lines.

$ kubectl describe pod nginx-65fd88fc88-8xjc9 | awk '/^Events:/ {found=1; next} found'
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m21s default-scheduler Successfully assigned default/nginx-65fd88fc88-8xjc9 to minikube
Normal Pulling 4m17s (x5 over 7m20s) kubelet Pulling image "REGISTRY/test:v1"
Warning Failed 4m14s (x5 over 7m17s) kubelet Failed to pull image "REGISTRY/test:v1": Error response from daemon: pull access denied for REGISTRY/test, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Warning Failed 4m14s (x5 over 7m17s) kubelet Error: ErrImagePull
Warning Failed 2m41s (x19 over 7m16s) kubelet Error: ImagePullBackOff
Normal BackOff 2m20s (x21 over 7m16s) kubelet Back-off pulling image "REGISTRY/test:v1"

• Here the logs clearly state, requested access to the resource is denied and in-turn the status of the pod is ImagePullBackOff.

$ k get pods
NAME READY STATUS RESTARTS AGE
nginx-65fd88fc88-8xjc9 0/1 ImagePullBackOff 0 76s

References:

• Google.
• Kubernetes docs (
  
  

  Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.

  
  
  ).
• kubectl explain.
• Linux man pages.

PS: This works with any flavor of Kubernetes: minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.

I hope this will be helpful, feel free to add your thoughts and experiences, Happy Learning! ?

Пожалуйста Авторизируйтесь или Зарегистрируйтесь для просмотра скрытого текста.