Secrets

All modern network systems need to deal with sensitive information, such as username, passwords, SSH keys, etc. in the platform. The same applies to the pods in a Kubernetes environment. However, exposing this information in your pod specs as cleartext may introduce security concerns and you need a tool or method to resolve the issue – at least to avoid the cleartext credentials as much as possible.

The Kubernetes secrets object is designed specifically for this purpose – it encodes all sensitive data and exposes it to pods in a controlled way.

The official definition of Kubernetes secrets is:

"A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in an image; putting it in a secret object allows for more control over how it is used and reduces the risk of accidental exposure."

Users can create secrets, and the system also creates secrets. To use a secret, a pod needs to reference the secret.

There are many different types of secrets, each serving a specific use case, and there are also many methods to create a secret and a lot of different ways to refer to it in a pod. A complete discussion of secrets is beyond the scope of this book, so please refer to the official documentation to get all of the details and track all up-to-date changes.

Here, we’ll look at some commonly used secret types. You will also learn several methods to create a secret and how to refer to it in your pods. And once you get to the end of the section, you should understand the main benefits of a Kubernetes secrets object and how it can help improve your system security.

Let’s begin with a few secret terms:

• Opaque: This type of secret can contain arbitrary key-value pairs, so it is treated as unstructured data from Kubernetes’ perspective. All other types of secret have constant content.

• Kubernetes.io/Dockerconfigjson: This type of secret is used to authenticate with a private container registry (for example, a Juniper server) to pull your own private image.

• TLS: A TLS secret contains a TLS private key and certificate. It is used to secure an ingress. You will see an example of an ingress with a TLS secret in Chapter 4.

• Kubernetes.io/service-account-token: When processes running in containers of a pod access the API server, they have to be authenticated as a particular account (for example, account default by default). An account associated with a pod is called a service-account. Kubernetes.io/service-account-token type of secret contains information about Kubernetes service-account. We won’t elaborate on this type of secret and service-account in this book.

• Opaque secret: The secret of type opaque represents arbitrary user-owned data – usually you want to put some kind of sensitive data in secret, for example, username, password, security pin, etc., just about anything you believe is sensitive and you want to carry into your pod.

Define Opaque Secret

First, to make our sensitive data looks less sensitive, let’s encode it with the base64 tool:

$ echo -n 'username1' | base64 
dXNlcm5hbWUx
$ echo -n 'password1' | base64 
cGFzc3dvcmQx

Then put the encoded version of the data in a secret definition YAML file:

apiVersion: v1 
kind: Secret 
metadata:
name: secret-opaque 
type: Opaque
data:
username: dXNlcm5hbWUx 
password: cGFzc3dvcmQx

Alternatively, you can define the same secret from kubectl CLI directly, with the --from-literal option:

kubectl create secret generic secret-opaque    \
--from-literal=username='username1'	\
--from-literal=password='password1'

Either way, a secret will be generated:

$ kubectl get secrets
NAME	TYPE	DATA AGE
secret-opaque	Opaque	2	8s

$ kubectl get secrets secret-opaque -o yaml 
apiVersion: v1
data:
password: cGFzc3dvcmQx 
username: dXNlcm5hbWUx 
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"password":"cGFzc3dvcmQx","username":"dXNlcm5hbWUx"},"kind":"Secret","metadata":{"annotations":{},"name"
:"secret-opaque","namespace":"ns-user-1"},"type":"Opaque"} 
creationTimestamp: 2019-08-22T22:51:18Z
name: secret-opaque 
namespace: ns-user-1 
resourceVersion: "885702"
selfLink: /api/v1/namespaces/ns-user-1/secrets/secret-opaque
 

uid: 5a78d9d4-c52f-11e9-90a3-0050569e6cfc 
type: Opaque

Refer Opaque Secret

Next you will need to use the secret in a pod, and the user information contained in the secret will be carried into the pod. As mentioned, there are different ways to refer the opaque secret in a pod, and correspondingly, the result will be different.

Typically, user information carried from a secret can appear in a container in one of these forms:

• Files

• Environmental variables

Now let’s demonstrate using secret to generate environmental variables in a container:

#pod-webserver-do-secret.yaml 
apiVersion: v1
kind: Pod 
metadata:
name: contrail-webserver-secret 
labels:
app: webserver 
spec:
containers:
- name: contrail-webserver-secret
image: contrailk8sdayone/contrail-webserver 
#envFrom:
#- secretRef:
# name: test-secret 
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: secret-opaque 
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: secret-opaque 
key: password

Spawn the pod and container from this YAML file:

$ kubectl apply -f pod/pod-webserver-do-secret.yaml
pod/contrail-webserver-secret created

Log in the container and verify the generated environmental variables:

$ kubectl exec -it contrail-webserver-secret -- printenv | grep 
SECRET SECRET_USERNAME=username1 
SECRET_PASSWORD=password1

The original sensitive data encoded with base64 is now present in the container!

Dockerconfigjson Secret

The dockerconfigjson secret, as the name indicates, carries the Docker account credential information that is typically stored in a .docker/config.json file. The image in a Kubernetes pod may point to a private container registry. In that case, Kubernetes needs to authenticate it with that registry in order to pull the image. The dockerconfigjson type of secret is designed for this very purpose.

Docker Credential Data

The most straightforward method to create a kubernetes.io/dockerconfigjson type of secret is to provide login information directly with the kubectl command and let it generate the secret:

$ kubectl create secret docker-registry secret-jnpr1 \
--docker-server=hub.juniper.net	\
--docker-username=JNPR-FieldUser213	\
--docker-password=CLJd2kpMsVc9zrAuTFPn 
secret/secret-jnpr created

Verify the secret creation:

$ kubectl get secrets
NAME	TYPE	DATA AGE
secret-jnpr	kubernetes.io/dockerconfigjson	1		6s #<--- 
default-token-hkkzr   kubernetes.io/service-account-token   3		62d

Now inspect the details of the secret:

$ kubectl get secrets secret-jnpr -o yaml 
apiVersion: v1
data:
.dockerconfigjson: eyJhdXRocyI6eyJodWIuanVuaXBlci5uZXQvc2...<snipped>... 
kind: Secret
metadata:
creationTimestamp: 2019-08-14T05:58:48Z 
name: secret-jnpr
namespace: ns-user-1 
resourceVersion: "870370"
selfLink: /api/v1/namespaces/ns-user-1/secrets/secret-jnpr 
uid: 9561cdc3-be58-11e9-9367-0050569e6cfc
type: kubernetes.io/dockerconfigjson

Not surprisingly, you don’t see any sensitive information in the form of cleartext. There is a data portion of the output where you can see a very long string as the value of key: dockerconfigjson. Its appearance seems to have transformed from the original data, but at least it does not contain sensitive information anymore – after all one purpose of using a secret is to improve the system security.

However, the transformation is done by encoding, not encryption, so there is still a way to manually retrieve the original sensitive information: just pipe the value of key .dockerconfigjson into the base64 tool, and the original username and password information is viewable again:

$ echo "eyJhdXRocyI6eyJodWIuanVua..." | base64 -d | python -mjson.tool
{
		"auths": { 
			"hub.juniper.net": {
						"auth": "Sj5QUi1GaWVsZFVzZXIyMTM6Q0xKZDJqcE1zVmM5enJBdVRGUG4=",
						"password": "CLJd2kpMsVc9zrAuTFPn", "username": "JNPR-FieldUser213"
			}
		}
}

Some highlights in this output are:

• The python -mjson.tool is used to format the decoded json data before displaying to the terminal.

• There is an auth key-value pair. It is the token generated based on the authentication information you gave (username and password).

• Later on, when equipped with this secret, a pod will use this token, instead of the username and password to authenticate itself towards the private Docker registry hub.juniper.net in order to pull a Docker image.

$ kubectl get secret secret-jnpr1 \
--output="jsonpath={.data.\.dockerconfigjson}" \
| base64 --decode | python -mjson.tool
{
			"auths": { 
					"hub.juniper.net/security": {
							"auth": "Sj5QUi1GaWVsZFVzZXIyMTM6Q0xKZDJqcE1zVmM5enJBdVRGUG4=",
							"password": "CLJd2kpMsVc9zrAuTFPn", "username": "JNPR-FieldUser213"
					}
			}
}

The --output=xxxx option filters the kubectl get output so only the value of .dockerconfigjson under data is displayed. The value is then piped into base64 with option --decode (alias of -d) to get it decoded.

A docker-registry secret created manually like this will only work with a single private registry. To support multiple private container registries you can create a secret from the Docker credential file.

Docker Credential File (~/.Docker/config.json)

As the name of the key .dockerconfigjson in the secret we created indicates, it serves a similar role as the Docker config file: .docker/config.json. Actually, you can generate the secret directly from the Docker configuration file.

To generate the Docker credential information, first check the Docker config file:

$ cat .docker/config.json
{
......
"auths": {},
......
}

There’s nothing really here. Depending on the usage of the set up you may see different output, but the point is that this Docker config file will be updated automatically every time you docker login a new registry:

$ cat mydockerpass.txt | \ 
	docker login hub.juniper.net \
		--username JNPR-FieldUser213 \
		--password-stdin 
Login Succeeded

The file mydockerpass.txt is the login password for username JNPR-FieldUser213. Saving the password in a file and then piping it to the docker login command with --password-stdin option has an advantage of not exposing the password cleartext in the shell history.

$ docker login hub.juniper.net --username XXXXXXXXXXXXXXX --password XXXXXXXXXXXXXX 
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded

Now the Docker credential information is generated in the updated config.json file:

$ cat .docker/config.json
{
......
		"auths": { #<---
				"hub.juniper.net": {
				"auth": "Sj5QUi1GaWVsZFVzZXIyMTM6Q0xKZDJqcE1zVmM5enJBdVRGUG4="
				}
		},
......
}

The login process creates or updates a config.json file that holds the authorization token. Let’s create a secret from the .docker/config.json file:

$ kubectl create secret generic secret-jnpr2 \
--from-file=.dockerconfigjson=/root/.docker/config.json \
--type=kubernetes.io/dockerconfigjson 
secret/secret-jnpr2 created

$ kubectl get secrets
 

NAME	TYPE	DATA AGE
secret-jnpr2 kubernetes.io/dockerconfigjson 1 8s #<--- default-token-
hkkzr kubernetes.io/service-account-token 3 63d secret-jnpr
	kubernetes.io/dockerconfigjson 1 26m
$ kubectl get secrets secret-jnpr2 -o yaml 
apiVersion: v1
data:
.dockerconfigjson: ewoJImF1dGhzIjoIlNrNVFVaTFHYVdWc1pGVnpaWEl5TVRNNlEweEtaREpxY0UxelZtTTVlbkpCZ 
FZSR1VHND0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE4LjAzLjE 
tY2UgKGxpbnV4KSIKCX0sCgkiZGV0YWNoS2V5cyI6ICJjdHJsLUAiCn0=
kind: Secret 
metadata:
creationTimestamp: 2019-08-15T07:35:25Z 
name: csrx-secret-dr2
namespace: ns-user-1 
resourceVersion: "878490"
selfLink: /api/v1/namespaces/ns-user-1/secrets/secret-jnpr2 
uid: 3efc3bd8-bf2f-11e9-bb2a-0050569e6cfc
type: kubernetes.io/dockerconfigjson

$ kubectl get secret secret-jnpr2 --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode
{
......
			"auths": { "hub.juniper.net": {
					"auth": "Sj5QUi1GaWVsZFVzZXIyMTM6Q0xKZDJqcE1zVmM5enJBdVRGUG4="
			}
					},
......
}

YAML File

You can also create a secret directly from a YAML file the same way you create other objects like service or ingress.

To manually encode the content of the .docker/config.json file:

$ cat .docker/config.json | base64 
ewoJImF1dGhzIjogewoJCSJodWIuanVuaXBlci5uZXQiOiB7CgkJCSJhdXRoIjogIlNrNVFVaTFH 
YVdWc1pGVnpaWEl5TVRNNlEweEtaREpxY0UxelZtTTVlbkpCZFZSR1VHND0iCgkJfQoJfSwKCSJI 
dHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE4LjAzLjEtY2Ug 
KGxpbnV4KSIKCX0sCgkiZGV0YWNoS2V5cyI6ICJjdUAiCn0=

Then put the base64 encoded value of the .docker/config.json file as data in below the YAML file:

#secret-jnpr.yaml 
apiVersion: v1 
kind: Secret
type: kubernetes.io/dockerconfigjson 
metadata:
	name: secret-jnpr3 
	namespace: ns-user-1 
data:
.dockerconfigjson: ewoJImF1dGhzIjogewoJCSJodW......
$ kubectl apply -f secret-jnpr.yaml 
secret/secret-jnpr3 created
 

$ kubectl get secrets
NAME	TYPE	DATA AGE
default-token-hkkzr   kubernetes.io/service-account-token   3		64d
secret-jnpr1	kubernetes.io/dockerconfigjson	1		9s
secret-jnpr2	kubernetes.io/dockerconfigjson	1		6m12s
secret-jnpr3	kubernetes.io/dockerconfigjson	1		78s

Keep in mind that base64 is all about encoding instead of encryption – it is considered the same as plain text. So sharing this file compromises the secret.

Refer Secret in Pod

After a secret is created, it can be referred to by a pod/rc or deployment in order to pull an image from the private registry. There are many ways to refer to secrets. This section will examine using imagePullSecrets under pod spec to refer to the secret.

An imagePullSecret is a way to pass a secret that contains a Docker (or other) image registry password to the kubelet so it can pull a private image on behalf of your pod.

Create a pod pulling the Juniper cSRX container from the private repository:

apiVersion: v1 
kind: Pod 
metadata:
	name: csrx-jnpr 
labels:
app: csrx 
annotations:
k8s.v1.cni.cncf.io/networks: '[
{ "name": "vn-left-1" },
{ "name": "vn-right-1" }
]'
spec:
containers:
#- name: csrx 
# image: csrx
-	name: csrx
image: hub.juniper.net/security/csrx:18.1R1.9 
ports:
-	containerPort: 22 
#imagePullPolicy: Never 
imagePullPolicy: IfNotPresent 
stdin: true
tty: true 
securityContext:
privileged: true 
imagePullSecrets:
-	name: secret-jnpr

Now, generate the pod:

$ kubectl apply -f csrx/csrx-with-secret.yaml 
pod/csrx-jnpr created

The cSRX is up and running:

$ kubectl get pod
NAME	READY STATUS RESTARTS AGE
csrx-jnpr		1/1     Running   0		20h

And behind the scenes, the pod authenticates itself towards the private registry, pulls the image, and launches the cSRX container:

$ kubectl describe pod csrx
......
Events:
19h Normal Scheduled Pod Successfully assigned ns-user-1/csrx to cent333
19h Normal Pulling Pod pulling image "hub.juniper.net/security/csrx:18.1R1.9"
19h Normal Pulled Pod  Successfully pulled image "hub.juniper.net/security/csrx:18.1R1.9" 
19h Normal Created Pod Created container
19h Normal Started Pod Started container

As you saw from our test, the secret objects are created independently of the pods, and inspecting the object spec does not provide the sensitive information directly on the screen.

Secrets are not written to the disk, but are instead stored in a tmpfs file system, only on nodes that need them. Also, secrets are deleted when the pod that is dependent on them is deleted.

On most native Kubernetes distributions, communication between users and the API server is protected by SSL/TLS. Therefore, secrets transmitted over these channels are properly protected.

Any given pod does not have access to the secrets used by another pod, which facilitates encapsulation of sensitive data across different pods. Each container in a pod has to request a secret volume in its volumeMounts for it to be visible inside the container. This feature can be used to construct security partitions at the pod level.