Deploying Docker Registry on Kubernetes using S3 Storage

Rahul Mahale

By Rahul Mahale

on May 3, 2018

In today's era of containerization, no matter what container we are using we need an image to run the container. Docker images are stored on container registries like Docker hub(cloud), Google Container Registry(GCR), AWS ECR, quay.io etc.

We can also self-host docker registry on any docker platform. In this blog post, we will see how to deploy docker registry on kubernetes using storage driver S3.

Pre-requisite:

As per docker registry documentation, We can simply start the registry using docker image registry.

Basic parameters when deploying production registry are:

  • Authentication
  • SSL
  • Storage

We will use htpasswd authentication for this post though registry image supports silly and token based authentication as well.

Docker registry requires applications to use SSL certificate and key in the registry. We will use kubernetes service, which terminates SSL on ELB level using annotations.

For registry storage, we can use filesystem, s3, azure, swift etc. For the complete list of options please visit docker site site.

We need to store the docker images pushed to the registry. We will use S3 to store these docker images.

Steps for deploying registry on kubernetes.

Get the ARN of the SSL certificate to be used for SSL.

If you don't have SSL on AWS IAM, upload it using the following command.

1$aws iam upload-server-certificate --server-certificate-name registry --certificate-body file://registry.crt --private-key file://key.pem
2

Get the arn for the certificate using the command.

1$aws iam get-server-certificate --server-certificate-name registry  | grep Arn
2

Create S3 bucket which will be used to store docker images using s3cmd or aws s3.

1
2$s3cmd mb s3://myregistry
3Bucket 's3://myregistry/' created
4

Create a separate namespace, configmap, deployment and service for registry using following templates.

1---
2apiVersion: v1
3kind: Namespace
4metadata:
5name: container-registry
6
7---
8apiVersion: v1
9kind: ConfigMap
10metadata:
11  name: auth
12  namespace: container-registry
13data:
14  htpasswd: |
15    admin:$2y$05$TpZPzI7U7cr3cipe6jrOPe0bqohiwgEerEB6E4bFLsUf7Bk.SEBRi
16
17---
18apiVersion: extensions/v1beta1
19kind: Deployment
20metadata:
21  labels:
22    app: registry
23  name: registry
24  namespace: container-registry
25spec:
26  replicas: 1
27  strategy:
28    type: RollingUpdate
29  template:
30    metadata:
31      labels:
32        app: registry
33    spec:
34      containers:
35        - env:
36            - name: REGISTRY_AUTH
37              value: htpasswd
38            - name: REGISTRY_AUTH_HTPASSWD_PATH
39              value: /auth/htpasswd
40            - name: REGISTRY_AUTH_HTPASSWD_REALM
41              value: Registry Realm
42            - name: REGISTRY_STORAGE
43              value: s3
44            - name: REGISTRY_STORAGE_S3_ACCESSKEY
45              value: <your-s3-access-key>
46            - name: REGISTRY_STORAGE_S3_BUCKET
47              value: <your-registry-bucket>
48            - name: REGISTRY_STORAGE_S3_REGION
49              value: us-east-1
50            - name: REGISTRY_STORAGE_S3_SECRETKEY
51              value: <your-secret-s3-key>
52          image: registry:2
53          name: registry
54          ports:
55            - containerPort: 5000
56          volumeMounts:
57            - name: auth
58              mountPath: /auth
59      volumes:
60        - name: auth
61          configMap:
62            name: auth
63---
64apiVersion: v1
65kind: Service
66metadata:
67  annotations:
68    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <your-iam-certificate-arn>
69    service.beta.kubernetes.io/aws-load-balancer-instance-protocol: http
70    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
71  labels:
72    app: registry
73  name: registry
74  namespace: container-registry
75spec:
76  ports:
77    - name: "443"
78      port: 443
79      targetPort: 5000
80  selector:
81    app: registry
82type: LoadBalancer

Let's launch this manifest using kubectl apply.

1
2kubectl apply -f registry-namespace.yml registry-configmap.yml registry-deployment.yaml registry-namespace.yml
3namespace "registry" created
4configmap "auth" created
5deployment "registry" created
6service "registry" created
7

Now that we have created registry, we should map DNS to web service ELB endpoint. We can get the webservice ELB endpoint using the following command.

1
2$kubectl -n registry get svc registry -o wide
3
4NAME       CLUSTER-IP      EXTERNAL-IP                                                               PORT(S)         AGE       SELECTOR
5registry   100.71.250.56   abcghccf8540698e8bff782799ca8h04-1234567890.us-east-2.elb.amazonaws.com   443:30494/TCP   1h       app=registry
6

We will point DNS to this ELB endpoint with domain registry.myapp.com

Once we have registry running, now it's time to push the image to a registry.

First, pull the image or build the image locally to push.

On local machine run following commands:

1
2$docker pull busybox
3latest: Pulling from busybox
4f9ea5e501ad7: Pull complete
5ac3f08b78d4e: Pull complete
6Digest: sha256:da268b65d710e5ca91271f161d0ff078dc63930bbd6baac88d21b20d23b427ec
7Status: Downloaded newer image for busybox:latest
8

Now login to our registry using the following commands.

1
2$ sudo docker login registry.myapp.com
3
4Username: admin
5
6Password:
7
8Login Succeeded
9

Now tag the image to point it to our registry using docker tag command

1$ sudo docker tag busybox registry.myapp.com/my-app:latest
2

Once the image is tagged we are good to push.

Using the docker push command let's push the image.

1
2$ sudo docker push docker.gocloudlogistics.com/my-app:latest
3
4The push refers to a repository [registry.myapp.com/my-app]
5
605732a3f47b5: Pushed
730de36c4bd15: Pushed
85237590c0d08: Pushed
9latest: digest: sha256:f112e608b2639b21498bd4dbca9076d378cc216a80d52287f7f0f6ea6ad739ab size: 205
10

We are successfully able to push image to registry running on kunbernetes and stored on S3. Let's verify if it exists on S3.

Navigate to our s3 bucket and we can see the docker registry repository busybox has been created.

1$ s3cmd ls s3://myregistry/docker/registry/repositories/
2DIR   s3://myregistry/docker/registry/repositories/busybox/
3

All our image related files are stored on S3.

In this way, we self-host container registry on kubernetes backed by s3 storage.

Stay up to date with our blogs. Sign up for our newsletter.

We write about Ruby on Rails, ReactJS, React Native, remote work,open source, engineering & design.