In today's era of containerization, no matter what container we are using we need an image to run the container. Docker images are stored on container registries like Docker hub(cloud), Google Container Registry(GCR), AWS ECR, quay.io etc.
We can also self-host docker registry on any docker platform. In this blog post, we will see how to deploy docker registry on kubernetes using storage driver S3.
Pre-requisite:
-
Access to working kubernetes cluster.
-
Understanding of Kubernetes terms like pods, deployments, services, configmap and ingress.
As per docker registry documentation, We can simply start the registry using docker image registry.
Basic parameters when deploying production registry are:
- Authentication
- SSL
- Storage
We will use htpasswd authentication for this post though registry image supports silly and token based authentication as well.
Docker registry requires applications to use SSL certificate and key in the registry. We will use kubernetes service, which terminates SSL on ELB level using annotations.
For registry storage, we can use filesystem, s3, azure, swift etc. For the complete list of options please visit docker site site.
We need to store the docker images pushed to the registry. We will use S3 to store these docker images.
Steps for deploying registry on kubernetes.
Get the ARN of the SSL certificate to be used for SSL.
If you don't have SSL on AWS IAM, upload it using the following command.
1$aws iam upload-server-certificate --server-certificate-name registry --certificate-body file://registry.crt --private-key file://key.pem 2
Get the arn for the certificate using the command.
1$aws iam get-server-certificate --server-certificate-name registry | grep Arn 2
Create S3 bucket which will be used to store docker images using s3cmd or aws s3.
1 2$s3cmd mb s3://myregistry 3Bucket 's3://myregistry/' created 4
Create a separate namespace, configmap, deployment and service for registry using following templates.
1--- 2apiVersion: v1 3kind: Namespace 4metadata: 5name: container-registry 6 7--- 8apiVersion: v1 9kind: ConfigMap 10metadata: 11 name: auth 12 namespace: container-registry 13data: 14 htpasswd: | 15 admin:$2y$05$TpZPzI7U7cr3cipe6jrOPe0bqohiwgEerEB6E4bFLsUf7Bk.SEBRi 16 17--- 18apiVersion: extensions/v1beta1 19kind: Deployment 20metadata: 21 labels: 22 app: registry 23 name: registry 24 namespace: container-registry 25spec: 26 replicas: 1 27 strategy: 28 type: RollingUpdate 29 template: 30 metadata: 31 labels: 32 app: registry 33 spec: 34 containers: 35 - env: 36 - name: REGISTRY_AUTH 37 value: htpasswd 38 - name: REGISTRY_AUTH_HTPASSWD_PATH 39 value: /auth/htpasswd 40 - name: REGISTRY_AUTH_HTPASSWD_REALM 41 value: Registry Realm 42 - name: REGISTRY_STORAGE 43 value: s3 44 - name: REGISTRY_STORAGE_S3_ACCESSKEY 45 value: <your-s3-access-key> 46 - name: REGISTRY_STORAGE_S3_BUCKET 47 value: <your-registry-bucket> 48 - name: REGISTRY_STORAGE_S3_REGION 49 value: us-east-1 50 - name: REGISTRY_STORAGE_S3_SECRETKEY 51 value: <your-secret-s3-key> 52 image: registry:2 53 name: registry 54 ports: 55 - containerPort: 5000 56 volumeMounts: 57 - name: auth 58 mountPath: /auth 59 volumes: 60 - name: auth 61 configMap: 62 name: auth 63--- 64apiVersion: v1 65kind: Service 66metadata: 67 annotations: 68 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <your-iam-certificate-arn> 69 service.beta.kubernetes.io/aws-load-balancer-instance-protocol: http 70 service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" 71 labels: 72 app: registry 73 name: registry 74 namespace: container-registry 75spec: 76 ports: 77 - name: "443" 78 port: 443 79 targetPort: 5000 80 selector: 81 app: registry 82type: LoadBalancer
Let's launch this manifest using kubectl apply.
1 2kubectl apply -f registry-namespace.yml registry-configmap.yml registry-deployment.yaml registry-namespace.yml 3namespace "registry" created 4configmap "auth" created 5deployment "registry" created 6service "registry" created 7
Now that we have created registry, we should map DNS to web service ELB endpoint. We can get the webservice ELB endpoint using the following command.
1 2$kubectl -n registry get svc registry -o wide 3 4NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR 5registry 100.71.250.56 abcghccf8540698e8bff782799ca8h04-1234567890.us-east-2.elb.amazonaws.com 443:30494/TCP 1h app=registry 6
We will point DNS to this ELB endpoint with domain registry.myapp.com
Once we have registry running, now it's time to push the image to a registry.
First, pull the image or build the image locally to push.
On local machine run following commands:
1 2$docker pull busybox 3latest: Pulling from busybox 4f9ea5e501ad7: Pull complete 5ac3f08b78d4e: Pull complete 6Digest: sha256:da268b65d710e5ca91271f161d0ff078dc63930bbd6baac88d21b20d23b427ec 7Status: Downloaded newer image for busybox:latest 8
Now login to our registry using the following commands.
1 2$ sudo docker login registry.myapp.com 3 4Username: admin 5 6Password: 7 8Login Succeeded 9
Now tag the image to point it to our registry using docker tag command
1$ sudo docker tag busybox registry.myapp.com/my-app:latest 2
Once the image is tagged we are good to push.
Using the docker push command let's push the image.
1 2$ sudo docker push docker.gocloudlogistics.com/my-app:latest 3 4The push refers to a repository [registry.myapp.com/my-app] 5 605732a3f47b5: Pushed 730de36c4bd15: Pushed 85237590c0d08: Pushed 9latest: digest: sha256:f112e608b2639b21498bd4dbca9076d378cc216a80d52287f7f0f6ea6ad739ab size: 205 10
We are successfully able to push image to registry running on kunbernetes and stored on S3. Let's verify if it exists on S3.
Navigate to our s3 bucket and we can see the docker registry repository busybox has been created.
1$ s3cmd ls s3://myregistry/docker/registry/repositories/ 2DIR s3://myregistry/docker/registry/repositories/busybox/ 3
All our image related files are stored on S3.
In this way, we self-host container registry on kubernetes backed by s3 storage.