docker registry搭建私有仓库
- Docker Registry官网:
-
搭建docker私有仓库:
- Docker入门-搭建docker私有仓库:https://zhuanlan.zhihu.com/p/78543733
- 一文搞定docker创建私有镜像仓库:https://blog.csdn.net/phoenix339/article/details/93979152
- 部署 docker 官方 registry 私有镜像仓库和镜像加速仓库: https://www.ioiox.com/archives/127.html
- Docker Registry v2 配置文件详解: https://cloud.tencent.com/developer/article/1047253
公有镜像仓库
仓库(Repository)是集中存放镜像的地方。
- Docker Hub: Docker官方维护的公共仓库,https://hub.docker.com/
# 可以在https://hub.docker.com 免费注册一个Docker账号 docker login docker search ubuntu docker pull ubuntu:22.04 # 将自己镜像推送到docker hub docker push <Local IMAGE ID> xstar/ubuntu:2204v1
搭建Harbor镜像仓库
Harbor: 由VMware公司开源的企业级的Docker Registry管理项目,它包括权限管理(RBAC)、LDAP、日志审核、管理界面、自我注册、镜像复制和中文支持等功能。
- https://goharbor.io/docs/2.10.0/
- https://github.com/goharbor/harbor/
- https://zhuanlan.zhihu.com/p/143779176
- trivyDB漏洞库:
由于ghcr.io/aquasecurity/trivy-db:2下载一直失败,最后用oras下载才成功下载。
harbor-trivy镜像扫描工具安装部署(离线漏洞库):https://blog.csdn.net/weixin_47055136/article/details/132423901
-
安装oras
-
下载地址:https://github.com/oras-project/oras/releases
tar -zxf oras_1.0.1_linux_amd64.tar.gz mv oras /usr/local/bin/ chown root. /usr/local/bin/oras oras version Version: 1.1.0 Go version: go1.21.0 Git commit: 7079c468a06fb5815c99395eb4aaf46dd459d3fa Git tree state: clean
-
下载地址:https://github.com/oras-project/oras/releases
-
漏洞库下载
-
trivy v1版本和V2版本漏洞库下载地址不一样
# trivy v2 oras pull ghcr.io/aquasecurity/trivy-db:2 oras pull ghcr.io/aquasecurity/trivy-java-db:1 # trivy v1 https://github.com/aquasecurity/trivy-db/releases oras pull ghcr.io/aquasecurity/trivy-java-db:1
-
trivy v1版本和V2版本漏洞库下载地址不一样
-
trivy设置离线配置:harbor.yml
trivy: #跳过更新 skip_update: true #离线扫描 offline_scan: true
-
设置后重新install harbor
—————————— # 停止harbor docker-compose down # 安装trivy ./install.sh --with-trivy # 启动harbor服务 docker-compose up -d
-
拷贝trivy-db到harbor-trivy数据目录:data_volume: /data/harbor/harborData
tar -zxvf db.tar.gz -C /data/harbor/harborData/trivy-adapter/trivy/db chown 10000:10000 /data/harbor/harborData/trivy-adapter/trivy/db/* tar -zxvf javadb.tar.gz -C /data/harbor/harborData/trivy-adapter/trivy/java-db chown 10000:10000 /data/harbor/harborData/trivy-adapter/trivy/java-db/*
搭建Registry镜像仓库
docker-registry是官方提供的工具,可以用于构建私有的镜像仓库。
可以通过获取官方registry镜像来运行。默认情况下,仓库会被创建在容器的/var/lib/registry目录下。可以通过-v参数来将镜像文件存放在本地的指定路径。
用docker运行registry
-
本地存储目录、配置文件
mkdir -p /data/registry
config.yml version: 0.1 log: fields: service: registry storage: delete: enable: true cache: blobdescriptor: inmemory filesystem: rootdirectory: /var/lib/registry http: addr: :5000 headers: X-Content-Type-Options: [nosniff] health: storagedriver: enabled: true interval: 10s threshold: 3
-
运行registry
docker run -d \ --restart=always \ --name registry \ -p 5000:5000 \ -v /data/registry:/var/lib/registry \ -v /data/registry/config.yml:/etc/docker/registry/config.yml \ registry:2.8.3
添加HTTPS自签证书
-
目录及文件
/data/registry/ ├── auth │ └── htpasswd ├── certs │ ├── registry.crt │ └── registry.key ├── config.yml └── docker └── registry
-
生成证书,Common Name要输入我们registry的域名,生成的证书只对该域名有效。其他的可以任意填。
cd /data/registry mkdir -p certs #openssl req \ # -newkey rsa:4096 -nodes -sha256 -keyout certs/registry.key \ # -addext "subjectAltName = DNS:myregistry.domain.com" \ # -x509 -days 365 -out certs/registry.crt openssl req -newkey rsa:4096 -nodes -sha256 \ -keyout certs/registry.key \ -x509 -days 365 -out certs/registry.crt
Country Name (2 letter code) [AU]:cn State or Province Name (full name) [Some-State]:sc Locality Name (eg, city) []:chengdu Organization Name (eg, company) [Internet Widgits Pty Ltd]:home Organizational Unit Name (eg, section) []:home Common Name (e.g. server FQDN or YOUR name) []:192.168.31.40 Email Address []:xstar@qq.com
-
生成http鉴权密码文件
# 安装htpasswd工具生成 apt install apache2-utils cd /data/registry mkdir auth htpasswd -Bbn myuser mypassword >> auth/htpasswd # -n Don't update file; display results on stdout. # -b Use the password from the command line rather than prompting for it. # -B Force bcrypt encryption of the password (very secure). # Registry基本身份验证仅支持 htpasswd 凭据 bcrypt 格式。 # 用docker容器生成 #mkdir -p auth #docker run --entrypoint htpasswd \ # httpd:2 -Bbn testuser testpassword > auth/htpasswd
-
重新启动registory(需先将之前的容器删除)
docker run -d -p 5000:5000 --restart=always --name registry \ -v /data/registry/auth:/auth \ -e "REGISTRY_AUTH=htpasswd" \ -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \ -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \ -v /data/registry/certs:/certs \ -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry.crt \ -e REGISTRY_HTTP_TLS_KEY=/certs/registry.key \ -v /data/registry:/var/lib/registry \ -v /data/registry/config.yml:/etc/docker/registry/config.yml \ registry:2.8.3
私有仓库操作
client设置
- 修改所有client和server的/etc/hosts,将私有镜像仓库的ip-域名对应关系加进去。
docker client不安装证书的话,进行pull/push操作,会出现x509: certificate signed by unknown authority的报错。
-
每个 Docker 守护程序设置信任该证书(无需重启 Docker)
# 将registry的证书文件复制到仓库对应目录 mkdir -p /etc/docker/certs.d/192.168.31.40:5000 cp /home/yuanxing/registry.crt /etc/docker/certs.d/192.168.31.40:5000 systemctl restart docker
-
在要访问这个Registry的所有主机上,将私有镜像配置到docker:/etc/docker/daemon.json。
-
修改后重启docker生效:
sudo systemctl restart docker
(重启docker所有容器都会重启,应该尽可能用redload)。 -
insecure-registries启动不安全的Registry后,Docker的处理步骤:
-
首先,尝试使用HTTPS。
- 如果HTTPS可用但证书无效,忽略证书验证。
- 如果HTTPS不可用,退回使用HTTP。
-
首先,尝试使用HTTPS。
-
修改后重启docker生效:
{ "registry-mirrors": [ "https://dockerproxy.com", "https://hub-mirror.c.163.com", "https://mirror.baidubce.com", "https://ccr.ccs.tencentyun.com" ], "dns": [ "114.114.114.114","61.139.2.69" ], "insecure-registries": [ "192.168.31.40:5000","192.168.31.40" ] }
docker login
不登录直接push镜像的话会失败,提示no basic auth credentials的错误。
docker login -u myuser -p mypassword 192.168.31.40:5000
WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded
docker pull/push
-
在内网其他节点测试pull(先将私有镜像设置添加到各个节点的daemon.json中,并relad docker生效)
docker pull 192.168.31.40:5000/xstar/ubuntu:22.04v1
-
push image到私有仓库
# 镜像推送到私有仓库 # 将镜像标记为私有仓库镜像 # docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG] # docker tag SOURCE_IMAGE[:TAG] [REGISTRY_HOST[:REGISTRY_PORT]/]TAGGET_IMAGE[:TAG] # docker tag xstar/ubuntu:22.04v1 192.168.31.40:5000/xstar/ubuntu:22.04v1 docker tag xstar/ubuntu:22.04v1 192.168.31.40/xstar/ubuntu:22.04v1 docker images #REPOSITORY TAG IMAGE ID CREATED SIZE #127.0.0.1:5000/xstar/ubuntu 22.04v1 4641f2e373d7 2 hours ago 138MB #xstar/ubuntu 22.04v1 4641f2e373d7 2 hours ago 138MB # 上传标记的镜像 #docker push 192.168.31.40:5000/xstar/ubuntu:22.04v1 docker push 192.168.31.40/xstar/ubuntu:22.04v1 # 用curl查看仓库中的镜像 curl -XGET http://192.168.31.40:5000/v2/_catalog # {"repositories":["xstar/ubuntu"]} # 删除本地镜像,测试从仓库下载 docker image rm 192.168.31.40:5000/xstar/ubuntu:22.04v1 docker pull 192.168.31.40:5000/xstar/ubuntu:22.04v1
查询私有仓库镜像
目前无法用docker search/images/image ls对私有仓库镜像进行查询,可以通过curl调用API列出仓库内所有的镜像。
-
curl查询
# -k, --insecure:自签证书需忽略证书校验 # -u,登录用户、密码 curl -k -u 'myuser:mypassword' -XGET https://192.168.31.40:5000/v2/_catalog curl -k -u 'myuser:mypassword' -XGET https://192.168.31.40:5000/v2/xstar/ubuntu/tags/list curl -s -k -u 'myuser:mypassword' -XGET https://192.168.31.40:5000/v2/_catalog | python3 -m json.tool
curl -k -u 'myuser:mypassword' -XGET https://192.168.31.40:5000/v2/_catalog {"repositories":["xstar/ubuntu"]} curl -k -u 'myuser:mypassword' -XGET https://192.168.31.40:5000/v2/xstar/ubuntu/tags/list {"name":"xstar/ubuntu","tags":["22.04v1"]}
-
list_private_images.py
#!/usr/bin/env python3 #-*- coding:utf-8 -*- # list_private_images.py import requests from requests.auth import HTTPBasicAuth import json import traceback import sys import urllib3 urllib3.disable_warnings() def getImagesNames(server,user,passwd): docker_images = [] try: url = server + "/v2/_catalog" res =requests.get(url=url,verify=False,auth=HTTPBasicAuth(user,passwd)).content.strip() res_dic = json.loads(res) images_type = res_dic['repositories'] for i in images_type: url2 = server + "/v2/" + str(i) + "/tags/list" res2 =requests.get(url=url2,verify=False,auth=HTTPBasicAuth(user,passwd)).content.strip() res_dic2 = json.loads(res2) name = res_dic2['name'] tags = res_dic2['tags'] if tags != None: for tag in tags: docker_name = server + "/" + name + ":" + tag docker_images.append(docker_name) print(docker_name) except: traceback.print_exc() return docker_images if __name__ == "__main__": if len(sys.argv)<4: print("Usage: list_private_images.py <server_url> <username> <password>") print("eg. list_private_images.py https://192.168.31.40:5000 myuser mypassword") sys.exit() server = sys.argv[1] user = sys.argv[2] passwd = sys.argv[3] getImagesNames(server,user,passwd)
cp list_private_images.py /usr/local/bin chmod 755 /usr/local/bin/list_private_images.py list_private_images.py https://192.168.31.40:5000 myuser mypassword #https://192.168.31.40:5000/xstar/ubuntu:22.04v1
管理镜像
- 确认是否开启删除功能
如果没有开启,执行删除镜像操作的时候,会返回如下两种错误:
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
HTTP/1.1 405 Method Not Allowed Content-Type: application/json; charset=utf-8 Docker-Distribution-Api-Version: registry/2.0 X-Content-Type-Options: nosniff Date: Fri, 18 Mar 2022 04:12:22 GMT Content-Length: 78
-
修改registry的config.yml文件,在storage中增加,修改重启容器:
docker restart registry
delete: enabled: true
-
查询镜像RepoDigests
docker image inspect -f '{{.RepoDigests}}' 192.168.31.40:5000/xstar/ubuntu:22.04v1 # [192.168.31.40:5000/xstar/ubuntu@sha256:b840318b448f3105284eda4241967055e2cd463a3742d1a55fb917eca5e84319] # curl方式查看镜像RepoDigests #curl --header "Accept:application/vnd.docker.distribution.manifest.v2+json" -I \ #-u <仓库用户名>:<用户名密码> http://<仓库ip地址>:<仓库端口>/v2/<镜像名称>/manifests/<镜像 tag> curl --header "Accept:application/vnd.docker.distribution.manifest.v2+json" -I \ -k -u 'myuser:mypassword' \ https://192.168.31.40:5000/v2/xstar/ubuntu/manifests/22.04v1 # docker-content-digest: sha256:b840318b448f3105284eda4241967055e2cd463a3742d1a55fb917eca5e84319
-
删除私有库中镜像
# [[curl]] -I -XDELETE -u <仓库用户名>:<用户名密码> http://<仓库ip地址>:<仓库端口>/v2/<镜像名称>/manifests/<获取的 hash 值> # 查看registry日志 # docker logs -f registry curl \ -k -u 'myuser:mypassword' \ -I -X DELETE \ https://192.168.31.40:5000/v2/xstar/ubuntu/manifests/\ sha256:b840318b448f3105284eda4241967055e2cd463a3742d1a55fb917eca5e84319
HTTP/2 202 docker-distribution-api-version: registry/2.0 x-content-type-options: nosniff content-length: 0 date: Mon, 15 Jan 2024 14:42:52 GMT # 删除后,curl仍能查询到镜像信息,但tags为null,下载也会失败 curl -k -u 'myuser:mypassword' -XGET https://192.168.31.40:5000/v2/_catalog {"repositories":["xstar/ubuntu"]} curl -k -u 'myuser:mypassword' -XGET https://192.168.31.40:5000/v2/xstar/ubuntu/tags/list {"name":"xstar/ubuntu","tags":null} docker pull 192.168.31.40:5000/xstar/ubuntu:22.04v1 Error response from daemon: manifest for 192.168.31.40:5000/xstar/ubuntu:22.04v1 not found: manifest unknown: manifest unknown
-
垃圾回收
-
镜像删除后,并不会释放空间,需要在registry上执行垃圾回收
registry garbage-collect /etc/docker/registry/config.yml
-
镜像删除后,并不会释放空间,需要在registry上执行垃圾回收
将Registry部署到Swarm
create service方式
如果使用分布式存储(如:Amazon S3),则可使用完全扩展的多节点Registry服务。
如果使用本地挂载的卷,只能使用单副本服务和Swarm节点约束,来确保只有一个工作节点副本写入。
-
将TLS证书和密钥另存为docker secret:
# docker secret create [OPTIONS] SECRET [file|-] # Create a secret from a file or STDIN as content # Options: # -d, --driver string Secret driver # -l, --label list Secret labels # --template-driver string Template driver docker secret create docker_secret_registry.crt certs/registry.crt docker secret create docker_secret_registry.key certs/registry.key docker secret ls #ID NAME DRIVER CREATED UPDATED #vlpkfqo7aljlbhk437v6cpn1m docker_secret_registry.crt About a minute ago About a minute ago #q5shlbwh86ygibx5092j4u70w docker_secret_registry.key About a minute ago About a minute ago
-
为节点添加registry Lalel
#docker node update --label-add registry=true swarm-node1 #docker node update --label-rm registry swarm-node1 docker node update --label-add registry=true swarm-manager docker node inspect -f '{{.Spec.Labels}}' swarm-manager
-
创建Registry服务
- 授予两个secret的访问权限
- 仅在带有registry标签的node上运行
-
一次只能运行一个副本
docker service create \ --name registry \ --restart-condition="any" \ --secret docker_secret_registry.crt \ --secret docker_secret_registry.key \ --constraint 'node.labels.registry==true' \ --mount type=bind,src=/data/registry,dst=/var/lib/registry \ --mount type=bind,src=/data/registry/auth,dst=/auth \ --mount type=bind,src=/data/registry/config.yml,dst=/etc/docker/registry/config.yml \ -e "REGISTRY_AUTH=htpasswd" \ -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \ -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \ -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \ -e REGISTRY_HTTP_TLS_CERTIFICATE=/run/secrets/docker_secret_registry.crt \ -e REGISTRY_HTTP_TLS_KEY=/run/secrets/docker_secret_registry.key \ --publish published=5000,target=5000 \ --replicas 1 \ registry:2.8.3
compose方式
Compose模式不支持external secret,且只能运行在当前节点,建议使用Stack模式。
-
compose部署Registry
# Registry-docker-compose.yml version: '3' services: registry: restart: always image: registry:2.8.3 ports: - 5000:5000 environment: REGISTRY_AUTH: htpasswd REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd REGISTRY_HTTP_ADDR: 0.0.0.0:5000 REGISTRY_HTTP_TLS_CERTIFICATE: /run/secrets/docker_secret_registry.crt REGISTRY_HTTP_TLS_KEY: /run/secrets/docker_secret_registry.key volumes: - /data/registry:/var/lib/registry - /data/registry/auth:/auth - /data/registry/config.yml:/etc/docker/registry/config.yml secrets: - docker_secret_registry.crt - docker_secret_registry.key deploy: replicas: 1 placement: constraints: - "node.labels.registry==true" secrets: docker_secret_registry.crt: file: /data/registry/certs/registry.crt #external: true # compose下不支持external: true docker_secret_registry.key: file: /data/registry/certs/registry.key
-
compose启动Registry
docker-compose -f Registry-docker-compose.yml up -d
WARNING: The Docker Engine you're using is running in swarm mode. Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node. To deploy your application across the swarm, use `docker stack deploy`. Creating registry_registry_1 ... done
stack方式
-
stack部署Registry
# Registry-docker-stack.yml version: '3' services: registry: image: registry:2.8.3 ports: - 5000:5000 environment: REGISTRY_AUTH: htpasswd REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd REGISTRY_HTTP_ADDR: 0.0.0.0:5000 REGISTRY_HTTP_TLS_CERTIFICATE: /run/secrets/docker_secret_registry.crt REGISTRY_HTTP_TLS_KEY: /run/secrets/docker_secret_registry.key volumes: - /data/registry:/var/lib/registry - /data/registry/auth:/auth - /data/registry/config.yml:/etc/docker/registry/config.yml secrets: - docker_secret_registry.crt - docker_secret_registry.key deploy: replicas: 1 restart_policy: condition: on-failure placement: constraints: - "node.labels.registry==true" secrets: docker_secret_registry.crt: external: true docker_secret_registry.key: external: true
-
docker stack启动Registry
docker stack deploy -c Registry-docker-stack.yml Registry
docker stack deploy -c Registry-docker-stack.yml Registry Creating network Registry_default Creating service Registry_registry docker service ls ID NAME MODE REPLICAS IMAGE PORTS inygepp4z78i Registry_registry replicated 1/1 registry:2 *:5000->5000/tcp docker service ps Registry_registry ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS qsiqlrrd0n9j Registry_registry.1 registry:2 swarm-manager Running Running about a minute ago curl -k -u 'myuser:mypassword' -XGET https://192.168.31.40:5000/v2/xstar/ubuntu/tags/list {"name":"xstar/ubuntu","tags":["22.04v1"]}
stack+NFS共享
由于依赖节点上的本地数据存储,Registry只能运行在一个指定的节点,无法故障切换、或者负载均衡。
考虑使用NFS作为基本的网络存储,实现Registry在swarm集群中多节点运行和切换。
准备NFS Volume服务
-
参考本文前面部份,完成/data/registry目录下的:config.yml、certs和auth文件的准备。
/data/registry/ ├── auth │ └── htpasswd ├── certs │ ├── registry.crt │ └── registry.key ├── config.yml └── docker # registry自动创建的数据目录 └── registry
-
首先参看docker-swarm中的NFS Volume章节,并完成:
- NFS Server部署、配置,设置NFS registry目录共享。
- 在所有node安装nfs-common,确认在NFS客户端可以正常mount和读写。
registry stack配置
-
stack部署Registry:NFS存储+configs+secrets
# Registry-docker-stack_v2.yml version: '3' services: registry: image: registry:2.8.3 ports: - 5000:5000 environment: REGISTRY_AUTH: htpasswd REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm REGISTRY_AUTH_HTPASSWD_PATH: /var/lib/registry/auth/htpasswd REGISTRY_HTTP_ADDR: 0.0.0.0:5000 REGISTRY_HTTP_TLS_CERTIFICATE: /run/secrets/docker_secret_registry.crt REGISTRY_HTTP_TLS_KEY: /run/secrets/docker_secret_registry.key volumes: - type: volume source: registry_storage target: /var/lib/registry configs: - source: registry_config_yml target: /etc/docker/registry/config.yml secrets: - docker_secret_registry.crt - docker_secret_registry.key deploy: replicas: 1 restart_policy: condition: on-failure placement: constraints: - "node.labels.registry==true" secrets: docker_secret_registry.crt: external: true docker_secret_registry.key: external: true volumes: registry_storage: driver_opts: type: "nfs" o: "addr=192.168.31.40,nolock,soft,rw" device: ":/data/registry" configs: registry_config_yml: file: /data/registry/config.yml
-
为更多节点增加label
#docker node update --label-rm registry swarm-node1 docker node update --label-add registry=true swarm-manager docker node update --label-add registry=true swarm-node1 docker node update --label-add registry=true swarm-node2 docker node inspect -f '{{.Spec.Labels}}' swarm-manager swarm-node1 swarm-node2 #map[registry:true] #map[registry:true] #map[registry:true]
Stack部署Registry
docker stack启动Registry
docker stack deploy -c Registry-docker-stack_v2.yml Registry
service部署时,自动创建了network、config和volume。 docker stack deploy -c Registry-docker-stack_v2.yml Registry Creating network Registry_default Creating config Registry_registry_config_yml Creating service Registry_registry docker stack ls NAME SERVICES Registry 1 docker stack ps Registry ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS swfcczvkl8zb Registry_registry.1 registry:2.8.3 swarm-manager Running Running 3 minutes ago ozxm1c2gqifs Registry_registry.2 registry:2.8.3 swarm-node2 Running Running 3 minutes ago 在运行的节点,用 docker volume ls和docker config ls可以查看到对应的卷和配置修改,用mount|grep regi可以看到对应NFS目录已挂载。 测试访问registry正常: curl -k -u 'myuser:mypassword' -XGET https://192.168.31.42:5000/v2/xstar/ubuntu/tags/list {"name":"xstar/ubuntu","tags":["22.04v1"]}