What does it mean pull Docker image from ACR in pipelines? It was difficult to find a title for this post.
First, I give you the scenario. In Azure, I have created an Azure Container Repository where I store the Docker containers. In my previous posts, I talked how to deploy a ShinyApp via pipeline in Azure DevOps.
Scenario
So, Azure DevOps has a connection with the Azure Container Repository already. The Azure DevOps pipeline builds the Docker container and pushes it in the Azure Container Repository. Then, the pipeline connects to the virtual server where I host all the applications (in this case ShinyApps) via SSH to pull the new image and run the container.
This solution is working. For your benefit, here you have the files.
azure-pipeline-yml
# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker
trigger:
- main
resources:
- repo: self
variables:
# Container registry service connection established during pipeline creation
dockerRegistryServiceConnection: 'yourconnection'
# Insert a name for your container
imageRepository: 'shinyappdemo'
containerRegistry: $(ACRLoginServer)
dockerfilePath: '$(Build.SourcesDirectory)/DOCKERFILE'
tag: '$(Build.BuildId)'
# Agent VM image name
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Check
condition: eq('${{ variables.imageRepository }}', '')
steps:
- script: |
echo '##[error] The imageRepository must have a value!'
exit 1
- job: Build
condition: not(eq('${{ variables.imageRepository }}', ''))
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker@2
displayName: Build and push an image to container registry
inputs:
command: buildAndPush
repository: $(imageRepository)
dockerfile: $(dockerfilePath)
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
latest
- task: SSH@0
displayName: 'Run shell commands on remote machine'
inputs:
sshEndpoint: 'ShinyServerDev'
commands: |
echo $(SSHPassword) | sudo -S docker pull $(containerRegistry)/$(imageRepository)
failOnStdErr: false
continueOnError: true
Dockerfile
FROM openanalytics/r-base
ARG project=testApp
# install Debian dependencies for R
RUN apt-get update && apt-get install -y \
sudo \
pandoc \
pandoc-citeproc \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
libssl-dev \
libssh2-1-dev \
libxml2-dev \
libgit2-dev
# packages needed renv and install
RUN R -e "install.packages('shiny', repos='https://cran.rstudio.com')"
RUN R -e "install.packages('shinydashboard', repos='https://cran.rstudio.com')"
RUN R -e "install.packages('shinythemes', repos='https://cran.rstudio.com')"
# create root folder for app in container
RUN mkdir /root/${project}
# copy the app to the image
COPY app /root/${project}
COPY Rprofile.site /usr/lib/R/etc/
EXPOSE 3838
# remember to update this line
CMD ["R", "-e", "shiny::runApp('/root/testApp')"]
So, how you see in the Dockerfile I pull the container from openanalytics/r-base. That mean, every time I have to install all the application I need and the pipeline takes ages to build the containers. For this reason, I decided to have a container to use as base for all my applications.
Then, I created this container and push in the Azure Container Registry (ACR). So, I changed the Dockerfile to pull the container from my ACR. An immediately I get an error
Denied: retrieving permissions failed in Azure DevOps pipeline
Denied: retrieving permissions failed in Azure DevOps pipeline
So, I checked the Service connection in the Project Settings but everything is correct.
I have tried to change the pipeline adding the login to the ACR but it didn’t work. What can I do?
Solution
So, I don’t want to bored you with everything I tried but I spent hours to find the solution. First, this is the Dockerfile that pull the container from the ACR.
Be careful. The file name must be
Dockerfile(no capital letters)
# OS & Base R Set Up
FROM youracr.azurecr.io/myrbase
RUN apt-get update && apt-get install -y libicu-dev make pandoc pandoc-citeproc && rm -rf /var/lib/apt/lists/*
RUN echo "options(repos = c(CRAN = 'https://cran.rstudio.com/'), download.file.method = 'libcurl')"
>> /usr/local/lib/R/etc/Rprofile.site
# Install renv to restore all Shiny app deps
RUN R -e "install.packages('renv'); renv::consent(provided = TRUE)"
# create root folder for app in container
RUN mkdir /root/app
# Define GITHUB_PAT
ARG github_pat=#{github_pat}#
# Restore Shiny app Deps
COPY renv.lock /root/app
RUN R -e "Sys.setenv(GITHUB_PAT='${github_pat}'); renv::restore(lockfile = '/root/app/renv.lock')"
# copy shiny app to image folder
COPY inst/shiny /root/app
EXPOSE 3838
# launch shiny app
CMD ["R", "-e", "options('shiny.port'=3838,shiny.host='0.0.0.0');shiny::runApp('/root/app')"]
Now, to connect properly to the ACR, we have to use a Docker Compose. So, in the root of your project, you have to create a docker-compose.yml and here you have the content
version: "3.7"
services:
web:
build: .
ports:
- '3838:3838'
Very basic! Now, the funny part with the pipeline. Edit the pipeline. In the Tasks list, search for docker.
And here the magic happens. When you click on it, you have a form with a long list of configurations but we are interested in those:
- Container Registry Type
- Azure subscription
- Azure Container Registry
- Docker Compose file
- Action
Container Registry Type is already selected with Azure Container Registry. Select your Azure subscription and automatically you can select from the Azure Container Registry list what ACR you want to use. Then, from the dropdown list Action select Build service images. Click Add.
After this command the container will be created but not pushed to the ACR. So, we have to add another step for it. Then, again add another Docker Compose but this time select as an Action the option Push services.
Now, your pipeline could be similar to this one (azureSubscription and azureContainerRegistry are a long string)
# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker
trigger:
- main
resources:
- repo: self
variables:
# Container registry service connection established during pipeline creation
dockerRegistryServiceConnection: '...'
# Insert a name for your container
imageRepository: 'p200403'
containerRegistry: $(ACRLoginServer)
dockerfilePath: '$(Build.SourcesDirectory)/DOCKERFILE'
tag: '$(Build.BuildId)'
# Agent VM image name
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Check
condition: eq('${{ variables.imageRepository }}', '')
steps:
- script: |
echo '##[error] The imageRepository must have a value!'
exit 1
- job: Build
condition: not(eq('${{ variables.imageRepository }}', ''))
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: qetza.replacetokens.replacetokens-task.replacetokens@3
displayName: 'Replace tokens'
inputs:
targetFiles: '**/Dockerfile'
- task: DockerCompose@0
inputs:
containerregistrytype: 'Azure Container Registry'
azureSubscription: '...'
azureContainerRegistry: '...'
dockerComposeFile: '**/docker-compose.yml'
action: 'Build services'
projectName: 'p200403'
- task: DockerCompose@0
inputs:
containerregistrytype: 'Azure Container Registry'
azureSubscription: '...'
azureContainerRegistry: '...'
dockerComposeFile: '**/docker-compose.yml'
action: 'Push services'
projectName: 'p200403'
- task: SSH@0
displayName: 'Run shell commands on remote machine'
inputs:
sshEndpoint: 'ShinyServerDev'
commands: |
echo $(SSHPassword) | sudo -S docker pull $(containerRegistry)/$(imageRepository)_web
failOnStdErr: false
continueOnError: true
The last thing to say is about azureSubscription and azureContainerRegistry and it is a bad news. I am trying to replace them using parameters. I can print any parameter inside the yaml. But when I use parameter with any task that requires Azure Subscription that will make the pipeline always fail with
“The pipeline is not valid. Job myDeployment: Step input azureSubscription references service connection $(mySubscription) which could not be found.”
This is a known issue / limitation. You have to pass the Azure subscription as a literal. No way around it that I know of, unfortunately.
It’s been a point of discussion for literally years on this GitHub issue: https://github.com/microsoft/azure-pipelines-agent/issues/1307
Conclusion
So, what do you think about “Pull Docker image from ACR in pipelines”? Please leave a comment or use our Forum.
