Cloning all repositories from Azure DevOps

I will provide a script for cloning all repositories from Azure DevOps in an organisation with multiple projects and repositories using PowerShell.

Scenario

Often, I end up with a big mess in the repositories from a project at work. I want to download all the repositories again from a particular organisation. For example, I used to split my code into NuGet packages to manage the projects easily. Sometimes, I need to replace the NuGet packages with the projects to verify a bug.

There is another case. As a contractor, when I arrive at a customer site, the companies provide me access to several projects. These projects are in Azure DevOps. These projects include repositories that I’m supposed to contribute to.

In other cases, I just want to have a copy of all the repositories to save the code.

In all of these scenarios, I find myself copy-pasting URLs and running git clone a few times. I know, it doesn’t sound too hard, but I dislike doing things more than once!

My solution

There’s an extension to Azure CLI for integrating with Azure DevOps, which I’ve found really handy. Not many people are aware of the vast number of extensions available to the Azure CLI. Now, I want to show you how I can use this. It allows me to clone all Azure repositories I have access to in Azure DevOps.

I assume we already have Azure CLI installed; if not, it’s time to Install the Azure CLI.

First, we need to sign in. This is done by calling

az login

If I don’t have access to any subscriptions in Azure, I’ll have to add the parameter –allow-no-subscriptions.

az login --allow-no-subscriptions

Then, we need to make sure we have the right extension installed. Installed extensions can be listed with the command:

az extension list

If there are many extensions installed, this list can be overwhelming. I convert the output from JSON to objects using PowerShell. Then, I would filter the list as I would anything else in PowerShell. So, there’s a native way of filtering in the Azure CLI, so why not use that? The Azure CLI has a parameter called –query that takes a JMESPath expression. JMESPath is similar to XPath, but for JSON instead of XML; it is documented over at https://jmespath.org/.

The response I get from listing extensions is an array of extensions. To filter that, I can use the following expression: [?name == ‘‘azure-devops’’].name. I can also instruct the Azure CLI to output the result in a tab-separated format. I do this by adding the parameter -o tsv instead of JSON. This suits me well here since I’ll just get one name back.

az extension list  --query '[?name == ''azure-devops''].name' -o tsv

If from the list I can get azure-devops back, I’m good to go! If I get nothing back, I need to install the DevOps extension. This is done by running:

az extension add --name 'azure-devops'

Now, we’re ready to shoot some queries to Azure DevOps! To list repositories in Azure DevOps, we need to supply the organisation URL and the project name. Let’s store the organisation name in a variable and use that to list all projects. Once again, we use a query to get only the names back and choose to get output as TSV.

$Organization = 'https://dev.azure.com/enricorossini'
$Projects = az devops project list --organization $Organization --query 'value[].name' -o tsv

Now, I’m ready to list all repositories in each project!

foreach ($Proj in $Projects) {
  az repos list --organization $Organization --project $Proj | ConvertFrom-Json
}

The full script

This will output an object for each repository. The path to clone using HTTPS if found in the property webUrl. Now, if we combine this, we get the following script:

param(
    [string]$Organization
)

if ($Organization -notmatch '^https?://dev.azure.com/\w+') {
    $Organization = "https://dev.azure.com/$Organization"
}

# Make sure we are signed in to Azure
$AccountInfo = az account show 2>&1
try {
    $AccountInfo = $AccountInfo | ConvertFrom-Json -ErrorAction Stop
}
catch {
    az login --allow-no-subscriptions
}

# Make sure we have Azure DevOps extension installed
$DevOpsExtension = az extension list --query '[?name == ''azure-devops''].name' -o tsv
if ($null -eq $DevOpsExtension) {
    $null = az extension add --name 'azure-devops'
}

$Projects = az devops project list --organization $Organization --query 'value[].name' -o tsv
foreach ($Proj in $Projects) {
    if (-not (Test-Path -Path ".\$Proj" -PathType Container)) {
        New-Item -Path $Proj -ItemType Directory |
        Select-Object -ExpandProperty FullName |
        Push-Location
    }
    $Repos = az repos list --organization $Organization --project $Proj | ConvertFrom-Json
    foreach ($Repo in $Repos) {
        if(-not (Test-Path -Path $Repo.name -PathType Container)) {
            Write-Warning -Message "Cloning repo $Proj\$($Repo.Name) [$($Repo.webUrl)]"
            git clone $Repo.webUrl
        }
    }
}

Script for a list of repositories

So, with the above script, cloning all repositories from Azure DevOps is quite easy. Now, consider I want to add the list of repositories in a Wiki page: I want to have the name of each repository with the link. For that, I have modified the script and

param(
    [string]$Organization
)

if ($Organization -notmatch '^https?://dev.azure.com/\w+') {
    $Organization = "https://dev.azure.com/$Organization"
}

# Make sure we are signed in to Azure
$AccountInfo = az account show 2>&1
try {
    $AccountInfo = $AccountInfo | ConvertFrom-Json -ErrorAction Stop
}
catch {
    az login --allow-no-subscriptions
}

# Make sure we have Azure DevOps extension installed
$DevOpsExtension = az extension list --query '[?name == ''azure-devops''].name' -o tsv
if ($null -eq $DevOpsExtension) {
    $null = az extension add --name 'azure-devops'
}

$list = ""
$cmd = ""
$Projects = az devops project list --organization $Organization --query 'value[].name' -o tsv
foreach ($Proj in $Projects) {
    if (-not (Test-Path -Path ".\$Proj" -PathType Container)) {
        New-Item -Path $Proj -ItemType Directory |
        Select-Object -ExpandProperty FullName |
        Push-Location
    }
    $Repos = az repos list --organization $Organization --project $Proj | ConvertFrom-Json
    foreach ($Repo in $Repos) {
        if(-not (Test-Path -Path $Repo.name -PathType Container)) {
            $list = $list + "[" + $Repo.Name + "](" + $Repo.webUrl + ")`n"
            $cmd = $cmd + "git clone " + $Repo.webUrl + "`n"
        }
    }
}

Write-Host "List of repositories"
Write-Host $list

Write-Host "`n`n-------`n`n"

Write-Host "Command to execute:"
Write-Host $cmd

Wrap up

In conclusion, I show a script in this post. It clones all repositories from Azure DevOps. It also lists them in Markdown format. I hope those scripts can help you. Keep in touch!

Happy coding!