Pro client generates apt sources with upstream GPG key even when using local custom apt mirror

[ggouzi]

Description of the bug

Cannot enable esm-infra when using pro client in airgap env with a local mirror configured with a GPG key

• Local mirror URL: https://landscape-d-api.<REDACTED>/repository/standalone/ubuntu
• Local mirror GPG key file: /etc/apt/trusted.gpg.d/custom.gpg
• Local pro airgap server: https://ubuntu-pro-contracting-d.<REDACTED>

Expected behavior

sudo pro enable esm-infra works as expected

Current behavior

sudo pro enable esm-infra fails. debug logs here: https://pastebin.ubuntu.com/p/WytZyKM4Cc/
Relevant logs:

Error updating apt cache: W:Updating from such a repository can't be done securely, and is therefore disabled by default., W:See apt-secure(8) manpage for repository creation and user configuration details., W:GPG error: https://landscape-d-api.<REDACTED>/repository/standalone/ubuntu noble-infra-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY <REDACTED>, E:The repository 'https://landscape-d-api.<REDACTED>/repository/standalone/ubuntu noble-infra-security InRelease' is not signed.

To Reproduce

Configure a local mirror using https://charmhub.io/pro-airgapped-server

cat /etc/ubuntu-advantage/uaclient.conf 
contract_url: https://ubuntu-pro-contracting-d.<REDACTED>
log_level: debug

ubuntu-pro-contracting-d.<REDACTED> references a local mirror on landscape

sudo pro attach <TOKEN>
sudo pro --debug enable esm-infra

Additional information:

The following file is generated by pro client

cat /etc/apt/sources.list.d/ubuntu-esm-infra.sources 
# Written by ubuntu-pro-client
Types: deb
URIs: https://landscape-d-api.<REDACTED>/repository/standalone/ubuntu
Suites: noble-infra-security
Components: main
Signed-By: /usr/share/keyrings/ubuntu-pro-esm-infra.gpg

However, the GPG key is not the correct one. It is the one used for upstream resources.

Replacing it with the custom GPG key of the local mirror

Signed-By: /etc/apt/trusted.gpg.d/custom.gpg

And then esm-infra can be enabled

Is there a way we can add a setting to customize GPG key on the pro client ? And also on the pro-client-operator as well ?

System information:

• Ubuntu 24.04
• Pro client: 37.1ubuntu0~24.04

Workaround

Configure apt to ignore all potential GPG keys in source files and trust all repositories

Edit: Workaround doesn't work

[nobuto-m]

It seems it's hard-coded by assuming a mirror without changing the signature.

ubuntu-pro-client/uaclient/apt.py

Lines 568 to 574 in 68d14de

	content = DEB822_REPO_FILE_CONTENT.format(
	url=repo_url,
	suites=" ".join(appliable_suites),
	keyrings_dir=KEYRINGS_DIR,
	keyring_file=keyring_file,
	deb_src=deb_src,
	)

[nobuto-m]

One dirty workaround I can think of is to do dpkg-divert to /usr/share/keyrings/ubuntu-pro-esm-infra.gpg and siblings so new updates of ubuntu-pro-client deb package doesn't override the custom GPG public key.

[VariableDeclared]

It seems the pro airgapped server has or had the ability to set keys, but the variable is unreferenced in the pro client:

ubuntu-pro-client/uaclient/entitlements/repo.py

Lines 656 to 659 in 68d14de

	aptKey = directives.get("aptKey")
	if not aptKey:
	raise exceptions.RepoNoAptKey(entitlement_name=self.name)
	repo_url = directives.get("aptURL")

I'm struggling to see where aptKey is utilised in the method.

[ggouzi]

If it can help future visitors, the following workaround works despite being ugly

write_files:
    - path: /etc/apt/trusted.gpg.d/custom-gpg-key.gpg
      permissions: '0644'
      encoding: b64
      # Output of gpg --yes --dearmor -o /dev/stdout custom-gpg-key.gpg | base64 -w0
      content: |
        <REDACTED>
  - path: /usr/local/bin/cowboy-patch-gpg-key-pro-client.sh
    permissions: '0744'
    content: |
      #!/bin/bash
      # Workaround for https://github.com/canonical/ubuntu-pro-client/issues/3519
      SERVICES="anbox cc-eal cis esm-apps-legacy esm-apps esm-infra-legacy esm-infra fips-preview fips realtime-kernel ros"
      for service in $SERVICES; do
        keypath="/usr/share/keyrings/ubuntu-pro-${service}.gpg"
        # Check if this file is already diverted
        if dpkg-divert --list "$keypath" 2>/dev/null | grep -q .; then
          echo "Diversion already exists for $keypath, skipping divert"
        else
          dpkg-divert --rename --divert "${keypath}.distro" "$keypath"
        fi
        cp /etc/apt/trusted.gpg.d/custom-gpg-key.gpg "$keypath"
      done
postruncmd:
  - /usr/local/bin/cowboy-patch-gpg-key-pro-client.sh