Cache and live LogStream problems with Copilot Agent running on self-hosted ARC

[Hyp3rSon1X]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

ARC (Actions Runner Controller)

Discussion Details

I'm running a selfhosted ARC for the Copilot Agent run its workspace on.
However there are a few quirks I don't know how to solve:

On my instance, no matter what I do - the runner on start always keeps downloading the CodeQL package a new

I tried caching it, I tried to create a pvc, pre download CodeQL into it and mounted that into the /opt/hostedtoolcache path of the runners (/opt/hostedtoolcache/CodeQL/2.23.8/x64) even create a .completed marker file because Copilot claimed that would make the Runner notice the file existed... but nothing helped. The huge CodeQL package keeps getting downloaded on every run... am I doing something wrong? Is there an elegant way to cache it or at least correctly bake it into a custom Image so it doesn't need to get downloaded?

How much and how can we cache the workspace the Copilot Agent needs or wants? I Also see it downloads some runtime in the beginning, but at least that one doesn't take so long. But of course if we can cache it away - even if with a limited lifetime of 1 day, it would be a huge help towards saving bandwidth.

The next problem:
On the official Github Runners Copilot always gives small insights into the files it is looking into or changing:

However... on my self-hosted runner, I don't see the small thoughts of Copilot, only the files it touched:

Is this expected behaviour? A bug? Or am I and idiot and I have misconfigured something?
In the Actions view (verbose view) I can see that the Agent still actually has those thoughts and they are logged there as well, but somehow these thoughts don't appear on the Agent Session view on Github.

Obviously I tried to solve all the mentioned problems with extensive discussions with Copilot - however I failed miserably so now I am at the mercy of people who are actually in the know...

Ah btw... before people tell me self-hosted runners are not supported by Github or something: https://github.blog/changelog/2025-10-28-copilot-coding-agent-now-supports-self-hosted-runners/

It is... I had Copilot already try to convince me otherwise. However sadly I couldn't find anything in the docs that would adress my issues.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬

[PaiavullaNikhil]

Copilot Agent on Self-Hosted ARC – Full Issue Overview

This file explains the two problems observed while running the Copilot Coding Agent on a self-hosted ARC (Actions Runner Controller) setup:

1. CodeQL downloads on every run
2. Copilot “thoughts” do not appear in the Agent Session UI

Self-hosted runners are officially supported, so this behavior is not due to unsupported configuration.

---

1. CodeQL Downloads Every Time

Problem

Even after:

• Pre-downloading CodeQL
• Mounting a PVC to /opt/hostedtoolcache
• Creating folder structure like
  /opt/hostedtoolcache/CodeQL/2.23.8/x64
• Adding a .complete marker file

The runner still downloads CodeQL again for every job.

---

Why This Happens

Most ARC deployments use ephemeral runners.

That means:

• Each job runs in a fresh pod
• Filesystem resets after job completion
• Tool cache does not persist automatically

The runner also checks:

• Exact version match
• Exact folder structure
• Correct architecture (x64 / arm64)
• Proper file permissions
• Presence of metadata markers

If anything is slightly different, it downloads CodeQL again.

---

Recommended Solution (Best Option)

Bake CodeQL into a Custom Runner Image

Instead of mounting via PVC:

1. Start from the official GitHub runner image
2. Download the exact CodeQL version used in workflows
3. Extract into
   /opt/hostedtoolcache/CodeQL/<version>/x64
4. Ensure:
   • Correct ownership (runner user)
   • Proper permissions
   • .complete file exists
   • Folder layout matches GitHub-hosted runner

Then configure ARC to use this custom image.

This prevents runtime downloads.

---

Alternative Option

Use persistent runners instead of ephemeral scale sets.

If runners are not destroyed after each job:

• PVC mounts become effective
• Tool cache survives between runs

With ephemeral runners, PVC caching usually does not help much.

---

2. Copilot Agent Workspace & Runtime Caching

Observation

At the start of a job, Copilot downloads:

• Runtime components
• Supporting dependencies

These are not fully cached between runs.

---

Current Limitation

There is no fully documented persistent caching mechanism for Copilot Agent runtime in self-hosted ARC environments.

GitHub-hosted runners likely use internal infrastructure optimizations that are not exposed externally.

---

What Can Be Tried

• Use persistent runners
• Mount workspace directory to PVC
• Use actions/cache for specific directories
• Keep ARC and runner versions updated
• Reduce pod teardown frequency

Full feature parity with GitHub-hosted runners may not be available yet.

---

3. Missing “Thoughts” in Agent Session View

Problem

On GitHub-hosted runners:

• Copilot displays small reasoning insights in the session UI.

On self-hosted ARC:

• Only touched files are visible
• Thoughts do not appear in Agent Session view
• Thoughts are visible in verbose Action logs

---

Likely Explanation

Since:

• Thoughts exist in logs
• Agent generates them correctly

This suggests:

• Not a configuration mistake
• Possibly a streaming or UI limitation
• May be related to how live log events are forwarded in self-hosted setups

Self-hosted streaming behavior may differ from GitHub-hosted environments.

---

Things to Verify

• Runner version is up to date
• ARC version is updated
• No firewall/proxy blocks outbound GitHub communication
• No network policy blocks streaming
• No custom log filtering interfering

If all checks pass, this may be a platform-level limitation.

---

Summary

Issue: CodeQL re-download
Cause: Ephemeral runners + cache mismatch
Fix: Bake into custom runner image

Issue: Workspace not cached
Cause: Pod lifecycle resets
Fix: Use persistent runners

Issue: Missing live thoughts in UI
Cause: Self-hosted streaming limitation
Fix: Likely expected behavior currently

---

Final Note

Self-hosted runners are supported for Copilot Agent.

However:

• Behavior is not identical to GitHub-hosted runners
• Some caching and UI features may depend on GitHub infrastructure
• Documentation around ARC + Copilot Agent is still evolving

If everything is up to date and the issue persists, clarification from GitHub support may be required.