Skip to content

[Bug]: Report screenshots are completely black in headless Linux environment #2970

Open

Description

@leiyaoyu187-del

Version

Environment
@midscene/core: 1.10.3
@midscene/web: 1.10.3
Playwright: 1.48.0
Node.js: 22.23.1
OS: Ubuntu 22.04 LTS (headless, no GPU)
Browser: Chromium (headless mode)
MIDSCENE_OUTPUT_FORMAT: single-html

Details

Describe the bug

When running Midscene.js with Playwright in a headless Linux environment (no GPU), the AI agent correctly identifies and interacts with page elements, but the screenshots saved in the HTML report are completely black.

Environment

  • @midscene/core: 1.10.3
  • @midscene/web: 1.10.3
  • Playwright: 1.48.0
  • Node.js: 22.23.1
  • OS: Ubuntu 22.04 LTS (headless, no GPU)
  • Browser: Chromium (headless mode)
  • MIDSCENE_OUTPUT_FORMAT: single-html

Steps to Reproduce

  1. Run a Midscene script on a Linux server with headless Chromium
  2. The target page has CSS animations and gradient backgrounds (a login page with animated background)
  3. Script uses agent.aiTap(), agent.aiAssert() etc.
  4. All AI actions succeed — the agent correctly locates buttons, input fields, and verifies page content
  5. Open the generated HTML report

Expected Behavior

The HTML report should show screenshots of the page at each step, matching what the AI "saw" during execution.

Actual Behavior

All screenshots in the report are completely black. The HTML file is 3-4MB (screenshots ARE embedded as base64 in single-html mode), but the image content is black.

Key Observation

The AI agent can correctly identify elements on the page (e.g., "账号输入框" at coordinates [1330, 346], "登录按钮", etc.). This means the screenshots taken for AI processing are rendered correctly. However, the screenshots saved to the report are black.

This suggests that the screenshots used for AI inference and the screenshots saved to the report are captured at different times or through different mechanisms, and the report screenshots fail to render properly.

What I've Tried

  1. --disable-gpu flag → screenshots still black
  2. --use-angle=swiftshader flag → screenshots completely broken (file not found)
  3. MIDSCENE_OUTPUT_FORMAT=single-html → screenshots embedded but still black
  4. MIDSCENE_OUTPUT_FORMAT=html-and-external-assets → external PNG files not generated at all
  5. Adding waitForTimeout(3000) before actions → no effect on report screenshots

Script Example

page.setViewportSize({ width: 1600, height: 900 });
await page.goto('xxxxx');
await page.waitForTimeout(3000);
await page.getByText('账号密码登录').click();
await page.waitForTimeout(1000);
await agent.aiTap('账号输入框');
await page.keyboard.type('xxx');
await agent.aiTap('密码输入框');
await page.keyboard.type('password');
await agent.aiTap('登录按钮');
await page.waitForTimeout(3000);

### Reproduce link

N/A

### Reproduce Steps

1. Set up a Linux server (Ubuntu 22.04) with Node.js 22+ and no GPU
2. Install Chromium via Playwright: npx playwright install chromium
3. Install Midscene: npm install @midscene/core @midscene/web
4. Create a script that uses PlaywrightAgent to interact with a page that has CSS animations/gradients:

const { chromium } = require('playwright');
const { PlaywrightAgent } = require('@midscene/web/playwright');

process.env.MIDSCENE_OUTPUT_FORMAT = 'single-html';
process.env.MIDSCENE_MODEL_NAME = 'your-model';
process.env.MIDSCENE_MODEL_API_KEY = 'your-key';
process.env.MIDSCENE_MODEL_FAMILY = 'your-family';

(async () => {
  const browser = await chromium.launch({ headless: true, args: ['--no-sandbox'] });
  const page = await browser.newPage({ viewport: { width: 1600, height: 900 } });
  const agent = new PlaywrightAgent(page);
  
  await page.goto('xxxx');
  await page.waitForTimeout(3000);
  await agent.aiTap('账号输入框');
  await page.keyboard.type('test');
  
  // Check the generated HTML report - screenshots will be black
  await browser.close();
})();

5. Open the generated HTML report in a browser
6. Observe that all screenshots in the report are completely black, even though AI correctly identified elements

Activity

  1. quanru commented on Sep 8, 2026

    @quanru
    Collaborator

    We tested this on Ubuntu 22.04 CI with Node.js 22.23.1, including Midscene 1.10.3 + Playwright 1.48.0 and the current workspace version. Using an animated-gradient login fixture, both report formats passed with default Chromium flags and --disable-gpu. The saved images and rendered report player displayed correctly. These tests use a local model stub to isolate screenshot and report handling. CI results and artifacts.

    Could you please provide either:

    • A generated HTML report that shows the black screenshots. For html-and-external-assets, please include the entire report directory.
    • A minimal reproduction repository or a standalone HTML/CSS page, together with the script that reproduces the issue.

    Please remove any sensitive information before sharing. Also, are the extracted images themselves black, or do they only appear black in the report viewer? If it is viewer-only, please include the browser/version used to open the report and any console errors.

    One configuration detail: MIDSCENE_OUTPUT_FORMAT is not read by the SDK. To generate external image assets, pass outputFormat: 'html-and-external-assets' in the PlaywrightAgent options.

  2. quanruzhuoxiu commented on Oct 30, 2026

    @quanruzhuoxiu
    Collaborator

    Following up — our Ubuntu 22.04 CI runs (including animated-gradient pages, with and without --disable-gpu) produce normal screenshots, so we still need a bit more to reproduce: 1) the base image/Dockerfile (or image name) you run in, 2) the exact Chromium/Playwright launch flags, 3) whether the page itself is black when you open the same URL in that environment (i.e. is it a rendering issue or a screenshot-capture issue), 4) one affected report file. With that we can tell whether it's a missing GPU/swiftshader dependency in the container or a Midscene capture issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions