Skip to content

bug/2418 throttle saving markdown files for automation and dataExtension#2419

Merged
JoernBerkefeld merged 1 commit into
developfrom
bug/2418-dataextension-docs-cannot-be-saved-if-too-many-des-are-found
Nov 11, 2025
Merged

bug/2418 throttle saving markdown files for automation and dataExtension#2419
JoernBerkefeld merged 1 commit into
developfrom
bug/2418-dataextension-docs-cannot-be-saved-if-too-many-des-are-found

Conversation

@JoernBerkefeld
Copy link
Copy Markdown
Contributor

PR details

What changes did you make? (Give an overview)

Further details (optional)

...

Checklist

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • test scripts updated
  • Wiki updated (if applicable)

@JoernBerkefeld JoernBerkefeld added this to the 8.2.1 milestone Nov 11, 2025
@JoernBerkefeld JoernBerkefeld self-assigned this Nov 11, 2025
@JoernBerkefeld JoernBerkefeld linked an issue Nov 11, 2025 that may be closed by this pull request
2 tasks
@github-actions github-actions Bot added the bug Something isn't working label Nov 11, 2025
@github-actions
Copy link
Copy Markdown

Coverage Report

Commit:9e5cf9e
Base: develop@07ff145

Type Base This PR
Total Statements Coverage  70.76%  70.77%  (+0.01%)
Total Branches Coverage  70.77%  70.75%  (-0.02%)
Total Functions Coverage  83.42%  83.42%  (+0%)
Total Lines Coverage  70.76%  70.77%  (+0.01%)
Details (changed files):
File Statements Branches Functions Lines
lib/metadataTypes/Automation.js  77.71%  70.47%  91.17%  77.71%
lib/metadataTypes/DataExtension.js  79.19%  72.05%  94.44%  79.19%

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements throttling for saving markdown/HTML documentation files for Automation and DataExtension metadata types to address issue #2418. The changes use the p-limit library (already imported in both files) to limit concurrent file write operations.

  • Added rate limiting with pLimit(100) to document generation methods
  • Wrapped _writeDoc calls in throttling functions to prevent overwhelming the file system

Reviewed Changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated 1 comment.

File Description
lib/metadataTypes/DataExtension.js Added throttling to document generation with pLimit(100), but introduced a bug where 'both' document type only generates HTML
lib/metadataTypes/Automation.js Added throttling to document generation with pLimit(100) for markdown files
@types/lib/metadataTypes/DataExtension.d.ts.map Auto-generated TypeScript declaration map file updated
@types/lib/metadataTypes/Automation.d.ts.map Auto-generated TypeScript declaration map file updated

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 1442 to 1463
if (['html', 'both'].includes(this.properties.options.documentType)) {
return this._writeDoc(
docPath + '/',
key,
metadataMap[key],
'html',
columnsToPrint
return docLimit(() =>
this._writeDoc(
docPath + '/',
key,
metadataMap[key],
'html',
columnsToPrint
)
);
}
if (['md', 'both'].includes(this.properties.options.documentType)) {
return this._writeDoc(
docPath + '/',
key,
metadataMap[key],
'md',
columnsToPrint
return docLimit(() =>
this._writeDoc(
docPath + '/',
key,
metadataMap[key],
'md',
columnsToPrint
)
);
}
Copy link

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When documentType is 'both', only the HTML document will be generated because the function returns early after the first condition. The MD generation code (lines 1453-1462) will never be reached. To fix this, the code should collect both promises when documentType is 'both' and return them together, for example:

const promises = [];
if (['html', 'both'].includes(this.properties.options.documentType)) {
    promises.push(docLimit(() =>
        this._writeDoc(docPath + '/', key, metadataMap[key], 'html', columnsToPrint)
    ));
}
if (['md', 'both'].includes(this.properties.options.documentType)) {
    promises.push(docLimit(() =>
        this._writeDoc(docPath + '/', key, metadataMap[key], 'md', columnsToPrint)
    ));
}
return Promise.all(promises);

Copilot uses AI. Check for mistakes.
@JoernBerkefeld JoernBerkefeld merged commit 405c1a8 into develop Nov 11, 2025
16 checks passed
@JoernBerkefeld JoernBerkefeld deleted the bug/2418-dataextension-docs-cannot-be-saved-if-too-many-des-are-found branch November 11, 2025 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working c/automation COMPONENT c/dataExtension COMPONENT

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] dataExtension docs cannot be saved if too many DEs are found

2 participants