bug/2418 throttle saving markdown files for automation and dataExtension#2419
Conversation
Coverage ReportCommit:9e5cf9eBase: develop@07ff145 Details (changed files):
|
There was a problem hiding this comment.
Pull Request Overview
This PR implements throttling for saving markdown/HTML documentation files for Automation and DataExtension metadata types to address issue #2418. The changes use the p-limit library (already imported in both files) to limit concurrent file write operations.
- Added rate limiting with
pLimit(100)to document generation methods - Wrapped
_writeDoccalls in throttling functions to prevent overwhelming the file system
Reviewed Changes
Copilot reviewed 2 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| lib/metadataTypes/DataExtension.js | Added throttling to document generation with pLimit(100), but introduced a bug where 'both' document type only generates HTML |
| lib/metadataTypes/Automation.js | Added throttling to document generation with pLimit(100) for markdown files |
| @types/lib/metadataTypes/DataExtension.d.ts.map | Auto-generated TypeScript declaration map file updated |
| @types/lib/metadataTypes/Automation.d.ts.map | Auto-generated TypeScript declaration map file updated |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (['html', 'both'].includes(this.properties.options.documentType)) { | ||
| return this._writeDoc( | ||
| docPath + '/', | ||
| key, | ||
| metadataMap[key], | ||
| 'html', | ||
| columnsToPrint | ||
| return docLimit(() => | ||
| this._writeDoc( | ||
| docPath + '/', | ||
| key, | ||
| metadataMap[key], | ||
| 'html', | ||
| columnsToPrint | ||
| ) | ||
| ); | ||
| } | ||
| if (['md', 'both'].includes(this.properties.options.documentType)) { | ||
| return this._writeDoc( | ||
| docPath + '/', | ||
| key, | ||
| metadataMap[key], | ||
| 'md', | ||
| columnsToPrint | ||
| return docLimit(() => | ||
| this._writeDoc( | ||
| docPath + '/', | ||
| key, | ||
| metadataMap[key], | ||
| 'md', | ||
| columnsToPrint | ||
| ) | ||
| ); | ||
| } |
There was a problem hiding this comment.
When documentType is 'both', only the HTML document will be generated because the function returns early after the first condition. The MD generation code (lines 1453-1462) will never be reached. To fix this, the code should collect both promises when documentType is 'both' and return them together, for example:
const promises = [];
if (['html', 'both'].includes(this.properties.options.documentType)) {
promises.push(docLimit(() =>
this._writeDoc(docPath + '/', key, metadataMap[key], 'html', columnsToPrint)
));
}
if (['md', 'both'].includes(this.properties.options.documentType)) {
promises.push(docLimit(() =>
this._writeDoc(docPath + '/', key, metadataMap[key], 'md', columnsToPrint)
));
}
return Promise.all(promises);
PR details
What changes did you make? (Give an overview)
Further details (optional)
...
Checklist