Skip to content
Draft
Show file tree
Hide file tree
Changes from 72 commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
fc2619e
trying to fix my merge
czirker Jan 31, 2024
4c5a2d2
package cleanup
czirker Jan 31, 2024
129f9c4
Fixing es callbacks and simplifying returning the response body
czirker Jan 31, 2024
c094f10
fixing version to not have -awsv3
czirker Mar 27, 2024
dadf9e7
update moment and async [synk vunerabilities]
jearle10 Nov 26, 2024
181392d
Bump versions
jearle10 Nov 27, 2024
71f7e7f
Save things that are too large to S3.
jgrantr Jan 7, 2025
f1eae33
Merge branch 'feature/aws-sdk-v3-again' into feature/awsv3-s3-entity-…
jgrantr Jan 7, 2025
2f94cdc
Updated version
jgrantr Jan 7, 2025
13778bc
Two more updates:
jgrantr Jan 7, 2025
d20a5a4
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 7, 2025
54ea54f
Don't need `.promise()` with AWS V3
jgrantr Jan 7, 2025
fe3cd4d
Moved delete so it is only done if it was successful
jgrantr Jan 7, 2025
ac12573
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 7, 2025
01237e2
Bumped version number
jgrantr Jan 7, 2025
cad6476
Bumped version number
jgrantr Jan 7, 2025
e42c814
Save stringified original payload
jgrantr Jan 7, 2025
141c0c1
Wrap in try/catch and return an async closure
jgrantr Jan 7, 2025
1d2f3f0
Bumped version
jgrantr Jan 7, 2025
4d20c2e
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 7, 2025
3e53aea
bumped package-lock.json
jgrantr Jan 7, 2025
0799821
ES-2352 - fixed typo
jgrantr Jan 8, 2025
05e4b01
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 8, 2025
127e182
ES-2352 - updated version
jgrantr Jan 8, 2025
8408b30
ES-2352 - renamed parameter to be what is expected
jgrantr Jan 8, 2025
18b3c74
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 8, 2025
b4da3b6
ES-2352 - update version number again
jgrantr Jan 8, 2025
85df1ed
ES-2352 - fixed file/key mixup
jgrantr Jan 8, 2025
41cc606
ES-2352 - new version again
jgrantr Jan 8, 2025
bd0124e
ES-2352 - don't write the data to S3 until it's put into DynamoDB
jgrantr Jan 10, 2025
fe38bde
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 10, 2025
6d0415c
ES-2352 - bumped package version
jgrantr Jan 11, 2025
d12a93e
Merge pull request #222 from jearle10/feature/aws-sdk-v3-again
czirker Jan 27, 2025
3968848
ES-2352 - added logging messages
jgrantr Jan 30, 2025
293c2f7
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 30, 2025
d1d01ea
ES-2352 - bumped version
jgrantr Jan 30, 2025
1f1dad3
ES-2352 - async lib didn't like the function style
jgrantr Jan 31, 2025
335ba5b
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 31, 2025
8a1aba2
ES-2352 - bumped version #
jgrantr Jan 31, 2025
11b03f9
ES-2352 - one more update to make async lib happy
jgrantr Jan 31, 2025
cc0a9eb
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 31, 2025
4a4b8b6
ES-2352 - bumped package version
jgrantr Jan 31, 2025
d2a4788
ES-2352 - make sure old S3 files get removed
jgrantr Jan 31, 2025
2f91304
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 31, 2025
7988756
ES-2352 - adding logging function back in
jgrantr Jan 31, 2025
1894318
Merge branch 'feature/s3-entity-table-loading' into feature/awsv3-s3-…
jgrantr Jan 31, 2025
7ff558a
Merge branch 'feature/aws-sdk-v3-again' into feature/awsv3-s3-entity-…
jgrantr Jan 31, 2025
f09592d
Merge pull request #223 from LeoPlatform/feature/awsv3-s3-entity-tabl…
jgrantr Jan 31, 2025
4ce6b0f
removed extraneous require
jgrantr Feb 19, 2025
2b6a116
Use the new Upload class for the ES connector
jgrantr Feb 20, 2025
1fcbbda
Merge pull request #227 from LeoPlatform/feature/fix-es-connector-awsv3
jgrantr Feb 20, 2025
00a111d
default host to https:// if not specified
jgrantr Feb 20, 2025
4272acf
Merge branch 'feature/aws-sdk-v3-again' into feature/fix-es-connector…
jgrantr Feb 20, 2025
3568f90
Merge pull request #228 from LeoPlatform/feature/fix-es-connector-awsv3
jgrantr Feb 20, 2025
16a4f81
Updated package and package-log
jgrantr Feb 20, 2025
cd80f6d
Two fixes for connectors
jgrantr May 6, 2025
87832c5
Changed entity-table connector package version to drop -awsv3 postfix
jgrantr May 6, 2025
63e992f
chore: upgrade js-beautify version in leo-connector-entity-table to…
mariagr-chub May 12, 2025
d5d876c
Merge pull request #229 from mariagr-chub/feature/aws-sdk-v3-again
mecham-lynn May 12, 2025
5b0bce6
Make sure to use `suggestCompression` by default.
jgrantr Sep 2, 2025
89c996f
Updated the types to reflect the `returnFullResponse` behavior
jgrantr Sep 2, 2025
9d91a5f
Updated working types
jgrantr Sep 2, 2025
e72ca4a
fix: new OpenSearch client has a query package, this works around that
jgrantr Sep 9, 2025
973f4ff
ES-2352 - use a proxy
jgrantr Sep 10, 2025
31545df
Merge pull request #230 from LeoPlatform/feature/ES-2352-use-proxy-fo…
jgrantr Sep 10, 2025
02b8fa1
Updated package version
jgrantr Sep 10, 2025
6039451
fix: new OpenSearch client doesn't use scrollId
jgrantr Sep 11, 2025
49fcc4a
Fixed one more instance.
jgrantr Sep 11, 2025
57021a5
ES-2352 - make sure to pass along error
jgrantr Sep 11, 2025
8d4b938
ES-2352 - improvements to entity table connector
jgrantr Oct 21, 2025
7d23302
ES-2352 - allow the transform function to be asynchronous
jgrantr Oct 31, 2025
e60af1b
Merge pull request #231 from LeoPlatform/feature/ES-2352-filtering-an…
jgrantr Oct 31, 2025
10b4209
ES-2352 - new connector versions
jgrantr Nov 10, 2025
355760a
ES-2352 - add retries for conflicting updates
jgrantr Feb 24, 2026
47bd6cc
ES-2352 - added retryOnConflict to streamParallel
jgrantr Feb 24, 2026
378e169
ES-2352 - use nullish coalescing instead
jgrantr Feb 24, 2026
88f7e83
Merge pull request #233 from LeoPlatform/feature/ES-2352-add-conflict…
jgrantr Feb 24, 2026
b471834
Updated the typescript types
jgrantr Feb 24, 2026
7a28795
ES-2352 - retry 429 errors
jgrantr Mar 12, 2026
a09e672
Merge pull request #234 from LeoPlatform/feature/ES-2352-retyr-429
jgrantr Mar 12, 2026
f2be291
ES-2352 - update version
jgrantr Mar 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 14 additions & 13 deletions common/checksum/index.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
let leo = require("leo-sdk");
const aws = require("aws-sdk");
const { Lambda } = require("@aws-sdk/client-lambda");
const checksum = require("./lib/checksumNibbler.js");
const leoaws = require('leo-aws');
let cron = leo.bot;
Expand Down Expand Up @@ -271,17 +271,18 @@ module.exports = {
lastUpdate: moment.now(),
status: status,
statusReason: err ? err.toString() : stopReason
})).then(result => {
async.parallel(tasks, (err, data) => {
if (err) {
reject(err);
} else {
resolve(session);
}
}))
.then(result => {
async.parallel(tasks, (err, data) => {
if (err) {
reject(err);
} else {
resolve(session);
}
});
}, err => {
reject(err);
});
}, err => {
reject(err);
});
});
});
}, logError);
Expand All @@ -290,7 +291,7 @@ module.exports = {
},
lambdaConnector: function(id, lambdaName, settings) {
let region = (lambdaName.match(/arn:aws:lambda:(.*?):/) || [])[1];
const lambdaInvoker = new aws.Lambda({
const lambdaInvoker = new Lambda({
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lambda invoke uses v2 callback pattern, never resolves

High Severity

The lambdaInvoker.invoke() call passes a callback as the second argument, but AWS SDK v3's Lambda client does not support callbacks — it returns a Promise instead. The callback will be silently ignored, and since it's wrapped in a new Promise() that only resolves/rejects inside that callback, every Lambda connector operation (init, range, nibble, getChecksum, etc.) will hang forever without resolving or rejecting.

Fix in Cursor Fix in Web

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lambda response Payload is Uint8Array in SDK v3

High Severity

In AWS SDK v3, the Payload field in a Lambda invoke response is a Uint8Array, not a string. The comparisons data.Payload != 'null' and JSON.parse(data.Payload) will not behave correctly — the string comparison against a Uint8Array will never match, and JSON.parse may not properly decode the binary payload. The payload needs conversion via Buffer.from(data.Payload).toString() or equivalent before parsing.

Fix in Cursor Fix in Web

region: region || this.configuration._meta.region,
credentials: this.configuration ? this.configuration.credentials : null
});
Expand Down Expand Up @@ -567,7 +568,7 @@ module.exports = {
let session = null;

function invoke(method, func) {
func = func || ((d) => {});
func = func || ((d) => { });
return (data) => {
return new Promise((resolve, reject) => {
//logger.log(method, data)
Expand Down
Loading