-
Notifications
You must be signed in to change notification settings - Fork 1k
Description
Description
The sendWithHttp function in packages/otlp-exporter-base/src/transport/http-transport-utils.ts does not attach an error event handler on the HTTP response (http.IncomingMessage) object. When the remote server resets the TCP connection mid-response (e.g., load balancer rotation, server restart), the ECONNRESET error is emitted on the res stream with no listener, causing Node.js to throw an uncaughtException that crashes the process.
Location
packages/otlp-exporter-base/src/transport/http-transport-utils.ts, inside sendWithHttp():
const req = request(options, (res: http.IncomingMessage) => {
const responseData: Buffer[] = [];
res.on('data', chunk => responseData.push(chunk));
res.on('end', () => {
// ... handle status codes ...
});
// ❌ MISSING: res.on('error', ...) — no error handler on the response stream
});
req.setTimeout(timeoutMillis, () => { ... });
req.on('error', (error: Error) => { ... }); // ← only covers request-phase errorsThe req.on('error') handler at the bottom catches errors during the request write phase. However, once the response callback fires (headers received), socket errors are emitted on the res (IncomingMessage) object instead. Since there is no res.on('error') listener, Node.js treats it as an unhandled error event and throws an uncaughtException.
How to reproduce
const http = require('http');
// Server that sends a partial chunked response then keeps the stream open
const server = http.createServer((_req, res) => {
res.writeHead(200, {
'Content-Type': 'application/json',
'Transfer-Encoding': 'chunked',
});
res.write('{"partial":"data"}');
});
process.on('uncaughtException', (err) => {
console.log('uncaughtException:', err.message, err.code);
// Prints: "write ECONNRESET" "ECONNRESET"
server.close();
process.exit(0);
});
server.listen(0, '127.0.0.1', () => {
const port = server.address().port;
const req = http.request({
hostname: '127.0.0.1', port, path: '/v1/traces', method: 'POST',
headers: { 'Content-Type': 'application/json' },
}, (res) => {
const data = [];
res.on('data', (chunk) => {
data.push(chunk);
// Simulate what the Linux kernel does on a TCP RST during response reading
setTimeout(() => {
const err = new Error('write ECONNRESET');
err.code = 'ECONNRESET';
res.emit('error', err);
}, 50);
});
res.on('end', () => {});
// No res.on('error') — same as the real code
});
req.on('error', () => {});
req.write('{}');
req.end();
});Expected behavior
Socket errors during the response read phase should be caught and routed through the onDone callback, the same way request-phase errors are handled.
Suggested fix
Add a res.on('error') handler inside the response callback:
const req = request(options, (res: http.IncomingMessage) => {
const responseData: Buffer[] = [];
res.on('data', chunk => responseData.push(chunk));
res.on('end', () => { /* ... existing logic ... */ });
+ res.on('error', (error: Error) => {
+ onDone({
+ status: isExportRetryable(error) ? 'retryable' : 'failure',
+ error,
+ });
+ });
});Environment
@opentelemetry/otlp-exporter-base: 0.208.0- Node.js: 20.x
- OS: Linux (observed on cloud hosting; timing-dependent, harder to reproduce on macOS)
Related
- [http exporter] error callback is called twice #5990 — identifies other error handling issues in
sendWithHttp(doubleonDonecallback) - fix(otlp-exporter-base): ensure retry on network errors during HTTP export #6147 — made
ECONNRESETretryable on the request side, but did not addres.on('error') - OTLPExporterBase reports connection issues to otel sink #4444 — user report of
ECONNRESETerrors from the OTLP exporter