fix: resolve duplicate data, optimize csv/excel export, and fix progress bar#2015
Merged
fix: resolve duplicate data, optimize csv/excel export, and fix progress bar#2015
Conversation
ni00
force-pushed
the
bugfix-export-data
branch
from
f7d7c6c to
81e38a7
Compare
ni00
changed the title
ni00
changed the title
There was a problem hiding this comment.
Pull request overview
Fixes the “Export Data” workflow to eliminate duplicated MQTT5 fields in JSON/YAML/XML exports, produce usable CSV/Excel outputs (one message per row), and make the progress bar reflect message-level progress rather than connection-level progress.
Changes:
- Refactors
entityToModel/modelToEntitymappings for Connection/Message models to prevent duplicating fields intoproperties. - Reworks CSV/Excel export to emit flattened message rows (instead of embedding JSON blobs) and updates progress after each streamed batch.
- Adds
MessageService.countByConnectionId()to pre-compute total message counts for smoother progress reporting.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| src/main/streamExportData.ts | Message-level progress tracking; CSV/Excel export rewritten to emit one row per message with flattened columns |
| src/database/services/MessageService.ts | Fixes model/entity conversions to avoid duplicating MQTT5 props; adds countByConnectionId() |
| src/database/services/ConnectionService.ts | Fixes connection/message mapping to avoid duplicating entity fields into properties; improves Will/MQTT5 property mapping |
| src/database/services/TopicNodeService.ts | Import order / constructor formatting changes |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| private connectionRepository: Repository<ConnectionEntity>, | ||
| private messageService: MessageService, | ||
| ) {} | ||
| ) { } |
Comment on lines
7
to
20
| @@ -14,29 +16,48 @@ export default class MessageService { | |||
| // @ts-ignore | |||
| @InjectRepository(ConnectionEntity) | |||
| private connectionRepository: Repository<ConnectionEntity>, | |||
| ) {} | |||
| ) { } | |||
|
|
|||
Comment on lines
17
to
35
| @@ -28,48 +31,83 @@ export default class ConnectionService { | |||
| // @ts-ignore | |||
| @InjectRepository(WillEntity) | |||
| private willRepository: Repository<WillEntity>, | |||
| ) {} | |||
| ) { } | |||
|
|
|||
Comment on lines
+92
to
+96
| let totalMessages = 0 | ||
| for (const conn of connections) { | ||
| totalMessages += await messageService.countByConnectionId(conn.id!) | ||
| } | ||
| this.totalMessages = totalMessages |
Comment on lines
+188
to
+197
| const headers = [ | ||
| 'client_id', | ||
| 'connection_name', | ||
| 'topic', | ||
| 'payload', | ||
| 'qos', | ||
| 'retain', | ||
| 'direction', | ||
| 'createAt', | ||
| 'payload_format_indicator', |
Comment on lines
+216
to
+223
| client_id: connection.clientId, | ||
| connection_name: connection.name, | ||
| topic: msg.topic, | ||
| payload: msg.payload, // payload is string in MessageModel | ||
| qos: msg.qos, | ||
| retain: msg.retain, | ||
| direction: msg.out ? 'publish' : 'received', | ||
| createAt: msg.createAt, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2006
Problems Fixed
1. Duplicate data in JSON/YAML/XML exports
Every exported message contained all fields twice — once at the top level and again inside a
propertiessub-object. Root cause:entityToModelused...entityspread into both the top-level object andproperties, causing full duplication.Fix: destructure MQTT5 property fields explicitly, spread only
...restat the top level, and buildpropertiesfrom the extracted fields only.2. Broken CSV/Excel format
Fix: flatten each message into individual rows with clean columns (
client_id,connection_name,topic,payload,qos,retain,direction,createAt, plus MQTT5 properties). CSV writes one row per message with proper headers; Excel creates one sheet per connection with the same flat structure.3. Inaccurate progress bar
Progress was tracked per-connection (
processedConnections / totalConnections). With 1-2 connections but 150k+ messages, the bar stayed at 0% then jumped to 100%.Fix: count total messages before export, then update progress after each batch (~1000 messages), giving smooth incremental progress across all 5 export formats.
Changed Files
src/database/services/ConnectionService.ts— fixentityToModelto avoid duplicating entity fields intopropertiessrc/database/services/MessageService.ts— fixentityToModel/modelToEntitydestructuring; addcountByConnectionId()src/main/streamExportData.ts— rewrite CSV/Excel export to one-row-per-message; switch progress tracking to message-level granularitySelf-tested
All cases below have been verified locally:
propertiesnestingWould appreciate a second pair of eyes on these scenarios. Thanks!