Add batch processing support for processing multiple files#276
Merged
Add batch processing support for processing multiple files#276
Conversation
Add comprehensive batch processing functionality to process multiple
accelerometer files from a folder in a single command.
Changes to src/accelerometer/accProcess.py:
- Add matchesExtension() to match file extensions with compression support
- Add discoverFiles() to discover files by extension with recursive option
- Add processSingleFile() extracted from main() for reusability
- Add processBatch() to orchestrate batch processing with error handling
- Add --fileExtensions argument (required for batch mode)
- Add --recursive argument for subdirectory search
- Refactor main() to detect and route file vs directory input
- Implement try-finally cleanup to ensure intermediate files deleted on error
- Catch SystemExit exceptions to prevent device.py sys.exit() from killing batch
- Detect duplicate basenames and exit with error to prevent data loss
- Exit with non-zero code when no files found or any files fail
- Respect --verbose flag for detailed error reporting in batch mode
- Remove unused atexit dependency (cleanup now in finally block)
Changes to README.md:
- Add batch processing usage section with examples
- Update output paths to reflect new outputs/{filename}/ structure
- Document --fileExtensions and --recursive options
- Explain compression format auto-detection
Key features:
- Serial processing with per-file error isolation
- Comprehensive batch summary with success/failure counts
- Exit codes: 255 for no files/duplicates, 1 for any failures
- Smart extension matching (case-insensitive, compression aware)
- Duplicate basename detection prevents output overwrites
- Backward compatible: single file mode unchanged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add comprehensive batch processing functionality to process multiple accelerometer files from a folder in a single command. This enables efficient processing of large datasets without manual scripting.
Key Features
--fileExtensions(required)--recursiveflag for subdirectory traversal.gz,.zip,.bz2,.xzvariantsChanges
src/accelerometer/accProcess.pymatchesExtension()for file extension matching with compression supportdiscoverFiles()for file discovery with recursive optionprocessSingleFile()extracted frommain()for reusabilityprocessBatch()to orchestrate batch processing--fileExtensionsargument (required for batch mode)--recursiveargument for subdirectory searchmain()to detect and route file vs directory inputSystemExitto preventdevice.pyfrom killing batch--verboseflag in batch error reportingREADME.mdoutputs/{filename}/structureUsage Examples
Testing
Exit Codes
0: Success (all files processed successfully)1: Partial/complete failure (at least one file failed)255: Setup error (no files found, duplicates, invalid arguments)Breaking Changes
None. Single file processing remains unchanged. Batch mode is opt-in via directory input.