Skip to content

Commit 3fd2805

Browse files
authored
Chunked fs (#78)
* feat: ✨ Parameters per attempt, removing attempt from execution, chunked_fs * fix: 🐛 parallel's after long time * feat: ✨ chunked-fs * docs: 📚 chunked fs * docs: 📚 chunked fs
1 parent 0a6df58 commit 3fd2805

14 files changed

Lines changed: 896 additions & 273 deletions

File tree

docs/concepts/nodes.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,9 @@ Defaults to python if nothing is provided. For more information, please refer [c
8383
### retry (optional)
8484
The number of attempts to make before failing the node. Default to 1.
8585

86+
For local executions, this is always be 1 independent of the actual ```retry``` value.
87+
For cloud based implementations, the retry value is passed to the implementation.
88+
8689
### next (required)
8790
The name of the node in the graph to go if the node succeeds.
8891

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# File System Run Log store
2+
3+
This Run Log store stores the run logs on the file system as multiple thread safe files.
4+
5+
It creates a folder with ```run_id``` of the run and stores the contents of the run log in it.
6+
7+
8+
When to use:
9+
10+
- When you want to compare logs between runs.
11+
- During testing/debugging/developing in local environments.
12+
- Especially useful when you have parallel processing of tasks.
13+
14+
15+
When not to use:
16+
17+
- Only Local and Local Container compute modes accept this as a Run Log Store.
18+
19+
## Configuration
20+
21+
The configuration is as follows:
22+
23+
```yaml
24+
run_log:
25+
type: chunked-fs
26+
config:
27+
log_folder:
28+
```
29+
30+
### log_folder
31+
32+
The location of the folder where you want to write the run logs.
33+
34+
Defaults to .run_log_store

docs/concepts/run-log.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -314,6 +314,7 @@ The structure of attempt log along with inline definitions
314314
"end_time": "", # The end time of the attempt
315315
"duration": null, # The duration of the time taken for the command to execute
316316
"status": "",
317+
"parameters": "", # The parameters at that point of execution.
317318
"message": "" # If any exception was raised, this field captures the message of the exception
318319
}
319320
]

0 commit comments

Comments
 (0)