Checkpoint (Resume from Failure)

Dotflow can resume a workflow from the last completed step after a crash or failure. No server, no database — just storage.

How it works

1. Each completed task saves its output to storage (file, S3, GCS)
2. When resume=True, dotflow checks if a checkpoint exists before executing each task
3. If a checkpoint exists, the task is skipped and its saved output is used as context for the next task
4. If no checkpoint exists, the task executes normally

Requirements

• A fixed workflow_id — so dotflow can find previous checkpoints across executions
• A persistent storage provider — StorageFile, StorageS3, or StorageGCS

Warning

StorageDefault (in-memory) does not persist data. Checkpoints require a persistent storage provider.

Example

Python 3.8+

from dotflow import Config, DotFlow, action
from dotflow.providers import StorageFile


@action
def step_one():
    return {"extracted": True}


@action
def step_two(previous_context):
    return {"transformed": previous_context.storage}


@action
def step_three(previous_context):
    return {"loaded": previous_context.storage}


config = Config(storage=StorageFile())


def main():
    workflow = DotFlow(config=config, workflow_id="my-etl-pipeline")

    workflow.task.add(step=step_one)
    workflow.task.add(step=step_two)
    workflow.task.add(step=step_three)
    workflow.start(mode="sequential", resume=True)

    return workflow


if __name__ == "__main__":
    main()

Execution flow

First run — step_three fails:

graph LR
    A[step_one] -->|completed| B[step_two]
    B -->|completed| C[step_three]
    C -->|failed| D((stop))
    style A fill:#4caf50,color:#fff
    style B fill:#4caf50,color:#fff
    style C fill:#f44336,color:#fff

Second run with resume=True:

graph LR
    A[step_one] -->|skipped| B[step_two]
    B -->|skipped| C[step_three]
    C -->|completed| D((done))
    style A fill:#9e9e9e,color:#fff
    style B fill:#9e9e9e,color:#fff
    style C fill:#4caf50,color:#fff

Supported modes

Mode	Resume support
sequential	Yes
sequential_group	Yes
background	Yes
parallel	No — tasks run independently, no sequential checkpointing

References

• StorageFile
• Manager