Skip to main content

List Flow Steps with Client API

Question​

How to list all steps of a flow using the Client API?

Solution​

You can use the Client API to access by:

  • Flows and Runs (instances of Flows)
  • Steps and Tasks (instances of Steps)

You can also find more about the Client API in Metaflow's documentation.

1Run Flow​

This flow shows five steps. There is one data artifact that has its state changed in each of the first four steps. After running the flow you will see how to use the Client API to access the results of the tasks.

list_steps_flow.py
from metaflow import FlowSpec, step

class ListStepsFlow(FlowSpec):
    
    @step
    def start(self):
        self.art = 1
        self.next(self.a)
    
    @step
    def a(self):
        self.art = 2
        self.next(self.b)
        
    @step
    def b(self):
        self.art = 3
        self.next(self.c)
        
    @step
    def c(self):
        self.art = 5
        self.next(self.end)
        
    @step
    def end(self):
        pass
    
if __name__ == "__main__":
    ListStepsFlow()
python list_steps_flow.py run
     Workflow starting (run-id 1659644592744103):
     [1659644592744103/start/1 (pid 20429)] Task is starting.
     [1659644592744103/start/1 (pid 20429)] Task finished successfully.
     [1659644592744103/a/2 (pid 20432)] Task is starting.
     [1659644592744103/a/2 (pid 20432)] Task finished successfully.
     [1659644592744103/b/3 (pid 20435)] Task is starting.
     [1659644592744103/b/3 (pid 20435)] Task finished successfully.
     [1659644592744103/c/4 (pid 20438)] Task is starting.
     [1659644592744103/c/4 (pid 20438)] Task finished successfully.
     [1659644592744103/end/5 (pid 20441)] Task is starting.
     [1659644592744103/end/5 (pid 20441)] Task finished successfully.
     Done!

2Access Step Data​

This code snippet shows how to use the Client API to:

  • Gather all steps from the latest run of ListStepsFlow.
    • Note that Flow(flow_name).latest_run will return a generator that can be converted to a list.
  • Print the step name and artifact state at that step. 
from metaflow import Flow
msg = "Step {:5s} has data artifact `art` with value = {}"
steps = list(Flow('ListStepsFlow').latest_run)
for step in steps[::-1]:
    step_name = step.pathspec.split('/')[-1]
    artifact_value = step.task.data.art
    print(msg.format(step_name, artifact_value))
    Step start has data artifact `art` with value = 1
    Step a     has data artifact `art` with value = 2
    Step b     has data artifact `art` with value = 3
    Step c     has data artifact `art` with value = 5
    Step end   has data artifact `art` with value = 5

Further Reading​