Tracking Results
In the last three episodes of this season, you have created three flows:
RandomForestFlowGradientBoostedTreesFlowParallelTreesFlow
When you run a command like python my_flow.py run, one run of the flow contained in my_flow.py is triggered. If you do this many times there will be many run results for the flow. Metaflow stores all these results for you and makes them accessible via a client API. After this episode, you will be able to access data from flow runs using the client API.
1Check the Success of a Run
You can use the client API to fetch the results of the latest run of each flow type and ensure they ran successfully. Results can be accessed from your favorite IDE or notebook environment.
from metaflow import Flow
for name in ['RandomForestFlow', 'GradientBoostedTreesFlow', 'ParallelTreesFlow']:
run = Flow(name).latest_run
print(run.successful)
2Ways to Fetch a Run
In the previous example, you saw how to use the metaflow.Flow object to access the latest run of that flow like this:
flow_name = 'RandomForestFlow'
flow = Flow(flow_name)
latest_run = flow.latest_run
If you know the id of a specific run, you can access it directly without using the flow.
This is helpful when flows have many runs and you want to access a specific run instead of filtering through many of them.
For example, you can use the id property from the latest RandomForestFlow run. One way you can find a run's id is in the console output when you executed the command: python <FLOW SCRIPT> run.
from metaflow import Run
directly_accessed_run = Run('{}/{}'.format(
flow_name, latest_run.id))
For more examples of searching results with the client API see this guide on How-to filter flows on a condition.
3What Does the Run Contain?
Once you have fetched the run object, you can:
- View the structure of your flow.
- View metadata about the run such as the time it was run, and environment information like the Python and Metaflow versions.
To access the steps of this run you can call run.steps(). This will return a generator that you can iterate over to inspect the steps in your flow. Once you have a step you can inspect the runtime tasks created from the step template like step.tasks(). This also returns a generator. Here is an example showing how to access steps from a run, and then access tasks from the last step:
run = Flow(name).latest_run
steps = list(run.steps())[::-1]
for task in steps[-1].tasks():
print(task)
In addition to observing different versions of your flow and the runs it has produced, you can visualize the metadata of specific runs. You can use this for many things like tracking model hyperparameters, metric scores, and more.
Concretely, you can access any artifact stored using self with <RUN NAME>.data.<ARTIFACT NAME>.
Let's compare the accuracy of the models produced in the RandomForestFlow and the GradientBoostedTreesFlow.
for name in ['RandomForestFlow', 'GradientBoostedTreesFlow']:
run = Flow(name).latest_run
artifacts = run.data
print("Latest {} had model accuracy = {}%".format(name, artifacts.mean))
Bonus: If you are working on a team where multiple people run flows on the same Metaflow deployment you will most likely want to check out the related concept of namespaces.
In this season you how to train models in a flow. You saw how to create flows that can run the model training in parallel. In the final episode saw how to analyze the results using the client API. If you want to go further into the client API, you can find more details about the Metaflow object hierarchy here.
In the next season, you will see how to quickly visualize the results of your flow in the browser and get an introduction to debugging flows when steps fail. See you there!