Trigger a production run
Outerbounds base deployment comes with a production orchestrator built on Argo Workflows. This allows developers to deploy entire Metaflow workflows to run in the cloud.
Deploying workflows to the production orchestrator
After developing your workflow and testing it, you can deploy it to the production orchestrator.
python flow.py argo-workflows create
A common pattern is to deploy a workflow after it has been tested in CI. You can read more about continuous delivery here.
Manually trigger workflows from the CLI
To interactively run the deployed workflow from your terminal, run this command:
python flow.py argo-workflows trigger
If you are using Metaflow Parameter
objects in your workflow, you can pass them as arguments to the trigger
command.
python flow.py argo-workflows create --param1 value1 --param2 value2
Manually trigger workflows from the UI
You can also run the equivalent of the trigger
command from the UI. Go to the Deployments
tab, click on the flow you want to run, then click Actions > Trigger.
@schedule workflows to run periodically
To schedule your workflows to run at specific times, you can use the @schedule
flow-level decorator. Before deploying your flow, modify the file:
from metaflow import FlowSpec, step, schedule
@scheudle(hourly=True)
class MyFlow(FlowSpec):
...
You can define the schedule with @schedule
in one of the following ways:
@schedule(weekly=True)
runs the workflow on Sundays at midnight.@schedule(daily=True)
runs the workflow every day at midnight.@schedule(hourly=True)
runs the workflow every hour.@schedule(cron='0 10 * * ? *', timezone='Etc/UTC')
runs the workflow at the given Cron schedule, in this case at 10am UTC every day.
@trigger workflows to run on events
Workflows can also be triggered to run in response to events. Similarly, to the previous section, we modify one line of the workflow to prior to deploying:
from metaflow import FlowSpec, step, trigger
@trigger(events=['event1', 'event2'])
class MyFlow(FlowSpec):
...
To populate the event bus - which Outerbounds manages for you - the following code snippet can be run anywhere you have authenticated to Outerbounds.
from metaflow.integrations import ArgoEvent
ArgoEvent(name="event1").publish()
For advanced deployment patterns, especially in large teams, you can leverage Metaflow namespaces to create many parallel deployments, each listening to the event bus within a specific namespace.
Exogenous events
Outerbounds can help you send events directly from external systems. For example, you can trigger a workflow to run when a new file is uploaded to a cloud storage bucket or when new rows are added to a Snowflake table. Reach out to us in your companies Slack channel with Outerbounds for more information if you have questions about setting up a more-specific eventing system.
@trigger_on_finish to run on completion of another workflow
Workflows can also be triggered to run in response to the completion of another workflow. Again, we modify one line of the workflow to prior to deploying:
from metaflow import FlowSpec, step, trigger_on_finish
@trigger_on_finish(flow='MyOtherFlow')
class MyFlow(FlowSpec):
...
You can also depend on multiple flows completing before starting a flow. Simply define a list of flows:
@trigger_on_finish(flows=['FirstFlow', 'AnotherFlow'])
Each flow needs to complete within a configured time windows for the flow to trigger.