Skip to main content

Trigger a production run

Outerbounds base deployment comes with a production orchestrator built on Argo Workflows. This allows developers to deploy entire Metaflow workflows to run in the cloud.

Deploying workflows to the production orchestrator​

After developing your workflow and testing it, you can deploy it to the production orchestrator. 

python flow.py argo-workflows create
tip

A common pattern is to deploy a workflow after it has been tested in CI. You can read more about continuous delivery here.

Manually trigger workflows from the CLI​

To interactively run the deployed workflow from your terminal, run this command:

python flow.py argo-workflows trigger

If you are using Metaflow Parameter objects in your workflow, you can pass them as arguments to the trigger command. 

python flow.py argo-workflows create --param1 value1 --param2 value2

Manually trigger workflows from the UI​

You can also run the equivalent of the trigger command from the UI. Go to the Deployments tab, click on the flow you want to run, then click Actions > Trigger.

@schedule workflows to run periodically​

To schedule your workflows to run at specific times, you can use the @schedule flow-level decorator. Before deploying your flow, modify the file:

from metaflow import FlowSpec, step, schedule

@scheudle(hourly=True)
class MyFlow(FlowSpec):
    ...

You can define the schedule with @schedule in one of the following ways:

  • @schedule(weekly=True) runs the workflow on Sundays at midnight.
  • @schedule(daily=True) runs the workflow every day at midnight.
  • @schedule(hourly=True) runs the workflow every hour.
  • @schedule(cron='0 10 * * ? *', timezone='Etc/UTC') runs the workflow at the given Cron schedule, in this case at 10am UTC every day.

@trigger workflows to run on events​

Workflows can also be triggered to run in response to events. Similarly, to the previous section, we modify one line of the workflow to prior to deploying:

from metaflow import FlowSpec, step, trigger

@trigger(events=['event1', 'event2'])
class MyFlow(FlowSpec):
    ...

To populate the event bus - which Outerbounds manages for you - the following code snippet can be run anywhere you have authenticated to Outerbounds.

from metaflow.integrations import ArgoEvent

ArgoEvent(name="event1").publish()
tip

For advanced deployment patterns, especially in large teams, you can leverage Metaflow namespaces to create many parallel deployments, each listening to the event bus within a specific namespace.

Exogenous events​

Outerbounds can help you send events directly from external systems. For example, you can trigger a workflow to run when a new file is uploaded to a cloud storage bucket or when new rows are added to a Snowflake table. Reach out to us in your companies Slack channel with Outerbounds for more information if you have questions about setting up a more-specific eventing system.

@trigger_on_finish to run on completion of another workflow​

Workflows can also be triggered to run in response to the completion of another workflow. Again, we modify one line of the workflow to prior to deploying:

from metaflow import FlowSpec, step, trigger_on_finish

@trigger_on_finish(flow='MyOtherFlow')
class MyFlow(FlowSpec):
    ...

You can also depend on multiple flows completing before starting a flow. Simply define a list of flows:

@trigger_on_finish(flows=['FirstFlow', 'AnotherFlow'])

Each flow needs to complete within a configured time windows for the flow to trigger.