Skip to main content

Reuse Parameters Across Flows

Question​

How can I reuse parameters across multiple Metaflow flows?

Solution​

One way is to return metaflow.Parameter objects from a function. Then you can call this function at the beginning of your flows. Another way is to use a mixin class that includes parameter definitions. This is useful when the reused parameter relates to other functionality you want multiple flows to inherit from. For example, if you have a common dataset that requires the same transformation across several flows, then you may want to group the dataset location parameter and the function that defines transformation logic in the same mixin class.

The rest of this page walks through examples of each method.

1aCreate a Function with Common Parameters​

First you will see a function parameterize_flow being defined in shared_params.py. This highlights the first case where we want to reuse parameters and do not care about coupling the parameter definitions with other common functionality. The function can then be imported and used to define parameters in a flow.

shared_params.py
from metaflow import Parameter

def parameterize_flow():
    
    param_a = Parameter("a", default = 11)
    param_b = Parameter("b", default = 77)
    return param_a, param_b

1bImport Common Parameters in a Flow​

This flow shows how to:

  • Import the function that adds the Parameters to a flow.
  • Use the data the parameters imported in steps of the flows.
reuse_parameters_across_flows.py
from metaflow import FlowSpec, step, Parameter
from shared_params import parameterize_flow

class ReuseParameters(FlowSpec):
    
    param_a, param_b = parameterize_flow()
    
    @step
    def start(self):
        self.next(self.end)
    
    @step
    def end(self):
        print("FlowA.param_a is {}".format(self.param_a))
        print("FlowA.param_b is {}".format(self.param_b))
    
if __name__ == "__main__":
    ReuseParameters()

1cRun the Flow​

python reuse_parameters_across_flows.py run
     Workflow starting (run-id 1663274891128793):
     [1663274891128793/start/1 (pid 40538)] Task is starting.
     [1663274891128793/start/1 (pid 40538)] Task finished successfully.
     [1663274891128793/end/2 (pid 40544)] Task is starting.
     [1663274891128793/end/2 (pid 40544)] FlowA.param_a is 11
     [1663274891128793/end/2 (pid 40544)] FlowA.param_b is 77
     [1663274891128793/end/2 (pid 40544)] Task finished successfully.
     Done!

2aCreate Mixin with Common Parameters​

In this section you will see an example using a mixin class. The mixin approach helps with parameter reuse when the parameters are coupled with additional functionality you want to reuse across flows. This code defines a minimal example of a mixin class that a flow will inherit from. To see this pattern in a more realistic use case you can view the mixin classes and flows from a project using Metaflow with the text-to-image model Stable Diffusion from Stability AI.

shared_params_and_functionality.py
from metaflow import Parameter

class FlowMixin:

    param_a = Parameter("a", default = 11)
    param_b = Parameter("b", default = 77)

    @property
    def model_config(self):
        return {"a": self.param_a, "b": self.param_b}

2bCreate a Flow that Inherits from the Mixin​

Now you can implement a flow that inherits from metaflow.FlowSpec and the custom FlowMixin you just defined. In this example the parameters and functionality of FlowMixin will be inherited by ReuseParametersMixin. This is visible in the end step where self.model_config is called.

reuse_parameters_across_flows_mixin.py
from metaflow import FlowSpec, step, Parameter
from shared_params_and_functionality import FlowMixin

class ReuseParametersMixin(FlowSpec, FlowMixin):
    
    @step
    def start(self):
        self.next(self.end)
    
    @step
    def end(self):
        print(self.model_config)
    
if __name__ == "__main__":
    ReuseParametersMixin()

2cRun the Flow​

python reuse_parameters_across_flows_mixin.py run
     Workflow starting (run-id 1663275411181645):
     [1663275411181645/start/1 (pid 40942)] Task is starting.
     [1663275411181645/start/1 (pid 40942)] Task finished successfully.
     [1663275411181645/end/2 (pid 40946)] Task is starting.
     [1663275411181645/end/2 (pid 40946)] {'a': 11, 'b': 77}
     [1663275411181645/end/2 (pid 40946)] Task finished successfully.
     Done!

Further Reading​