Skip to main content

Define Lists as Parameters

Question​

How do I define Python list objects as parameters of a flow?

Solution​

The Parameter documentation says you can "specify one of str, float, int, bool, or JSONType". This page will show how you can define lists using the str parameter type or with the multiple option in the parameter definition.

1String List Parameter​

1aWrite a Flow​

This flow defines a Parameter called my_values in the flow code and vals as a command line argument. The default is set to the string '1,2,3'. Notice the separator argument to the Parameter constructor. The separator is eventually passed as an argument to Python's str.split function when the my_values constant is assigned a value. In this case the value of my_values is ['1', '2', '3']. The elements of the list can be mapped into any type you want as the start step shows.

define_list_as_str_param.py
from metaflow import FlowSpec, step, Parameter

class ListStringParamFlow(FlowSpec):
    
    my_values = Parameter("vals", default = '1,2,3', separator = ',')
    
    @step
    def start(self):
        self.int_data = list(map(int, self.my_values))
        self.next(self.end)
        
    @step
    def end(self):
        print(self.int_data)

if __name__ == "__main__":
    ListStringParamFlow()

1bRun Flow with Default Parameters​

python define_list_as_str_param.py run
     Workflow starting (run-id 1666829547854560):
     [1666829547854560/start/1 (pid 76010)] Task is starting.
     [1666829547854560/start/1 (pid 76010)] Task finished successfully.
     [1666829547854560/end/2 (pid 76013)] Task is starting.
     [1666829547854560/end/2 (pid 76013)] [1, 2, 3]
     [1666829547854560/end/2 (pid 76013)] Task finished successfully.
     Done!

1cPass Parameter Values to the Flow​

python define_list_as_str_param.py run --vals '4,5,6'
     Workflow starting (run-id 1666829549149463):
     [1666829549149463/start/1 (pid 76030)] Task is starting.
     [1666829549149463/start/1 (pid 76030)] Task finished successfully.
     [1666829549149463/end/2 (pid 76033)] Task is starting.
     [1666829549149463/end/2 (pid 76033)] [4, 5, 6]
     [1666829549149463/end/2 (pid 76033)] Task finished successfully.
     Done!

2List Multiple Parameter Values​

In the previous section, you saw how to pass many values into a flow as single parameter values using strings. In this section, you will see how to reuse the same parameter name while passing many parameter values in their native type. This lets you use the values in flows without extra typecasting, and can make it easier to read flow run commands.


2aWrite a Flow​

This flow defines a Parameter called my_values in the flow code and vals as a command line argument. The parameter definition includes setting multiple=True, which means we can pass multiple values to this parameter at flow run time. In this case, the value of my_values is [2] by default.

define_multiple_params.py
from metaflow import FlowSpec, step, Parameter

class ListMultipleParamFlow(FlowSpec):
    
    my_values = Parameter("val", default = 2, multiple = True)

    @step
    def start(self):
        self.next(self.end)
        
    @step
    def end(self):
        print(list(self.my_values))

if __name__ == "__main__":
    ListMultipleParamFlow()

2bRun Flow with List of Parameter Values​

To run the flow and access a list of values in self.my_values, you can then pass as many values as you want to the parameter name in the run.

python define_multiple_params.py run --val 1 --val 2 --val 3
     Workflow starting (run-id 1666829550543713):
     [1666829550543713/start/1 (pid 76050)] Task is starting.
     [1666829550543713/start/1 (pid 76050)] Task finished successfully.
     [1666829550543713/end/2 (pid 76053)] Task is starting.
     [1666829550543713/end/2 (pid 76053)] [1, 2, 3]
     [1666829550543713/end/2 (pid 76053)] Task finished successfully.
     Done!

Further Reading​