Set Environment Variables with .env File
Question
How can I use environment variables in a step of my Metaflow flow with .env
file?
Solution
One way you can read environment variables in Python is using the PyPi module python_dotenv
to load a .env
file. You can customize your environment variables as desired by modifying your .env
file. If you don't need to use a .env
file to manage environment variables, you may consider using Metaflow's environment decorator.
1Create .env File
This .env
file will be used in this example:
.env
SUPER_DUPER_SECRET=hi
2Run Flow
The flow shows how to:
- Use
python_dotenv
to load contents of a.env
file in a Python script. - Use a secret from the
.env
file in theread_locally
step. - Use a secret from the
.env
file in theread_in_container
step that runs with Metaflow's@batch
decorator.- Note you can also use remote compute using the
@kubernetes
decorator. - The Dockerfile used to build the image used in the batch step is shown below the flow.
- Note you can also use remote compute using the
set_environment_variables_file.py
from metaflow import FlowSpec, step, batch
from dotenv import load_dotenv
import os
IMAGE = "public.ecr.aws/outerbounds/dotenv:latest"
class EnvVarFlow(FlowSpec):
@step
def start(self):
self.next(self.read_locally, self.read_in_container)
@step
def read_locally(self):
secret = os.getenv("SUPER_DUPER_SECRET")
print(f"secret message: {secret} " + \
"from local environment")
self.next(self.join)
@batch(image=IMAGE, cpu=1)
@step
def read_in_container(self):
secret = os.getenv('SUPER_DUPER_SECRET')
print(f"secret message: {secret} " + \
"from a container")
self.next(self.join)
@step
def join(self, inputs):
self.next(self.end)
@step
def end(self):
pass
if __name__ == "__main__":
load_dotenv()
EnvVarFlow()
Dockerfile
FROM --platform=linux/amd64 python:3
WORKDIR /usr/src/app
COPY .env .env
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir python_dotenv
python set_environment_variables_file.py run