Speed Up Conda

Question

Why is conda "bootstrapping environment" for so long? Can I speed this up?

Solution

When you run a flow for the first time, each conda environment you define needs to gather and install the dependencies which can take a long time. This will be faster the second time you run the flow.

To speed up conda, you can install mamba which is a reimplementation of conda's package manager in C++.

To install mamba:

1. Install conda. Here is the command to download on MacOS or Linux. You can find Windows installation instructions here.

bash Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

2. Install mamba.

conda install mamba -n base -c conda-forge

3. Set Metaflow's dependency resolver environment variable.

export METAFLOW_CONDA_DEPENDENCY_RESOLVER=mamba

You can also set the METAFLOW_CONDA_DEPENDENCY_RESOLVER environment variable in your Metaflow config which by default lives at $HOME/.metaflowconfig/config.json.

Further Reading

• Managing dependencies with Metaflow
• Single flow, multiple developers