Configuring secrets with AWS Secret Manager

info

This document is if you are specifically looking to use AWS secrets manager. If you are looking for the standard secrets management solution on Outerbounds, read Configuring secrets first.

Metaflow provides a built-in mechanism, the @secrets decorator for accessing secrets like database passwords securely in tasks. In the case of Outerbounds, the secrets are stored and managed for you. By following the instructions below, you can grant tasks access to specific secrets.

In the case of OBP, the secrets can be stored and managed by AWS Secrets Manager on your account. By following the instructions below, you can grant tasks executing on OBP access to specific secrets.

1. Add secrets in AWS Secrets Manager

Add secrets to AWS Secrets Manager as usual. A secret should be a JSON dictionary like here:

{
    "SECRET_KEY1": "secret_value1",
    "SECRET_KEY2": "secret_value2"
}

Let's say we store this secret as basic-secret-kv - you can choose any name. Users will refer to specific secrets using the name.

2. Attach a tag to allow OBP tasks access a specific secret

Add the following tag to the secret object in your AWS Secret Manager console:

• Name: outerbounds.com/accessible-by-deployment
• Value: <deployment_name> (replace with the name of your deployment, e.g. speedyhawk)

tip

Your deployment name can be inferred from the Outerbounds UI URL. It is of the format, ui.<deployment_name>.obp.outerbounds.com/dashboard.

For instance, in the figure below the deployment name is nephalem:

Using secrets

After you have configured one or more secrets as described above, you can access them in your flows using the @secrets decorator.

During task execution, the secrets are retrieved from Secrets Manager and made available through environment variables.

For instance, we can retrieve the two secrets behind basic-secret-kv like here:

from metaflow import FlowSpec, step, secrets

class SecretsFlow(FlowSpec):
    @secrets(sources=["basic-secret-kv"])
    @step
    def start(self):
        import os
        assert os.environ.get("SECRET_KEY1") == "secret_value1"
        assert os.environ.get("SECRET_KEY2") == "secret_value2"
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    SecretsFlow()

Using non-JSON secrets

In some special cases, you may not be able to store a secret as a JSON object. Let's say a secret named my-secret-plain contains an arbitrary non-JSON string. You can expose it through an environment variable as follows:

from metaflow import FlowSpec, step, secrets

class SpecialSecretFlow(FlowSpec):
    @secrets(sources=[{"id": "my-secret-plain", "options": {"json": False} ])
    @step
    def start(self):
        import os
        assert os.environ.get("my_secret_plain") == "secret_value1"
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    SpecialSecretFlow()

In this case, the name of the environment variable is derived from the secret name by replacing special characters with underscores. Here, my-secret-plain becomes my_secret_plain.