Skip to main content

Configuring secrets

Metaflow provides a built-in mechanism, the @secrets decorator for accessing secrets like database passwords securely in tasks. In the case of Outerbounds, the secrets are stored and managed for you. By following the instructions below, you can grant tasks access to specific secrets.

info

By default, secrets are stored in the control plance account operated by Outerbounds. If you'd like to move secrets completely into the data plane account you control, please reach out to Outerbounds support directly.

Integrations view​

In your Outerbounds deployment, navigate to the Integrations view to manage your secrets. Here you can configure custom secrets as key-value pairs, or use any of the various integrations available for popular services like databases, IAM roles, and API keys.

Using secrets​

After you have configured one or more secrets as described above, you can access them in your flows @secrets decorator. During task execution, the secrets are retrieved automatically and made available through environment variables. For instance, if you named a secret my-secret, it will be accessible in your flow as follows:

from metaflow import FlowSpec, step, secrets

class SecretsFlow(FlowSpec):
    @secrets(sources=["outerbounds.my-secret"])
    @step
    def start(self):
        import os
        assert os.environ.get("SECRET_KEY1") == "secret_value1"
        assert os.environ.get("SECRET_KEY2") == "secret_value2"
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    SecretsFlow()

For each specific integration, you can find the relevant code snippet in the form on the Integrations view.

Using custom IAM roles​

If you need to access secrets stored in AWS Secrets Manager using a custom IAM role, you can configure an IAM role integration and reference it in your flow. This is particularly useful when secrets are stored in a different AWS account.

Setting up the IAM role integration​

  1. Navigate to the Integrations view
  2. Set up an AWS IAM integration
  3. Follow the instructions provided to create an IAM role with the appropriate trust policy in your AWS account
  4. Configure the IAM role with permissions to access the secret (e.g., secretsmanager:GetSecretValue and kms:Decrypt for the specific secret ARN and KMS key used to encrypt the secret)
  5. Configure the integration with the role ARN

Using the IAM role with secrets​

Once you've configured the IAM role integration, you can use it with the @secrets decorator by specifying the role parameter:

from metaflow import FlowSpec, step, secrets
import os

class CustomRoleSecretsFlow(FlowSpec):
    @secrets(
        sources=["some/test/secret-name"],
        role="arn:aws:iam::123456789012:role/external-secret-test-role"
    )
    @step
    def start(self):
        # The secret will be available as an environment variable
        print("secret value", os.environ['test_secret'])
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    CustomRoleSecretsFlow()

This approach works for accessing AWS Secrets Manager whether the secrets are in the same AWS account or a different one. Make sure the IAM role has the necessary permissions to access the specific secrets you need. You can also control access through the Secret's resource-based policy as an alternative to IAM roles.