Secrets management is a part of most applications and cloud systems. An application that does not require a single secret is rare.

Secrets comes in different forms:

• Key/value secrets

• Certificates and private keys

• API keys and tokens

• Database credentials

• Cloud provider credentials

In a sense, most secrets can be reduced to a string (e.g. using base64 encoding).

The main service offering for secrets management on AWS is AWS Secrets Manager. In this blog post we will learn about what the AWS Secrets Manager is, how you can configure and work with it using Terraform, and learn about the best practices for this service.

What is AWS Secrets Manager?

AWS Secrets Manager is the native service for secrets management on AWS.

The AWS Secrets Manager service primarily concerns secrets in the form of key/value (JSON) or plaintext. As mentioned earlier, by utilizing base64 encoding you could store any type of secret in AWS Secrets Manager.

A few of the important features of AWS Secrets Manager are:

• Automatic scheduled secret rotation using a custom AWS Lambda function.

• Secret replication across AWS regions to increase availability in case of regional issues.

• Secret versioning allows you to go back to previous values of the secrets if needed.

• Secrets can be configured with a retention window where they can be restored after they have been deleted.

• Supports custom KMS keys to encrypt the secret values.

We will see how to configure a few of these features using Terraform in the next section.

AWS Secrets Manager is not a full-fledged secrets management system that you can use as the main secret management service in your multi-cloud strategy. However, for infrastructure running on AWS this is the most convenient option for secret management.

At the time of writing there are 55 other AWS services that integrate with AWS Secrets Manager in some capacity. For these services you will often get a lot of the secrets management automated, removing the need for you to manage the secrets actively. One example is AWS RDS service which automates the rotation of database credentials using AWS Secrets Manager.

Managing AWS Secrets Manager using Terraform

Terraform allows you to configure anything related to AWS Secrets Manager service. Below we will go through how to configure the resources that are important.

The basic secret is created using the aws_secretsmanager_secret resource. An example of this resource looks like this:

resource "aws_secretsmanager_secret" "static" {
  name_prefix             = "anyshift-static-secret"
  description             = "My static secret"
  recovery_window_in_days = 0
  replica {
    region = "eu-west-2"
  }
}

This resource does not contain any secret values. Instead, the aws_secretsmanager_secret resource is the metadata for the secret. You can also configure replication, secret access policy and encryption details.

The recovery_window_in_days is for how long the secret can be recovered after it has been deleted. The default value is 30 days, but it can be configured between 7 and 30 days. If you do not wish to have a recovery window you can set the argument to 0. This is a good option for experimentation and testing where you want to be able to clean up your AWS environment when you are done.

Two other important attributes of the secret resource are the name and name_prefix. You have to configure one of them, but not both.

If you set the name attribute your secret will have that exact name. If you set the name_prefix attribute the secret will start with the prefix that you specify, followed by a timestamp string similar to 20250401103434744600000001.

The purpose of the suffix is to make the secret name unique. If you delete a secret that has a recovery window between 7 and 30 days, then you will not be able to create a new secret using the same name during this window. This would result in errors from Terraform.

To give your secret an actual value you must create a secret version. This is done using the aws_secretsmanager_secret_version resource type. A basic example that sets a static value:

resource "aws_secretsmanager_secret_version" "plaintext" {
  secret_id     = aws_secretsmanager_secret.static.id
  secret_string = "my-static-secret-value"
}

This will create a static plaintext secret value. You can also specify static key/value pairs by passing in a JSON-encoded value:

resource "aws_secretsmanager_secret_version" "keyvalue" {
  secret_id = aws_secretsmanager_secret.static.id
  secret_string = jsonencode({
    username = "static-secret-username"
    password = "static-secret-password"
  })
}

It is clear that it is a bad idea to provide static values for your secrets directly in the Terraform code. What better options are there? Luckily, there are a few options to consider!

A simple approach is to use the Terraform random provider to generate a random string and set that as the secret value (you could also use the random_password resource type):

resource "random_string" "db" {
  length = 50
}
resource "aws_secretsmanager_secret_version" "plaintext" {
  secret_id     = aws_secretsmanager_secret.static.id
  secret_string = random_string.db.result
}

To avoid using additional Terraform providers you can also use a data source offered by the AWS Secrets Manager service called aws_secretsmanager_random_password:

data "aws_secretsmanager_random_password" "db" {
  password_length            = 50
  exclude_characters         = false
  exclude_lowercase          = false
  exclude_uppercase          = false
  exclude_numbers            = true
  exclude_punctuation        = true
  include_space              = false
  require_each_included_type = true
}

You do not need to configure the data source with all the arguments shown in the example above, but they show the configuration options that are available to you.

You can reference the result of the data source in the secret version resource:

resource "aws_secretsmanager_secret_version" "plaintext" {
  secret_id     = aws_secretsmanager_secret.static.id
  secret_string = data.aws_secretsmanager_random_password.db.random_password
}

The previous two options both generate a random password that is then set as the secret_string argument for the secret version. These two options produce secrets that are persisted to your Terraform state file. If this is unacceptable in your environment you can use one of the following two options.

Terraform 1.10 and 1.11 have brought new features that allow you to restrict certain values from being persisted to the Terraform state. These features are available for the secret version resource. To use them, set the secret_string_wo (wo for write-only) instead of configuring the secret_string argument (note, trying to configure both will result in an error):

resource "aws_secretsmanager_secret_version" "wo" {
  secret_id = aws_secretsmanager_secret.static.id
  secret_string_wo = "this-value-will-not-be-persisted-in-the-state"
  secret_string_wo_version = 1
}

You must also configure the secret_string_wo_version argument. The purpose of the version number is to tell Terraform when to update the value of the secret version. If the version remains the same, Terraform will not update the resource even if you provide a new secret value in the secret_string_wo argument.

It's still a bad idea to write the secret string directly in your Terraform files. You can instead pass in the secret value using ephemeral variables:

variable "password" {
  type      = string
  ephemeral = true
}

Configuring a variable as ephemeral tells Terraform to not persist this value to the state file.

Another option to avoid storing secret values in the state file is to set up secret rotation. With secret rotation you set an initial value for the secret value using any of the methods outlined above, but this value will then be rotated to a new value according to a schedule.

You set up secret rotation using the aws_secretsmanager_secret_rotation resource type:

resource "aws_secretsmanager_secret_rotation" "example" {
  secret_id           = aws_secretsmanager_secret.static.id
  rotation_lambda_arn = aws_lambda_function.rotation_lambda.arn
  rotate_immediately  = true
  rotation_rules {
    schedule_expression = "cron(0 /4 * * ? *)"
  }
}

You configure rotation_rules for how the secret is rotated. You can use two different types of rotation rules:

• schedule_expression allows you to set a cron expression for when rotations take place. This is the most flexible option. The cron expression in the example above specifies that the secret should be rotated every four hours.

• automatically_after_days allows you to specify for how many days the secret should live before it is rotated.

The actual rotation is handled by an AWS Lambda function that you need to set up as well. You provide the ARN to the Lambda function in the rotation_lambda_arn argument. This may seem like an unnecessary dependency, but it allows for full flexibility in how the secret is rotated. This is good news if you need to interact with multiple third-party systems for notifications and coordination when you rotate a secret. Through the AWS Lambda function you can do any customization to the process that you require.

How you configure the AWS Lambda function to handle secret rotation is outside the scope of this blog post. You can read more about this in the AWS documentation.

When you set up secret rotation, make sure to also specify that the secret should be rotated immediately using the rotate_immediately argument. This is to avoid an initial period where the secret value you set using Terraform is active.

There is one last important resource in the AWS Secret Manager ecosystem, the secret policy. 

If you need to share a secret between AWS accounts you must use secret policies. These are similar to other resource policies on AWS (e.g. for SNS topics or S3 buckets).

An example of allowing a different AWS account to read the static secret we have been working with so far:

data "aws_iam_policy_document" "sharing" {
  statement {
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::123XXXXXX789:root"]
    }
    actions   = ["secretsmanager:GetSecretValue"]
    resources = ["*"]
  }
}
resource "aws_secretsmanager_secret_policy" "sharing" {
  secret_arn = aws_secretsmanager_secret.static.arn
  policy     = data.aws_iam_policy_document.example.json
}

With secret policies you can securely share secrets in your overall AWS environment without having to replicate secret values across accounts.

Other native options for secrets management on AWS

If you want native options for managing secrets on AWS there is only one other option apart from AWS Secrets Manager.

The AWS Systems Manager (SSM) service supports parameter resources. Parameters can be used for key/value data that you want to access from services within SSM and beyond.

Specifically for secrets you can create a new aws_ssm_parameter resource type and configure it to be a SecureString:

resource "aws_ssm_parameter" "secret" {
  name        = "/environments/prod//db/password"
  description = "The production database password"
  type        = "SecureString"
  value       = "s3cr3tp4ssw0rd"
  tags = {
    environment = "prod"
    owner = "platform"
  }
}

Note that the aws_ssm_parameter also supports write-only arguments to avoid storing the provided secret value in the state file. An example of this:

resource "aws_ssm_parameter" "secret" {
  name             = "/environments/prod//db/password"
  description      = "The production database password"
  type             = "SecureString"
  value_wo         = var.secret_value
  value_wo_version = 1
  tags = {
    environment = "prod"
    owner = "platform"
  }
}

This resource is a good option if you require simple password secrets that you set once and do not need all the bells and whistles (e.g. built-in support for automatic rotation with a Lambda function) of the AWS Secrets Manager service.

If you look outside the world of AWS there are many other options for secret management. One example is HashiCorp Vault.

Best practices for AWS Secrets Manager

Avoid using secrets for more than one purpose

A common secrets management best practice is to avoid using the same secret for more than one purpose. If you do use a password for multiple purposes and need to rotate the secret value, you need to be sure that all of the systems using this password work after the rotation. This could take some coordination and luck to get right.

A similar best practice is to avoid repeating the actual secret value across multiple secrets. If one of these secrets is compromised then all of the other secrets with the same value should be considered to be compromised as well.

Set up secret rotation

If possible you should use the automatic rotation feature for your static secrets.

This should be possible for all applications that read the secret from AWS Secrets Manager on a regular basis. The opposite of this would be an application that starts up and reads the secret once, then keeps the value in memory until it shuts down. Even for these applications it should be fine to rotate the secret, as long as the rotation period matches the typical application lifecycle.

Since you set up the AWS Lambda function that performs the secret rotation you could take any actions necessary to make sure the secret rotation is successful. This means the Lambda function could restart any long-running applications that read the secret value at startup to make sure the new value is read.

Some AWS services support managed rotation for secrets. These are typically set up from the service that supports the rotation, and not directly in AWS Secrets Manager itself. One example is the Relational Database Service (RDS).

Limited access to secrets using IAM and resource policies

As with any other AWS service you should carefully restrict access to your secrets in AWS Secrets Manager using IAM policies and resource policies for your secrets.

Resource policies are necessary when you want to allow IAM principals in other AWS accounts to read secrets in your AWS account. This is common in larger organizations where you have multiple AWS accounts. As with any cross-account actions, these must be allowed through IAM policies for principals in the source account, and in resource policies in the target account.

Set up dedicated secrets for an application that only that application has access rights to. Avoid using too broad IAM permissions including the asterisk symbol (e.g. arn:aws:secretsmanager:*).

Use IAM roles to access secrets from applications

Another best practice that is common for most AWS services is that you should interact with AWS Secrets Manager using IAM roles, not AWS access keys. IAM roles is a secret-free solution that builds on IAM policies and trust relationships instead of an access key pair.

This allows you to easily set up roles for different purposes, for instance one role with read access to a secret and a different role for managing the secret (e.g. updating it with new values or updating a rotation schedule).

Use the recovery window wisely

Secrets on AWS Secrets Manager supports a recovery window that you can configure for up to 30 days. This is a safety mechanism that allows you to recover a deleted secret up to the configured recovery window.

Just remember that if you have a deleted secret named my-secret-1, then you will not be able to provision a new secret with the same name my-secret-1 until the recovery window has passed. The workaround for this is to provision secrets using the name_prefix attribute that was shown earlier in this blog post.

Consider how likely it is that you will need to recover a secret that you deleted 30 days ago. In most cases you will discover the need to retrieve the secret much faster than 30 days. However, if you have a batch job that runs every 30 days that needs to read a secret to access a given system then this would not be discovered until the batch job starts.

Monitor secret usage

You can monitor your AWS Secret Manager usage using AWS CloudWatch. You can monitor your total number of secrets and the use of individual secrets. Use these metrics to set up alerting if there is something that is outside of your expected secret usage.

For compliance reasons you should also monitor usage of AWS Secrets Manager using CloudTrail. There are also a number of AWS Config rules you can use to get an overview of how your secrets are configured and if there are unused secrets you could remove.

Finally, AWS Secrets Manager is not an expensive service but you should still closely monitor the cost of this service. You pay a fixed cost for each secret, and an equal cost for each replica of a secret. You also pay a small fee for the number of API calls you do. A rapidly rising cost could indicate you are either storing too many secrets or using the API more than necessary.

Do not hardcode secrets in configuration files

In the walkthrough of how to use Terraform for AWS Secrets Manager above we saw examples of how to configure secret values. Remember that it is a bad practice to hardcode secret values in your Terraform configuration files (or other files).

Provide the secret values to your Terraform configuration through variables. You can set the variable values from a different secret store (e.g. GitHub Actions secrets) or generate them using Terraform code.

To avoid storing the secret values in your Terraform state file you should use the write-only arguments together with ephemeral variables. This tells Terraform to not persist the values to the state file.

Encrypt secrets with a custom KMS

AWS Secrets Manager supports using custom KMS keys to encrypt secret values. Use this feature if you want extra control over how the secrets are encrypted. Note the management overhead that comes with using custom KMS keys. In some environments this is a requirement so there is no way to avoid the extra overhead.

Terraform and Anyshift for AWS Secrets Manager

AWS Secrets Manager is used in many places in your AWS environment. As mentioned, 55 other AWS services integrate with AWS Secrets Manager. On top of this you will use AWS Secret Manager for custom applications and services that you build.

It's this second category of secret users (your custom applications) that are most sensitive for potential misconfiguration through Terraform. An example of this is when you are managing your secrets using a producer Terraform configuration, and using them in multiple other consumer Terraform configurations.

In this situation it is important that you have a robust and secure workflow for introducing changes into your environment. Removing secrets that are in use in other parts of your environment could have great consequences.

An example of this was outlined earlier in this blog post. Imagine we have a batch application that runs on a schedule, perhaps once a week or even once a month. This application consumes a number of secrets stored in AWS Secrets Manager. If these secrets are deleted and have a short recovery window, they might be irrecoverable by the time the next batch job runs.

Anyshift creates a digital twin of your AWS environment with your Terraform state files and Terraform configurations. This allows for contextual insights into how changes to resources in Terraform configuration will affect the rest of your AWS environment. This AWS knowledge graph is a powerful feature that allows Anyshift to have great insight into your current context and can judge changes to your Terraform infrastructure accurately to assess potential impact.

An SRE AI-copilot like Anyshift that can inform you before-the-fact that a change in your Terraform configuration could lead to potential issues in a different Terraform configuration. This is a powerful feature that could reduce the on-call load on your SREs. The only better option to resolving issues in your cloud infrastructure quickly is to avoid the issue to start with.

Visit the documentation for more on Anyshift.

Conclusions

The AWS Secrets Manager service is a fairly simple service for secrets management. It supports secret versioning, replication, sharing across AWS accounts, and automatic secret rotation. For basic secrets management needs this service offers all of what you might need.

A large number of AWS services integrate with AWS Secrets Manager. However, you will most likely integrate your own custom applications with AWS Secrets Manager as well. These custom applications and secrets are in the danger zone when working with multiple Terraform configurations where secrets are produced and consumed in different configurations. Keeping track of where secrets are consumer can be difficult. Bringing in an SRE AI-copilot like Anyshift can be beneficial to avoid making costly mistakes.