The Audit Finding That Changed My View of State Files
A security audit found our Terraform state files in an S3 bucket with public read enabled, and the state contained database passwords, API keys, and VPC peering configurations. I remember staring at the finding in our conference room while our manufacturing network kept humming behind us: PLC traffic, MES integrations, warehouse scanners, vendor VPNs, and a cloud environment that was supposed to be the clean part of our infrastructure story.
My first assumption was wrong. I thought the exposure had to be a recent IAM mistake, maybe a rushed policy change during an application deployment. The S3 bucket was created before the Block Public Access feature existed, and nobody had verified the bucket ACL since the initial configuration. That was my miss as much as anyone else’s, because my team had reviewed Terraform modules, IAM roles, and FortiGate firewall policy on FortiOS 7.4.3, but we had treated the backend bucket like old plumbing.
State is not paperwork.
In our environment, Terraform 1.6.6 had tracked RDS usernames, generated passwords, private subnet IDs, security group relationships, Route 53 records, VPC peering routes, and several outputs marked as operational conveniences. I had seen state files before, but I had not treated them with the same suspicion I give packet captures, firewall backups, or database dumps. That changed immediately.
My opinion now is blunt: a Terraform state file belongs in the same security class as a password vault export.
What I Found Inside Our Terraform State
Once we pulled the exposed objects and reviewed them from an Ubuntu 22.04 jump host with Python 3.11 tooling, the damage potential became obvious. Terraform state did not just list resource names. It described how those resources connected, which accounts trusted which roles, which CIDR blocks could talk across VPC peering, and which outputs had been convenient during buildout but reckless after launch.
The worst part was the plaintext. The word “sensitive” in a Terraform output controls display behavior in the CLI; it does not magically remove the value from state. If a module stores a password, token, secret ARN, or generated key in state, I assume that value is readable by anyone with access to the backend object. I do not care how clean the console output looks.
- Database usernames and generated passwords from RDS modules
- API keys passed into provider or application variables
- Private subnet IDs, route table IDs, and security group IDs
- VPC peering connection IDs and accepted route paths
- IAM role names, trust relationships, and policy attachments
- Terraform outputs marked sensitive but still present in state
Secrets hide in boring files.
What I didn’t expect was how useful the state file would be to someone planning lateral movement. In manufacturing, I worry about paths from business systems into production networks, and the state gave away architecture context that would have taken days to map by probing. A public state file was not just leaked credentials; it was leaked intent.
My opinion is that state review should be part of threat modeling, not an afterthought after infrastructure already works.
How We Rebuilt The S3 Backend
We moved every workspace to an encrypted S3 backend with private ACLs, bucket-level public access blocking, versioning, and DynamoDB state locking. I wanted controls that failed closed, because relying on everyone to remember the sensitivity of a backend bucket is a weak design. We also separated the state bucket from general artifact storage so that no CI job, developer utility, or data export process needed broad permissions against it.
terraform {
required_version = "1.6.6"
backend "s3" {
bucket = "prod-terraform-state-private"
key = "networking/prod/terraform.tfstate"
region = "us-east-1"
encrypt = true
kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/9f2b6c30-1111-4444-8888-7d7b11112222"
dynamodb_table = "prod-terraform-state-locks"
acl = "private"
}
}
resource "aws_s3_bucket_public_access_block" "state" {
bucket = aws_s3_bucket.state.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}The backend policy became small on purpose. My CI role could read and write the exact state prefixes it owned. My security team role could read state for investigation. Human write access required break-glass approval. KMS decrypt permissions matched those same boundaries, because S3 encryption without KMS access discipline is just a checkbox with nice lighting.
Lock the door and count the keys.
State locking with DynamoDB mattered more than I expected. We originally added it to prevent concurrent applies from stepping on each other, but it also gave us a cleaner operational story. When a pipeline failed midway, my team could see lock ownership instead of guessing which engineer had an old shell session open. That reduced risky manual cleanup during production changes.
My opinion is that local state has no place in a shared cloud environment once more than one person can change infrastructure.
Access Control And Audit Logging We Actually Use
After the backend move, I treated state access as a monitored event. We enabled S3 server access logging where it fit, CloudTrail data events for the state bucket, AWS Config rules for public bucket settings, and GuardDuty findings into our existing alert queue. I wanted a noisy first week because silence after a sensitive migration usually means we forgot to instrument something.
You may also find this useful: Check out our guide on Python Network Config Backup: Automating Multi-Vendor Device Snapshots for more practical tips.
The before and after number still sits in my notes: the first audit found 7 publicly readable Terraform state objects, and after moving all state files to encrypted S3 with private ACLs and enabling state locking with DynamoDB, the follow-up audit found zero public state file exposure. That metric did more to change behavior than any architecture diagram I had drawn.
Zero was the only acceptable number.
We also reviewed identity paths. Our Jenkins agents on Ubuntu 22.04 assumed a deployment role with short-lived AWS STS credentials. Engineer laptops did not get permanent backend write access. Emergency access went through an audited role, and we tested it rather than admiring it in IAM. I have learned that an untested break-glass process becomes folklore during an outage.
The access model I prefer is boring: few roles, narrow prefixes, explicit KMS permissions, CloudTrail data events, and automated checks that fail a pipeline if a backend bucket drifts toward public exposure. Fancy diagrams do not compensate for a bucket policy that accepts the world.
Handle Sensitive Outputs Like They Will Leak
Marking Terraform outputs as sensitive is still useful, but I now view it as screen hygiene rather than storage security. It keeps passwords out of terminal logs and CI output, which matters, but the underlying state still carries the value. When my team uses Terraform with AWS Secrets Manager, SSM Parameter Store, or Vault, I push hard to store references and generated identifiers instead of raw secret material whenever the provider and workflow allow it.
I also ask a simple question during review: does this value need to exist in state at all? For database passwords, I prefer rotation managed outside Terraform after initial provisioning, or a pattern where Terraform creates the container and a secret workflow populates the value. For API keys, I prefer manual import into a secrets platform or provider-specific resources that limit plaintext exposure. None of this is perfect, but perfect is not the bar; reducing blast radius is.
Convenience ages badly.
We added checks that parse planned output and state metadata using Python 3.11 scripts in CI, mainly looking for obvious names such as password, token, secret, private_key, and api_key. That control is not magic, and I do not pretend regex can understand every provider schema, but it catches careless outputs before they become permanent artifacts. We also run Terraform 1.6.6 with pinned providers so state behavior changes do not sneak in during routine module work.
My final position is simple: I trust encrypted remote state with private access, locking, versioning, and logging; I do not trust memory, habit, or a one-time setup from years ago. Terraform state security is infrastructure security, and I treat it with the same seriousness as firewall backups from FortiOS 7.4.3 or database exports from production.
Further Reading: For more in-depth information, refer to the official Fortinet Documentation.