Secrets Rotation Automation

Automated secrets rotation system for BuyWhere API services.

Overview

This system automates the rotation of secrets across:

• AWS Secrets Manager (ECS deployments)
• Kubernetes secrets (K8s staging/production)

Secrets Managed

Usage

CLI Commands

# Rotate specific service secrets
python scripts/secrets_rotation.py rotate --service database --provider aws

# Rotate all secrets (dry run)
python scripts/secrets_rotation.py rotate-all --dry-run

# Rotate all secrets (live)
python scripts/secrets_rotation.py rotate-all

# Check secrets status
python scripts/secrets_rotation.py status

# Restart services after rotation
python scripts/secrets_rotation.py restart

GitHub Actions

The workflow runs on:

• Schedule: 1st of every month at midnight UTC
• Manual trigger: Via workflow_dispatch with options

Rotation Process

1. Generate new secret values using cryptographically secure random generation
2. Update AWS Secrets Manager with new values
3. Update Kubernetes secrets in staging and production namespaces
4. Restart ECS services to pick up new secrets (force new deployment)
5. Restart K8s deployments (rollout restart)
6. Health check verifies services are operational

AWS IAM Permissions Required

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:PutSecretValue",
        "secretsmanager:ListSecrets"
      ],
      "Resource": "arn:aws:secretsmanager:ap-southeast-1:*:secret:buywhere/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecs:UpdateService",
        "ecs:DescribeServices",
        "ecs:ListServices"
      ],
      "Resource": "*"
    }
  ]
}

Kubernetes RBAC Required

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secrets-rotation
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "patch", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "patch"]

Environment Variables

Security Considerations

• All secrets are generated using Python's secrets module (cryptographically secure)
• Old secret values are overwritten in place
• Service restarts ensure no downtime during rotation
• Health checks verify operational status after rotation
• Rotation logs are maintained for audit trail

Troubleshooting

Rotation Fails

1. Check AWS credentials are valid and not expired
2. Verify IAM permissions for Secrets Manager
3. Ensure ECS service is not in a failing state
4. Check K8s secrets exist in the correct namespace

Services Don't Pick Up New Secrets

1. Verify rollout restart completed successfully
2. Check ECS service has force-new-deployment triggered
3. Verify pods are running with new secret values
4. Check application logs for authentication errors

Automation Schedule

Links

• BUY-2884 - Issue tracking this implementation