Secret Rotation Guide¶

Guide for rotating secrets when services and databases are already running with old passwords.

Overview¶

When you generate new secrets, existing services and databases are still using the old passwords. You have two options:

Migration Approach - Update passwords in services/databases first, then update .env files (no downtime)
Recreate Approach - Stop services, update secrets, recreate everything (downtime, data loss)

⚠️ Important Considerations¶

What Happens If You Just Change `.env` Files?¶

If you update .env files without updating the actual service passwords:

Services will fail to connect - PostgreSQL, Redis, RabbitMQ connections will be rejected
Data remains intact - Database data is not lost, just inaccessible until passwords match
Services restart loop - Containers will keep restarting trying to connect

What Can Be Safely Changed?¶

Safe to change (just restart services): - JWT secrets - Strapi keys - Service-specific secrets - External API keys (Stripe, SMTP, etc.)

Requires migration (update service first): - PostgreSQL password - Redis password - RabbitMQ password - MinIO password - Grafana password

Option 1: Migration Approach (Recommended for Production)¶

Step 1: Update Database Passwords First¶

PostgreSQL¶

# On VPS, connect to PostgreSQL with current password
docker exec -it po-postgres psql -U postgres

# In PostgreSQL shell, change password
ALTER USER postgres WITH PASSWORD 'NEW_PASSWORD_FROM_ENV_SHARED';

# Exit
\q

Redis¶

# On VPS, connect to Redis with current password
docker exec -it po-redis redis-cli -a CURRENT_PASSWORD

# Change password (Redis doesn't support ALTER, need to update config)
# Exit Redis
exit

# Update Redis configuration
docker exec -it po-redis redis-cli -a CURRENT_PASSWORD CONFIG SET requirepass NEW_PASSWORD_FROM_ENV_SHARED

# Or restart Redis with new password in .env file

RabbitMQ¶

# On VPS, connect to RabbitMQ container
docker exec -it po-rabbitmq rabbitmqctl change_password admin NEW_PASSWORD_FROM_ENV_SHARED

MinIO¶

MinIO passwords are set at container startup. You need to:

Stop MinIO container
Update .env.shared with new password
Restart MinIO container

# On VPS
cd /opt/po-platform

# Stop MinIO
docker compose -f infrastructure/compose/shared.yml stop minio

# Update .env.shared with new MINIO_ROOT_PASSWORD
nano infrastructure/compose/.env.shared

# Restart MinIO
docker compose -f infrastructure/compose/shared.yml up -d minio

Step 2: Update Environment Files¶

After updating passwords in services, update .env files:

# On VPS
cd /opt/po-platform

# Update .env.shared with new passwords (matching what you set in services)
nano infrastructure/compose/.env.shared

# Update .env.qual with matching passwords
nano .env.qual

Important: Ensure passwords match: - POSTGRES_PASSWORD in .env.shared = password you set in PostgreSQL - REDIS_PASSWORD in .env.shared = password you set in Redis - RABBITMQ_PASSWORD in .env.shared = password you set in RabbitMQ - MINIO_ROOT_PASSWORD in .env.shared = new MinIO password

Step 3: Restart Services¶

# Restart shared infrastructure
docker compose -f infrastructure/compose/shared.yml restart

# Restart qualification services
docker compose -f infrastructure/compose/qualification.yml restart

Option 2: Recreate Approach (For Fresh Start)¶

If you're okay with downtime and can recreate data:

Step 1: Backup Data (Optional but Recommended)¶

# On VPS
cd /opt/po-platform

# Backup PostgreSQL
docker exec po-postgres pg_dumpall -U postgres > backup_$(date +%Y%m%d).sql

# Backup Redis (if you have important data)
docker exec po-redis redis-cli SAVE
docker cp po-redis:/data/dump.rdb ./redis_backup_$(date +%Y%m%d).rdb

# Backup MinIO data
docker cp po-minio:/data ./minio_backup_$(date +%Y%m%d)

Step 2: Stop All Services¶

# Stop qualification services
docker compose -f infrastructure/compose/qualification.yml down

# Stop shared infrastructure
docker compose -f infrastructure/compose/shared.yml down

Step 3: Update Secrets¶

# Generate new secrets locally
make secrets

# Copy new secrets to VPS
scp infrastructure/compose/.env.shared root@qual.portugalodyssey.pt:/opt/po-platform/infrastructure/compose/
scp infrastructure/compose/.env.qualification root@qual.portugalodyssey.pt:/opt/po-platform/.env.qual

# Fill in external API keys
ssh root@qual.portugalodyssey.pt "cd /opt/po-platform && nano .env.qual"

Step 4: Remove Old Volumes (⚠️ Data Loss)¶

# On VPS
cd /opt/po-platform

# Remove old volumes (THIS DELETES DATA)
docker volume rm po-postgres-data po-redis-data po-rabbitmq-data po-minio-data-qual po-minio-data-prod 2>/dev/null || true

Step 5: Start Services Fresh¶

# Start shared infrastructure with new secrets
docker compose -f infrastructure/compose/shared.yml --env-file infrastructure/compose/.env.shared up -d

# Wait for infrastructure to be ready
sleep 10

# Start qualification services
docker compose -f infrastructure/compose/qualification.yml --env-file .env.qual up -d

Step 6: Restore Data (If Backed Up)¶

# Restore PostgreSQL
docker exec -i po-postgres psql -U postgres < backup_YYYYMMDD.sql

# Restore Redis
docker cp redis_backup_YYYYMMDD.rdb po-redis:/data/dump.rdb
docker restart po-redis

# Restore MinIO
docker cp minio_backup_YYYYMMDD/* po-minio:/data/

Option 3: Hybrid Approach (Minimal Downtime)¶

Best for production - update passwords one service at a time:

1. Update PostgreSQL Password¶

# Connect with current password
docker exec -it po-postgres psql -U postgres

# Change password
ALTER USER postgres WITH PASSWORD 'NEW_PASSWORD';

# Update .env.shared
nano infrastructure/compose/.env.shared  # Set POSTGRES_PASSWORD=NEW_PASSWORD

# Restart only PostgreSQL-dependent services
docker compose -f infrastructure/compose/qualification.yml restart

2. Update Redis Password¶

# Set new password in Redis
docker exec -it po-redis redis-cli -a OLD_PASSWORD CONFIG SET requirepass NEW_PASSWORD

# Update .env.shared
nano infrastructure/compose/.env.shared  # Set REDIS_PASSWORD=NEW_PASSWORD

# Restart Redis-dependent services
docker compose -f infrastructure/compose/qualification.yml restart

3. Update RabbitMQ Password¶

# Change password
docker exec -it po-rabbitmq rabbitmqctl change_password admin NEW_PASSWORD

# Update .env.shared
nano infrastructure/compose/.env.shared  # Set RABBITMQ_PASSWORD=NEW_PASSWORD

# Restart RabbitMQ-dependent services
docker compose -f infrastructure/compose/qualification.yml restart

4. Update MinIO Password¶

# Stop MinIO
docker compose -f infrastructure/compose/shared.yml stop minio

# Update .env.shared
nano infrastructure/compose/.env.shared  # Set MINIO_ROOT_PASSWORD=NEW_PASSWORD

# Restart MinIO
docker compose -f infrastructure/compose/shared.yml up -d minio

# Restart file-service
docker compose -f infrastructure/compose/qualification.yml restart file-service-qual

Verification After Rotation¶

Check Service Connections¶

# Check PostgreSQL
docker exec po-postgres psql -U postgres -c "SELECT 1;"

# Check Redis
docker exec po-redis redis-cli -a NEW_PASSWORD PING

# Check RabbitMQ
docker exec po-rabbitmq rabbitmqctl status

# Check MinIO
curl -u minioadmin:NEW_PASSWORD http://localhost:9000/minio/health/live

Check Service Logs¶

# Check if services are connecting successfully
docker compose -f infrastructure/compose/qualification.yml logs | grep -i "error\|failed\|denied" | tail -20

Common Issues¶

"Password Authentication Failed"¶

Cause: Password mismatch between .env file and actual service password.

Solution: 1. Verify password in .env file matches what you set in the service 2. Restart the service to pick up new password

"Connection Refused"¶

Cause: Service not running or wrong host/port.

Solution: 1. Check service is running: docker ps | grep postgres 2. Verify network configuration 3. Check service logs: docker logs po-postgres

"Database Does Not Exist"¶

Cause: Database was created with old password, now inaccessible.

Solution: 1. Connect with correct password 2. Or recreate database (if data can be regenerated)

Best Practices¶

Always backup before rotation - Even if you think you won't need it
Test in qualification first - Rotate secrets in qual environment before production
Rotate during maintenance window - Plan for potential downtime
Document current passwords - Keep a secure record (password manager) before rotation
Update all related services - Ensure all services using a password are updated
Verify after rotation - Check logs and test connections
Use migration approach for production - Avoid data loss

Quick Reference: Service Dependencies¶

PostgreSQL dependencies: - All backend services (auth-service, notification-service, etc.) - Strapi CMS

Redis dependencies: - Services using caching (if configured)

RabbitMQ dependencies: - notification-service - review-service - Other services using message queues

MinIO dependencies: - file-service