Skip to content

Immutable Files Architecture

Principle

Compose files and .env files are immutable - they are never modified by scripts or CI/CD. All customization happens via environment variables passed at runtime.

Architecture

Compose Files (Immutable)

Location: infrastructure/compose/*.yml

Status: ✅ READ-ONLY - Never modified by scripts

How they work: - Use environment variable substitution: ${IMAGE_TAG:-latest} - Read from .env files via env_file directive - Can be overridden by environment variables at runtime

Example: 

image: ${REGISTRY_IMAGE:-registry.gitlab.com/group/project}/service:${IMAGE_TAG:-latest}

Runtime override: 

IMAGE_TAG=abc123 docker compose -f qualification.yml --env-file .env.qual up -d

Environment Files (Generated, Not Modified)

Location: infrastructure/compose/.env.* and .env.*

Status: ✅ GENERATED FROM TEMPLATES - Never modified after generation

Generation: - Created from templates in infrastructure/env-templates/ - Generated by generate-secrets.sh script - Templates use placeholders like REPLACE_ME_STRONG_PASSWORD

After generation: - Files are never modified by scripts - Manual edits are allowed (for external API keys) - CI/CD generates fresh files from templates (doesn't modify existing)

CI/CD Process

Before (❌ Bad - Modified Files)

# OLD: Modified compose file
bash infrastructure/scripts/update-image-tags.sh qual "${IMAGE_TAG}"
docker compose -f qualification.yml up -d

Problem: Compose file gets modified with hardcoded tags

After (✅ Good - Uses Environment Variables)

# NEW: Uses environment variables
export IMAGE_TAG="${CI_COMMIT_SHORT_SHA}"
IMAGE_TAG="${IMAGE_TAG}" docker compose -f qualification.yml --env-file .env.qual up -d

Benefit: Compose file stays unchanged, uses environment variable

Scripts That Generate Files (Allowed)

These scripts generate files from templates (OK):

  1. generate-secrets.sh
  2. Generates .env.shared, .env.qualification, .env.production from templates

  3. ✅ Allowed - creates new files from templates

  4. CI/CD .env.qual generation

  5. Generates .env.qual from env.qualification.ci template
  6. ✅ Allowed - creates new file from template

Scripts That Modify Files (Not Allowed)

These scripts modify existing files (NOT ALLOWED):

  1. update-image-tags.sh ❌
  2. Status: Deprecated for manual use
  3. CI/CD: Still used but should be removed

  4. Action: Use environment variables instead

  5. Any script using sed on compose files ❌

  6. Status: Not allowed
  7. Action: Use environment variables

Best Practices

For Manual Deployments

# 1. Ensure .env.qual has IMAGE_TAG=latest
grep IMAGE_TAG .env.qual

# 2. Use environment variable (optional override)
IMAGE_TAG=latest docker compose -f qualification.yml --env-file .env.qual up -d

# 3. Compose file uses ${IMAGE_TAG:-latest} which reads from .env.qual

For CI/CD Deployments

# 1. Export IMAGE_TAG as environment variable
export IMAGE_TAG="${CI_COMMIT_SHORT_SHA}"

# 2. Pass to docker compose (overrides .env file)
IMAGE_TAG="${IMAGE_TAG}" docker compose -f qualification.yml --env-file .env.qual up -d

# 3. Compose file uses ${IMAGE_TAG:-latest} which reads from environment

For Script Development

✅ DO: - Read compose files - Read .env files - Generate new files from templates - Use environment variables for customization

❌ DON'T: - Modify compose files with sed or similar - Modify .env files after generation - Hardcode values in compose files - Use update-image-tags.sh for manual deployments

Verification

Check Compose Files Are Immutable

# Check git status - compose files should never show as modified
git status infrastructure/compose/*.yml

# Should show: nothing (or only intentional changes)

Check .env Files Are Generated, Not Modified

# .env files should be generated from templates
diff <(grep -v "^#" infrastructure/env-templates/env.qualification.template | sort) \
     <(grep -v "^#" infrastructure/compose/.env.qualification | sort)

# Differences should only be:
# - Generated secrets (passwords)
# - Manual edits (external API keys)

Verify Environment Variable Usage

# Compose files should use ${VARIABLE:-default} syntax
grep -E '\$\{[A-Z_]+\}' infrastructure/compose/qualification.yml

# Should show variables like:
# ${REGISTRY_IMAGE:-...}
# ${IMAGE_TAG:-latest}
# ${POSTGRES_PASSWORD_QUAL}

Migration Guide

If you have scripts that modify files:

  1. Identify the modification 

    # Find scripts that modify compose files
grep -r "sed.*qualification.yml\|sed.*production.yml" infrastructure/scripts/

  2. Replace with environment variables 

    # Instead of: sed -i 's/IMAGE_TAG=.*/IMAGE_TAG=new/' file
# Use: export IMAGE_TAG=new

  3. Update CI/CD 

    # Instead of: bash update-image-tags.sh qual "${TAG}"
# Use: IMAGE_TAG="${TAG}" docker compose ...

See Also