Skip to content

Fix IMAGE_TAG and Authentication Issues

Problem Analysis

Issue 1: Authentication Error

Error: Head "https://registry.gitlab.com/v2/.../manifests/88a9fcce": unauthorized
Cause: Docker login to GitLab Container Registry has expired.

Issue 2: IMAGE_TAG Being Changed Dynamically

Cause: The CI/CD pipeline (.gitlab-ci/infrastructure.yml) calls update-image-tags.sh which modifies the compose file directly, replacing ${IMAGE_TAG:-latest} with hardcoded commit SHA tags like 88a9fcce.

What happens: 1. CI/CD runs update-image-tags.sh qual 88a9fcce 2. Script modifies qualification.yml, replacing ${IMAGE_TAG:-latest} with :88a9fcce 3. When you run docker compose manually, it reads the modified compose file with hardcoded tags 4. .env.qual still has IMAGE_TAG=latest, but compose file ignores it

Solutions

Fix 1: Restore Compose File (On VPS)

# On VPS
cd /opt/po-platform

# Option A: Restore from git (recommended)
git checkout infrastructure/compose/qualification.yml

# Option B: Use restore script
./infrastructure/scripts/restore-compose-file.sh qual

# Verify it uses IMAGE_TAG variable
grep "image:.*IMAGE_TAG" infrastructure/compose/qualification.yml | head -3

Fix 2: Re-authenticate to Registry (On VPS)

# On VPS
docker login registry.gitlab.com -u <your-username> -p <your-personal-access-token>

# Or use the helper script
./infrastructure/scripts/docker-registry-login.sh

Fix 3: Lock IMAGE_TAG in .env.qual

The .env.qual file should have: 

IMAGE_TAG=latest

To ensure it stays locked:

# On VPS - Make .env.qual read-only (prevents accidental changes)
chmod 444 .env.qual

# Or add a comment in the file
# IMAGE_TAG=latest  # DO NOT CHANGE - Locked to 'latest' for manual deployments

Fix 4: Prevent CI/CD from Modifying Compose File

For manual deployments, ensure CI/CD doesn't run:

The CI/CD pipeline only runs on git push. For manual deployments: 1. Don't push changes that trigger CI/CD 2. Or modify CI/CD to skip update-image-tags.sh for manual deployments

Better approach: Use environment variable instead of modifying compose file:

# Instead of modifying compose file, just set IMAGE_TAG environment variable
export IMAGE_TAG=latest
docker compose -f infrastructure/compose/qualification.yml --env-file .env.qual up -d

Complete Fix Procedure

# On VPS
cd /opt/po-platform

# 1. Restore compose file
git checkout infrastructure/compose/qualification.yml

# 2. Verify .env.qual has IMAGE_TAG=latest
grep IMAGE_TAG .env.qual

# 3. Re-authenticate
docker login registry.gitlab.com -u <username> -p <token>

# 4. Deploy
docker compose -f infrastructure/compose/qualification.yml --env-file .env.qual up -d

Prevention

  1. Never run update-image-tags.sh manually - it modifies compose files
  2. Use environment variables - Set IMAGE_TAG=latest in .env.qual and let compose file use ${IMAGE_TAG:-latest}
  3. Keep compose files in git - Restore from git if modified
  4. Document manual deployment process - Use IMAGE_TAG=latest always for manual deployments

Verification

After fixes, verify:

# 1. Compose file uses variable
grep "image:.*IMAGE_TAG" infrastructure/compose/qualification.yml

# Should show: image: ${REGISTRY_IMAGE:-...}/service:${IMAGE_TAG:-latest}

# 2. .env.qual has IMAGE_TAG=latest
grep IMAGE_TAG .env.qual

# Should show: IMAGE_TAG=latest

# 3. Docker is authenticated
docker pull registry.gitlab.com/portugalodissey/po-platform/api-gateway:latest

# Should succeed without authentication errors