File Storage Options Comparison¶

Overview¶

The file-service supports 4 storage providers. You do NOT need an S3 account - you can use any of these options depending on your needs, budget, and infrastructure preferences.

Quick Answer¶

No, you don't need an S3 account. You have these options:

Local Storage (Free, simplest) - Files stored on the server's filesystem
MinIO (Free, self-hosted) - S3-compatible storage you run yourself
AWS S3 (Paid, managed) - Amazon's cloud storage service
Google Cloud Storage (Paid, managed) - Google's cloud storage service

Detailed Comparison¶

Feature	Local Storage	MinIO (Self-hosted)	AWS S3	Google Cloud Storage
Cost	Free	Free (infrastructure costs only)	Pay-as-you-go (~$0.023/GB/month)	Pay-as-you-go (~$0.020/GB/month)
Setup Complexity	⭐ Very Easy	⭐⭐ Easy	⭐⭐⭐ Moderate	⭐⭐⭐ Moderate
Scalability	Limited by server disk	Limited by server disk	Virtually unlimited	Virtually unlimited
Performance	Fast (local disk)	Fast (local/network disk)	Very Fast (CDN-backed)	Very Fast (CDN-backed)
Backup/Redundancy	Manual	Manual (you manage)	Automatic (99.999999999% durability)	Automatic (99.999999999% durability)
CDN Integration	❌ No	❌ No	✅ Yes (CloudFront)	✅ Yes (Cloud CDN)
Public URLs	❌ No (requires proxy)	✅ Yes	✅ Yes	✅ Yes
Signed URLs	❌ No	✅ Yes	✅ Yes	✅ Yes
Data Location	Your server	Your server	AWS data centers	Google data centers
Compliance	Your responsibility	Your responsibility	SOC 2, ISO 27001, etc.	SOC 2, ISO 27001, etc.
Bandwidth Costs	None	None	$0.09/GB outbound	$0.12/GB outbound
Best For	Development, small projects	Self-hosted, privacy-focused	Production, high scale	Production, Google ecosystem

Option 1: Local Storage (Current Default)¶

Configuration¶

environment:
  - FILE_PROVIDER=local
  - LOCAL_STORAGE_PATH=/app/storage  # Optional, defaults to /app/storage

Pros¶

✅ Free - No additional costs
✅ Simple - No external dependencies
✅ Fast - Direct disk access
✅ Privacy - Data stays on your server
✅ No setup - Works out of the box

Cons¶

❌ Limited scalability - Bound by server disk space
❌ No redundancy - Single point of failure
❌ Manual backups - You must implement backup strategy
❌ No CDN - Slower for global users
❌ No public URLs - Requires reverse proxy setup

Use Cases¶

Development environments
Small projects (< 100GB)
Internal tools
Testing/staging environments

Cost Estimate¶

Storage: $0/month
Bandwidth: $0/month
Total: $0/month

Option 2: MinIO (Self-Hosted S3-Compatible)¶

Configuration¶

environment:
  - FILE_PROVIDER=minio
  - S3_ENDPOINT=http://minio:9000  # Your MinIO server
  - S3_ACCESS_KEY=your-access-key
  - S3_SECRET_KEY=your-secret-key
  - S3_BUCKET=your-bucket-name
  - S3_REGION=us-east-1

Pros¶

✅ Free software - Open source
✅ S3-compatible - Easy migration to/from S3
✅ Self-hosted - Full control over data
✅ Privacy - Data stays on your infrastructure
✅ Public URLs - Built-in support
✅ Signed URLs - Time-limited access

Cons¶

❌ Infrastructure management - You run and maintain it
❌ Limited scalability - Bound by your infrastructure
❌ Backup responsibility - You manage backups
❌ No CDN - Requires separate CDN setup

Setup Requirements¶

Deploy MinIO container (can add to docker-compose)
Create bucket
Generate access keys
Configure file-service

Use Cases¶

Self-hosted platforms
Privacy-sensitive applications
Cost-conscious production deployments
On-premises deployments

Cost Estimate¶

Storage: $0/month (your infrastructure)
Bandwidth: $0/month (your infrastructure)
Infrastructure: ~$10-50/month (depending on server)
Total: ~$10-50/month (infrastructure only)

Option 3: AWS S3 (Amazon Simple Storage Service)¶

Configuration¶

environment:
  - FILE_PROVIDER=s3
  - S3_ACCESS_KEY=${AWS_ACCESS_KEY_ID}
  - S3_SECRET_KEY=${AWS_SECRET_ACCESS_KEY}
  - S3_BUCKET=${AWS_S3_BUCKET}
  - S3_REGION=${AWS_REGION}  # e.g., eu-west-1
  - S3_PUBLIC_URL=https://your-bucket.s3.amazonaws.com  # Optional

Pros¶

✅ Highly scalable - Virtually unlimited storage
✅ 99.999999999% durability - 11 nines
✅ Global CDN - CloudFront integration
✅ Automatic backups - Versioning and replication
✅ Compliance - SOC 2, ISO 27001, HIPAA
✅ Public URLs - Direct access
✅ Signed URLs - Secure time-limited access
✅ Lifecycle policies - Automatic archival

Cons¶

❌ Costs - Pay for storage and bandwidth
❌ AWS account required - Need to set up AWS account
❌ Vendor lock-in - Harder to migrate away
❌ Complexity - IAM, buckets, policies

Setup Steps¶

Create AWS account
Create S3 bucket
Create IAM user with S3 permissions
Generate access keys
Configure CORS (if needed)
Set up CloudFront (optional, for CDN)

Pricing (as of 2024)¶

Storage: $0.023/GB/month (Standard)
Requests: $0.005 per 1,000 PUT requests
Bandwidth: $0.09/GB outbound (first 10TB)
Example: 100GB storage + 500GB/month transfer = ~$63/month

Use Cases¶

Production applications
High-traffic websites
Global user base
Compliance requirements
Large-scale deployments

Cost Estimate (Example)¶

100GB storage: $2.30/month
500GB transfer: $45/month
Total: ~$47-50/month (for medium traffic)

Option 4: Google Cloud Storage¶

Configuration¶

environment:
  - FILE_PROVIDER=gcs
  - GCLOUD_PROJECT=your-project-id
  - GCS_BUCKET=your-bucket-name
  - GCLOUD_KEY_FILE=/path/to/service-account-key.json

Pros¶

✅ Highly scalable - Virtually unlimited storage
✅ 99.999999999% durability - 11 nines
✅ Global CDN - Cloud CDN integration
✅ Automatic backups - Versioning and replication
✅ Compliance - SOC 2, ISO 27001, HIPAA
✅ Public URLs - Direct access
✅ Signed URLs - Secure time-limited access
✅ Lifecycle policies - Automatic archival

Cons¶

❌ Costs - Pay for storage and bandwidth
❌ GCP account required - Need to set up Google Cloud account
❌ Vendor lock-in - Harder to migrate away
❌ Complexity - Service accounts, buckets, IAM

Setup Steps¶

Create Google Cloud account
Create GCS bucket
Create service account
Download service account key JSON
Configure CORS (if needed)
Set up Cloud CDN (optional)

Pricing (as of 2024)¶

Storage: $0.020/GB/month (Standard)
Requests: $0.05 per 10,000 operations
Bandwidth: $0.12/GB outbound (first 10TB)
Example: 100GB storage + 500GB/month transfer = ~$70/month

Use Cases¶

Production applications
Google Cloud ecosystem users
High-traffic websites
Global user base

Cost Estimate (Example)¶

100GB storage: $2.00/month
500GB transfer: $60/month
Total: ~$62-65/month (for medium traffic)

Recommendations by Use Case¶

Development/Testing¶

Recommended: Local Storage - No setup required - Free - Fast iteration

Small Production (< 50GB, < 100GB/month transfer)¶

Recommended: MinIO or Local Storage - MinIO if you need S3 compatibility - Local if simplicity is key

Medium Production (50-500GB, 100GB-1TB/month transfer)¶

Recommended: AWS S3 or MinIO - AWS S3 for managed service - MinIO for cost savings

Large Production (> 500GB, > 1TB/month transfer)¶

Recommended: AWS S3 or Google Cloud Storage - Managed service required - CDN integration important - Automatic scaling needed

Privacy-Sensitive/On-Premises¶

Recommended: MinIO or Local Storage - Data stays on your infrastructure - Full control

Migration Path¶

You can easily switch between providers by changing the FILE_PROVIDER environment variable:

# Switch from S3 to Local
FILE_PROVIDER=local

# Switch from Local to MinIO
FILE_PROVIDER=minio

# Switch from MinIO to S3
FILE_PROVIDER=s3

The service API remains the same regardless of the backend storage provider.

Current Configuration¶

Based on your current setup in qualification.yml:

FILE_PROVIDER=s3
S3_ACCESS_KEY=${AWS_ACCESS_KEY_ID_QUAL}
S3_SECRET_KEY=${AWS_SECRET_ACCESS_KEY_QUAL}
S3_REGION=${AWS_REGION_QUAL}
S3_BUCKET=${AWS_S3_BUCKET_QUAL}

You are currently configured for AWS S3, which means: - ✅ You need AWS credentials configured - ✅ You need an S3 bucket created - ✅ You'll pay AWS for storage and bandwidth

To switch to a free option, you can change to:

FILE_PROVIDER=local
# Remove S3_* variables

Quick Setup Guides¶

Switch to Local Storage (Free)¶

Update qualification.yml:

environment:
  - FILE_PROVIDER=local
  # Remove S3_* variables

Restart service: docker compose restart file-service-qual

Switch to MinIO (Free, Self-hosted)¶

Add MinIO to docker-compose
Create bucket and access keys

Update qualification.yml:

environment:
  - FILE_PROVIDER=minio
  - S3_ENDPOINT=http://minio:9000
  - S3_ACCESS_KEY=minioadmin
  - S3_SECRET_KEY=minioadmin
  - S3_BUCKET=po-files

Keep AWS S3 (Current)¶

Ensure AWS credentials are in .env.qual
Create S3 bucket in AWS console
Configure bucket permissions
Current setup should work

Summary¶

You don't need an S3 account - you have 4 options:

Local - Free, simple, good for dev/small projects
MinIO - Free, self-hosted, S3-compatible
AWS S3 - Paid, managed, scalable (your current config)
GCS - Paid, managed, scalable

Choose based on your needs, budget, and scale requirements.