Backup & Maintenance

Backup & Maintenance

This guide covers backup and recovery procedures, system maintenance tasks, database migrations, and version updates for Querri.

Backup Strategy

Backup Components

A complete Querri backup includes:

1. MongoDB database - All application data
2. Redis data (optional) - Session and cache data
3. File storage - Uploaded files (local or S3)
4. Configuration files - .env-prod, Traefik configs
5. OPA policies - Authorization policies
6. Custom integrations - Custom integration code

Backup Schedule

Recommended schedule:

• Daily: MongoDB incremental backup
• Weekly: Full MongoDB backup
• Monthly: Full system backup including files
• Before updates: Complete backup before version updates

MongoDB Backup

Manual Backup

Using mongodump

# Create backup directory
mkdir -p backups/$(date +%Y%m%d)

# Backup entire database
docker compose exec -T mongo mongodump \
  --username=querri \
  --password=your_password \
  --authenticationDatabase=admin \
  --db=querri \
  --out=/backup

# Copy from container to host
docker cp querri-mongo:/backup ./backups/$(date +%Y%m%d)/

Backup specific collections

# Backup only critical collections
docker compose exec -T mongo mongodump \
  --username=querri \
  --password=your_password \
  --authenticationDatabase=admin \
  --db=querri \
  --collection=projects \
  --out=/backup/projects_$(date +%Y%m%d)

# Backup users collection
docker compose exec -T mongo mongodump \
  --username=querri \
  --password=your_password \
  --authenticationDatabase=admin \
  --db=querri \
  --collection=users \
  --out=/backup/users_$(date +%Y%m%d)

Automated Backup Script

backup_mongodb.sh:

#!/bin/bash

# Configuration
BACKUP_DIR="/var/backups/querri"
MONGO_USER="querri"
MONGO_PASSWORD="your_password"
RETENTION_DAYS=30

# Create backup directory with timestamp
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_PATH="$BACKUP_DIR/$TIMESTAMP"
mkdir -p "$BACKUP_PATH"

# Perform backup
docker compose exec -T mongo mongodump \
  --username="$MONGO_USER" \
  --password="$MONGO_PASSWORD" \
  --authenticationDatabase=admin \
  --db=querri \
  --gzip \
  --out=/backup

# Copy from container
docker cp querri-mongo:/backup "$BACKUP_PATH/"

# Compress backup
cd "$BACKUP_DIR"
tar -czf "${TIMESTAMP}.tar.gz" "$TIMESTAMP"
rm -rf "$TIMESTAMP"

# Remove old backups
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +$RETENTION_DAYS -delete

# Log completion
echo "$(date): Backup completed - ${TIMESTAMP}.tar.gz" >> /var/log/querri-backup.log

# Upload to S3 (optional)
aws s3 cp "${TIMESTAMP}.tar.gz" s3://your-backup-bucket/querri-backups/

Make executable and schedule:

# Make executable
chmod +x backup_mongodb.sh

# Add to crontab (daily at 2 AM)
crontab -e
0 2 * * * /path/to/backup_mongodb.sh

Backup Verification

Verify backup integrity:

# List backup contents
tar -tzf backup_20240115_020000.tar.gz

# Test restore to temporary database
docker compose exec -T mongo mongorestore \
  --username=querri \
  --password=your_password \
  --authenticationDatabase=admin \
  --db=querri_test \
  --gzip \
  --archive=/backup/backup.gz

# Verify data
docker compose exec mongo mongosh -u querri -p
> use querri_test
> db.projects.count()
> db.users.count()

MongoDB Restore

Full Database Restore

# Stop application services (keep mongo running)
docker compose stop web-app server-api hub

# Restore from backup
docker compose exec -T mongo mongorestore \
  --username=querri \
  --password=your_password \
  --authenticationDatabase=admin \
  --db=querri \
  --drop \
  --gzip \
  --archive=/backup/backup.gz

# Restart services
docker compose start web-app server-api hub

Restore Specific Collection

# Restore only projects collection
docker compose exec -T mongo mongorestore \
  --username=querri \
  --password=your_password \
  --authenticationDatabase=admin \
  --db=querri \
  --collection=projects \
  --drop \
  /backup/querri/projects.bson

Point-in-Time Recovery

For replica set with oplog:

# Restore to specific timestamp
docker compose exec -T mongo mongorestore \
  --username=querri \
  --password=your_password \
  --authenticationDatabase=admin \
  --oplogReplay \
  --oplogLimit=1640000000:1 \
  /backup/

Redis Persistence

Enable Redis Persistence

Update docker-compose.yml:

redis:
  image: redis:alpine
  command: redis-server --appendonly yes
  volumes:
    - redis-data:/data
  ports:
    - "6379:6379"
  restart: unless-stopped

volumes:
  redis-data:

Redis Backup

# Trigger Redis save
docker compose exec redis redis-cli SAVE

# Copy RDB file
docker cp querri-redis:/data/dump.rdb ./backups/redis_$(date +%Y%m%d).rdb

# Or use BGSAVE for background save
docker compose exec redis redis-cli BGSAVE

Redis Restore

# Stop Redis
docker compose stop redis

# Copy backup to container
docker cp ./backups/redis_20240115.rdb querri-redis:/data/dump.rdb

# Start Redis
docker compose start redis

File Storage Backup

Local File Storage

# Backup files directory
BACKUP_DATE=$(date +%Y%m%d)
tar -czf backups/files_${BACKUP_DATE}.tar.gz \
  ./server-api/files/

# Verify backup
tar -tzf backups/files_${BACKUP_DATE}.tar.gz | head -20

S3 Storage Backup

If using S3 for file storage:

# Enable S3 versioning
aws s3api put-bucket-versioning \
  --bucket your-querri-files \
  --versioning-configuration Status=Enabled

# Cross-region replication for disaster recovery
aws s3api put-bucket-replication \
  --bucket your-querri-files \
  --replication-configuration file://replication-config.json

# Lifecycle policy for automated backups
aws s3api put-bucket-lifecycle-configuration \
  --bucket your-querri-files \
  --lifecycle-configuration file://lifecycle-config.json

lifecycle-config.json:

{
  "Rules": [
    {
      "Id": "Archive old files",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 30
      }
    }
  ]
}

Configuration Backup

Backup Configuration Files

# Create config backup
BACKUP_DATE=$(date +%Y%m%d)
mkdir -p backups/config_${BACKUP_DATE}

# Copy configuration files
cp .env-prod backups/config_${BACKUP_DATE}/
cp -r traefik/ backups/config_${BACKUP_DATE}/
cp -r opa/policies/ backups/config_${BACKUP_DATE}/
cp docker-compose.yml backups/config_${BACKUP_DATE}/

# Compress
tar -czf backups/config_${BACKUP_DATE}.tar.gz backups/config_${BACKUP_DATE}/
rm -rf backups/config_${BACKUP_DATE}/

# Secure storage (configs contain secrets)
chmod 600 backups/config_${BACKUP_DATE}.tar.gz

Version Control for Configuration

# Initialize git repo for configs (private repo only!)
cd /path/to/querri-configs
git init
echo ".env-prod" >> .gitignore  # Don't commit secrets!

# Commit configuration structure
git add docker-compose.yml traefik/ opa/
git commit -m "Querri configuration backup $(date +%Y-%m-%d)"

# Push to private repository
git remote add origin git@github.com:yourorg/querri-configs.git
git push origin main

Disaster Recovery

Recovery Plan

RTO (Recovery Time Objective): 4 hours RPO (Recovery Point Objective): 24 hours

Recovery Procedure

1. Provision new infrastructure (if needed)

   • New server or VM
   • Install Docker and Docker Compose
   • Configure network and firewall

2. Restore configuration:

   # Extract config backup
   tar -xzf config_20240115.tar.gz
   cd config_20240115/
   
   # Restore .env-prod
   cp .env-prod /path/to/querri/.env-prod
   
   # Restore Traefik config
   cp -r traefik/ /path/to/querri/traefik/

3. Restore MongoDB:

   # Start MongoDB
   docker compose up -d mongo
   
   # Wait for MongoDB to be ready
   sleep 10
   
   # Restore backup
   docker compose exec -T mongo mongorestore \
     --username=querri \
     --password=your_password \
     --authenticationDatabase=admin \
     --db=querri \
     --gzip \
     --archive < backups/backup_20240115.gz

4. Restore files:

   # Extract files backup
   tar -xzf backups/files_20240115.tar.gz -C ./server-api/

5. Start all services:

   docker compose up -d

6. Verify recovery:

   # Check health
   curl http://localhost:8180/healthz
   
   # Test login
   curl -I https://app.yourcompany.com
   
   # Verify database
   docker compose exec mongo mongosh -u querri -p
   > use querri
   > db.projects.count()
   > db.users.count()

Disaster Recovery Testing

Test recovery quarterly:

test_disaster_recovery.sh

#!/bin/bash
echo "=== Disaster Recovery Test ==="
echo "Starting: $(date)"

# 1. Create fresh backup
./backup_mongodb.sh

# 2. Stop services
docker compose down

# 3. Remove volumes (TEST ENVIRONMENT ONLY!)
docker volume rm querri_mongodb_data

# 4. Recreate volumes
docker volume create querri_mongodb_data

# 5. Start MongoDB
docker compose up -d mongo
sleep 10

# 6. Restore latest backup
LATEST_BACKUP=$(ls -t backups/*.tar.gz | head -1)
tar -xzf "$LATEST_BACKUP" -C /tmp/
docker compose exec -T mongo mongorestore --username=querri --password=your_password --authenticationDatabase=admin --db=querri --gzip /tmp/backup

# 7. Start all services
docker compose up -d

# 8. Verify
sleep 30
HEALTH=$(curl -s http://localhost:8180/healthz | grep -c "healthy")

if [ $HEALTH -eq 1 ]; then
    echo "✓ Recovery test successful"
else
    echo "✗ Recovery test failed"
fi

echo "Completed: $(date)"

Database Migrations

Migration Process

When updating Querri versions, database migrations may be required:

Pre-Migration Backup

# ALWAYS backup before migration
./backup_mongodb.sh

Running Migrations

# Check for pending migrations
docker compose exec server-api python -m alembic current

# Run migrations
docker compose exec server-api python -m alembic upgrade head

# Verify migration
docker compose exec server-api python -m alembic current

Rollback Migration

If migration fails:

# Rollback to previous version
docker compose exec server-api python -m alembic downgrade -1

# Or restore from backup
docker compose exec -T mongo mongorestore \
  --username=querri \
  --password=your_password \
  --authenticationDatabase=admin \
  --db=querri \
  --drop \
  --gzip \
  --archive < backups/pre_migration_backup.gz

Manual Schema Updates

For manual schema changes:

// Connect to MongoDB
docker compose exec mongo mongosh -u querri -p

use querri

// Add new field to all projects
db.projects.updateMany(
  {new_field: {$exists: false}},
  {$set: {new_field: "default_value"}}
)

// Create index
db.projects.createIndex({created_at: -1})

// Verify update
db.projects.findOne({}, {new_field: 1})

Updating Querri Versions

Pre-Update Checklist

• Review release notes
• Create full backup
• Test update in staging environment
• Schedule maintenance window
• Notify users of downtime

Update Procedure

1. Backup Current System

# Full backup
./backup_mongodb.sh

# Backup current config
cp .env-prod .env-prod.backup.$(date +%Y%m%d)
cp docker-compose.yml docker-compose.yml.backup.$(date +%Y%m%d)

2. Review Changes

# Check current version
docker compose exec web-app env | grep APP_VERSION

# Review release notes
curl https://api.github.com/repos/querri/querri-stack/releases/latest

3. Pull New Images

# Pull latest images
docker compose pull

# Verify new versions
docker compose images

4. Update Configuration

# Review .env changes
diff .env.example .env-prod

# Update environment variables if needed
nano .env-prod

5. Stop Services

# Graceful shutdown
docker compose down

6. Start with New Version

# Start services
docker compose up -d

# Monitor startup
docker compose logs -f

7. Run Migrations

# Run database migrations
docker compose exec server-api python -m alembic upgrade head

8. Verify Update

# Check health
curl http://localhost:8180/healthz

# Verify version
docker compose exec web-app env | grep APP_VERSION

# Test functionality
curl -H "Authorization: Bearer $JWT_TOKEN" \
  https://app.yourcompany.com/api/projects

Rollback Procedure

If update fails:

# 1. Stop services
docker compose down

# 2. Restore previous version
git checkout previous_version
# Or restore docker-compose.yml.backup

# 3. Restore database
docker compose up -d mongo
docker compose exec -T mongo mongorestore \
  --username=querri \
  --password=your_password \
  --authenticationDatabase=admin \
  --db=querri \
  --drop \
  --gzip \
  --archive < backups/pre_update_backup.gz

# 4. Start services
docker compose up -d

# 5. Verify rollback
curl http://localhost:8180/healthz

Scaling Services

Horizontal Scaling

Scale server-api replicas:

# Increase replicas
docker compose up -d --scale server-api=8

# Verify scaling
docker compose ps server-api

# Update .env-prod for persistent scaling
echo "SERVER_API_REPLICAS=8" >> .env-prod
docker compose up -d

Vertical Scaling

Add resource limits:

docker-compose.yml:

server-api:
  deploy:
    resources:
      limits:
        cpus: '2.0'
        memory: 4G
      reservations:
        cpus: '1.0'
        memory: 2G

Maintenance Tasks

Daily Tasks

# Check service health
docker compose ps

# Review error logs
docker compose logs --tail=100 server-api | grep ERROR

# Monitor disk usage
df -h /var/lib/docker

Weekly Tasks

# Clean Docker system
docker system prune -f

# Verify backups
ls -lh backups/

# Review MongoDB slow queries
docker compose exec mongo mongosh -u querri -p
> use querri
> db.setProfilingLevel(1, {slowms: 100})
> db.system.profile.find().sort({ts: -1}).limit(10)

# Check for updates
docker compose pull --dry-run

Monthly Tasks

# Full system backup
./backup_mongodb.sh
tar -czf backups/files_$(date +%Y%m%d).tar.gz ./server-api/files/

# Review and optimize indexes
docker compose exec mongo mongosh -u querri -p
> use querri
> db.projects.getIndexes()
> db.projects.stats().indexSizes

# Update Docker images
docker compose pull
docker compose up -d

# Security updates
apt update && apt upgrade -y

# Certificate renewal (if not automated)
certbot renew

Quarterly Tasks

# Disaster recovery test
./test_disaster_recovery.sh

# Review user access
docker compose exec mongo mongosh -u querri -p
> use querri
> db.users.find({is_admin: true})

# Rotate credentials
# - JWT private key
# - MongoDB password
# - API keys

# Performance review
# - Review slow queries
# - Analyze resource usage trends
# - Plan capacity upgrades

Maintenance Windows

Scheduling Maintenance

1. Notify users (7 days advance notice)
2. Choose low-traffic time (typically Sunday 2-6 AM)
3. Create backup before maintenance
4. Set maintenance mode (optional)

Maintenance Mode

Display maintenance page:

traefik/dynamic.yml:

http:
  services:
    maintenance:
      loadBalancer:
        servers:
          - url: "http://maintenance-page:80"

  routers:
    maintenance-mode:
      rule: "PathPrefix(`/`)"
      service: maintenance
      priority: 100

Or use environment variable:

# Enable maintenance mode
MAINTENANCE_MODE=true docker compose up -d

# Disable maintenance mode
MAINTENANCE_MODE=false docker compose up -d

Best Practices

1. 3-2-1 Backup Rule: 3 copies, 2 different media, 1 offsite
2. Test backups regularly: Monthly restore tests
3. Automate backups: Don’t rely on manual processes
4. Monitor backup success: Alert on backup failures
5. Document procedures: Keep runbooks current
6. Version configuration: Track config changes in git
7. Secure backups: Encrypt sensitive backup data
8. Retention policy: Balance storage costs vs. recovery needs

Next Steps

• Monitoring & Usage - Monitor backup success and system health
• Troubleshooting - Resolve common issues
• Installation & Deployment - Initial setup procedures
• Environment Configuration - Configuration management