Troubleshooting

Troubleshooting

This guide covers common deployment and operational issues, debugging procedures, and solutions for Querri administrators.

Common Deployment Issues

Services Failing to Start

Symptoms:

• Containers exit immediately after starting
• Services show “Exited (1)” status
• Application not accessible

Diagnosis:

# Check service status
docker compose ps

# View service logs
docker compose logs <service-name>

# Check container exit code
docker inspect <container-name> | grep -A 5 "State"

Common Causes & Solutions:

1. Missing Environment Variables

Error in logs:

KeyError: 'MONGODB_HOST'
Environment variable MONGODB_HOST not set

Solution:

# Verify .env-prod exists
ls -la .env-prod

# Check required variables are set
grep MONGODB_HOST .env-prod
grep WORKOS_API_KEY .env-prod

# Restart services
docker compose up -d

2. Database Connection Failure

Error in logs:

pymongo.errors.ServerSelectionTimeoutError: Connection refused

Solution:

# Check MongoDB is running
docker compose ps mongo

# Verify MongoDB credentials
docker compose exec mongo mongosh -u querri -p your_password

# Check network connectivity
docker compose exec server-api ping mongo

# Restart services in order
docker compose restart mongo
sleep 10
docker compose restart hub server-api

3. Port Conflicts

Error in logs:

Error starting userland proxy: listen tcp 0.0.0.0:8080: bind: address already in use

Solution:

# Find process using port
sudo lsof -i :8080

# Kill process or change port in docker-compose.yml
ports:
  - "8081:80"  # Changed from 8080 to 8081

# Restart services
docker compose up -d

Authentication Failures

Symptoms:

• Users cannot log in
• “Unauthorized” errors
• Redirect loops

Diagnosis:

# Check hub service logs
docker compose logs hub | grep -i auth

# Test WorkOS connection
curl -H "Authorization: Bearer ${WORKOS_API_KEY}" \
  https://api.workos.com/organizations

# Verify JWT configuration
docker compose exec hub env | grep WORKOS

Common Causes & Solutions:

1. Invalid WorkOS Credentials

Solution:

# Verify credentials in .env-prod
grep WORKOS_API_KEY .env-prod
grep WORKOS_CLIENT_ID .env-prod

# Test credentials
curl -u ${WORKOS_CLIENT_ID}:${WORKOS_API_KEY} \
  https://api.workos.com/user_management/users

# Update credentials if expired
nano .env-prod
docker compose restart hub

2. Incorrect Redirect URI

Error: “redirect_uri_mismatch”

Solution:

# Check configured redirect URI
grep WORKOS_REDIRECT_URI .env-prod

# Should match: {PUBLIC_BASE_URL}/hub/auth/callback
# Example: https://app.yourcompany.com/hub/auth/callback

# Update WorkOS dashboard to match
# Or update .env-prod
nano .env-prod
docker compose restart hub

3. JWT Validation Failure

Error in logs:

JWT signature verification failed

Solution:

# Verify JWT private key is set
grep JWT_PRIVATE_KEY .env-prod

# Ensure key format is correct (PEM)
# Key should start with: -----BEGIN PRIVATE KEY-----

# Regenerate if needed
openssl genrsa -out private.pem 2048
cat private.pem

# Update .env-prod
nano .env-prod
docker compose restart hub reverse-proxy

Database Connection Errors

Symptoms:

• “Connection refused” errors
• Timeout errors
• “Authentication failed”

Diagnosis:

# Check MongoDB status
docker compose ps mongo

# Test connection
docker compose exec mongo mongosh -u querri -p

# Check MongoDB logs
docker compose logs mongo | tail -50

Common Causes & Solutions:

1. MongoDB Not Running

Solution:

# Start MongoDB
docker compose up -d mongo

# Wait for MongoDB to be ready
sleep 10

# Verify it's running
docker compose ps mongo

2. Incorrect Credentials

Error:

Authentication failed

Solution:

# Verify credentials in .env-prod
grep MONGO_INITDB_ROOT_USERNAME .env-prod
grep MONGO_INITDB_ROOT_PASSWORD .env-prod

# Reset MongoDB with new credentials
docker compose stop mongo
docker volume rm querri_mongodb_data  # WARNING: Deletes data!
docker compose up -d mongo

# Or update application to use correct credentials
nano .env-prod
docker compose restart server-api hub

3. Network Issues

Solution:

# Check Docker network
docker network ls
docker network inspect querri_default

# Recreate network
docker compose down
docker compose up -d

# Verify connectivity
docker compose exec server-api ping mongo

Performance Issues

Symptoms:

• Slow page loads
• Timeouts
• High CPU/memory usage

Diagnosis:

# Check resource usage
docker stats --no-stream

# Check service logs for slow queries
docker compose logs server-api | grep -i "slow\|timeout"

# Check MongoDB performance
docker compose exec mongo mongosh -u querri -p
> use querri
> db.currentOp({"secs_running": {$gte: 5}})

Common Causes & Solutions:

1. Insufficient Resources

Solution:

# Check available resources
free -h
df -h

# Increase container limits (docker-compose.yml)
services:
  server-api:
    deploy:
      resources:
        limits:
          cpus: '4.0'
          memory: 8G

# Scale replicas
SERVER_API_REPLICAS=8 docker compose up -d --scale server-api=8

2. Missing Database Indexes

Solution:

// Connect to MongoDB
db.projects.createIndex({created_by: 1, created_at: -1})
db.projects.createIndex({organization_id: 1})
db.files.createIndex({uploaded_by: 1})
db.audit_log.createIndex({timestamp: -1})

// Verify indexes
db.projects.getIndexes()

3. Redis Memory Issues

Solution:

# Check Redis memory
docker compose exec redis redis-cli INFO memory

# Clear cache if needed
docker compose exec redis redis-cli FLUSHALL

# Configure maxmemory
docker compose exec redis redis-cli CONFIG SET maxmemory 2gb
docker compose exec redis redis-cli CONFIG SET maxmemory-policy allkeys-lru

Service Connectivity Problems

Traefik Routing Issues

Symptoms:

• 404 errors
• Services not accessible
• Incorrect routing

Diagnosis:

# Check Traefik logs
docker compose logs reverse-proxy

# View Traefik configuration
docker compose exec reverse-proxy cat /etc/traefik/traefik.yml

# Check Traefik dashboard (if enabled)
curl http://localhost:8080/dashboard/

Solutions:

# Verify service labels in docker-compose.yml
# Ensure services have correct Traefik labels

# Restart Traefik
docker compose restart reverse-proxy

# Check Traefik can reach services
docker compose exec reverse-proxy ping web-app
docker compose exec reverse-proxy ping server-api

Internal Service Communication

Symptoms:

• Services can’t communicate
• “Connection refused” between services

Diagnosis:

# Check Docker network
docker network inspect querri_default

# Test connectivity
docker compose exec web-app ping server-api
docker compose exec server-api ping mongo
docker compose exec server-api ping redis

Solutions:

# Ensure all services on same network
# In docker-compose.yml, verify network configuration

# Recreate network
docker compose down
docker compose up -d

# Check service DNS resolution
docker compose exec server-api nslookup mongo

File Upload Issues

Symptoms:

• File uploads fail
• “Permission denied” errors
• Files not appearing

Diagnosis:

# Check file storage configuration
grep FILE_STORAGE .env-prod

# Check local storage permissions
ls -la ./server-api/files/

# View server-api logs
docker compose logs server-api | grep -i "upload\|file"

Solutions:

1. Permission Issues (Local Storage)

# Fix directory permissions
sudo chown -R 1000:1000 ./server-api/files/
sudo chmod -R 755 ./server-api/files/

# Verify volume mount
docker compose exec server-api ls -la /app/files/

2. S3 Configuration Issues

# Verify S3 credentials
grep AWS_ACCESS_KEY_ID .env-prod
grep AWS_SECRET_ACCESS_KEY .env-prod

# Test S3 access
aws s3 ls s3://your-bucket-name/

# Verify bucket permissions
aws s3api get-bucket-policy --bucket your-bucket-name

3. File Size Limits

# Check Traefik file size limits
# Add to traefik/dynamic.yml
http:
  middlewares:
    limit:
      buffering:
        maxRequestBodyBytes: 104857600  # 100MB

# Update nginx if used
client_max_body_size 100M;

Integration Issues

Prismatic Integration Failures

Symptoms:

• Integrations not loading
• “Authentication failed”
• Sync failures

Diagnosis:

# Check Prismatic configuration
grep PRISMATIC_KEY .env-prod
grep PRISMATIC_TOKEN .env-prod

# View integration logs
docker compose logs server-api | grep -i prismatic

# Check database for integration errors
docker compose exec mongo mongosh -u querri -p
> use querri
> db.integration_logs.find({status: "error"}).sort({timestamp: -1})

Solutions:

1. Expired Prismatic Token

# Token needs refresh
# Get new token from Prismatic dashboard
# Update .env-prod
nano .env-prod
# Add new PRISMATIC_TOKEN

# Restart services
docker compose restart server-api

2. Integration Configuration Errors

// Check integration configuration
db.integrations.find({status: "error"})

// Update integration config
db.integrations.updateOne(
  {_id: "int_xxxxxxxxxxxxx"},
  {
    $set: {
      "config.api_key": "new_key",
      status: "active"
    }
  }
)

AI/OpenAI Issues

AI Features Not Working

Symptoms:

• Chat not responding
• Analysis failures
• “AI service unavailable”

Diagnosis:

# Check AI configuration
grep OPENAI_API_KEY .env-prod
grep AZURE_OPENAI_ENDPOINT .env-prod

# Test OpenAI connection
curl https://api.openai.com/v1/models \
  -H "Authorization: Bearer ${OPENAI_API_KEY}"

# Check server-api logs
docker compose logs server-api | grep -i "openai\|ai\|gpt"

Solutions:

1. Invalid API Key

# Verify API key is valid
curl https://api.openai.com/v1/models \
  -H "Authorization: Bearer ${OPENAI_API_KEY}"

# Update key if invalid
nano .env-prod
docker compose restart server-api

2. Rate Limiting

Error: “Rate limit exceeded”

# Check current usage in OpenAI dashboard
# Implement rate limiting in application
# Upgrade OpenAI tier if needed

# Temporary: Reduce AI usage
# Set in .env-prod:
AI_RATE_LIMIT_REQUESTS_PER_MINUTE=10

3. Model Not Available

Error: “Model not found”

# Verify model names
grep STANDARD_MODEL .env-prod

# For Azure OpenAI, use deployment names
# For OpenAI, use model IDs: gpt-4o, gpt-4o-mini

# Update model configuration
nano .env-prod
docker compose restart server-api

Log Locations and Debug Mode

Service Log Locations

# Docker logs (ephemeral)
docker compose logs <service-name>

# Persistent logs (if configured)
/var/log/querri/server-api.log
/var/log/querri/hub.log
/var/log/querri/web-app.log

# MongoDB logs
docker compose exec mongo cat /var/log/mongodb/mongod.log

# Traefik access logs
docker compose exec reverse-proxy cat /var/log/traefik/access.log

Enabling Debug Mode

Application Debug Mode

# Enable debug logging
echo "LOG_LEVEL=DEBUG" >> .env-prod

# Restart services
docker compose restart server-api hub

# View debug logs
docker compose logs -f server-api | grep DEBUG

Traefik Debug Mode

traefik/traefik.yml

log:
  level: DEBUG

accessLog:
  enabled: true
  filePath: /var/log/traefik/access.log
  format: json

# Restart Traefik
docker compose restart reverse-proxy

# View debug logs
docker compose logs reverse-proxy

MongoDB Debug Mode

# Enable profiling for slow queries
docker compose exec mongo mongosh -u querri -p
> use querri
> db.setProfilingLevel(2)  // Log all queries
> db.setProfilingLevel(1, {slowms: 100})  // Log queries > 100ms

# View profiled queries
> db.system.profile.find().sort({ts: -1}).limit(10)

Debugging Workflows

1. Service Won’t Start

# Step 1: Check logs
docker compose logs <service-name>

# Step 2: Check environment
docker compose exec <service-name> env

# Step 3: Check dependencies
docker compose ps

# Step 4: Manual start with debug
docker compose run --rm <service-name> bash
# Then manually start service to see errors

2. Request Failing

# Step 1: Check Traefik routing
docker compose logs reverse-proxy | grep <request-path>

# Step 2: Check service logs
docker compose logs server-api | grep <request-id>

# Step 3: Check database
docker compose exec mongo mongosh -u querri -p
> db.audit_log.find({request_id: "<request-id>"})

# Step 4: Reproduce with curl
curl -v -H "Authorization: Bearer $JWT" \
  https://app.yourcompany.com/api/endpoint

3. Performance Issues

# Step 1: Identify bottleneck
docker stats --no-stream

# Step 2: Check slow queries
docker compose exec mongo mongosh -u querri -p
> db.currentOp({"secs_running": {$gte: 5}})

# Step 3: Check application metrics
docker compose logs server-api | grep "response_time\|latency"

# Step 4: Profile code
# Add timing decorators to Python code

Advanced Troubleshooting

Container Debugging

Access Container Shell

# Access running container
docker compose exec server-api bash

# Or start new container for debugging
docker compose run --rm server-api bash

# Install debugging tools
apt update && apt install -y curl vim less procps

Inspect Container State

# Full container inspection
docker inspect querri-server-api

# Specific information
docker inspect querri-server-api | grep -A 10 "State"
docker inspect querri-server-api | grep -A 20 "NetworkSettings"
docker inspect querri-server-api | grep -A 10 "Mounts"

Network Debugging

# Install network tools
docker compose exec server-api apt update
docker compose exec server-api apt install -y iputils-ping dnsutils netcat

# Test connectivity
docker compose exec server-api ping mongo
docker compose exec server-api nc -zv mongo 27017
docker compose exec server-api nslookup mongo

# View network configuration
docker network inspect querri_default

Database Debugging

# Connect to MongoDB
docker compose exec mongo mongosh -u querri -p

# Check database stats
> use querri
> db.stats()

# Check collection stats
> db.projects.stats()

# Current operations
> db.currentOp()

# Server status
> db.serverStatus()

# Replication status (if using replica set)
> rs.status()

Getting Help

Information to Gather

When reporting issues, collect:

1. System information:

   uname -a
   docker --version
   docker compose version

2. Service status:

   docker compose ps
   docker compose logs --tail=100 <service-name>

3. Configuration (sanitized):

   # Remove sensitive data before sharing
   cat .env-prod | grep -v "PASSWORD\|KEY\|SECRET"

4. Error messages:

   docker compose logs | grep -i "error\|exception\|failed"

5. Resource usage:

   docker stats --no-stream
   df -h
   free -h

Support Channels

• Documentation: https://docs.querri.com
• GitHub Issues: https://github.com/querri/querri-stack/issues
• Email Support: support@querri.com
• Community Forum: https://community.querri.com

Preventive Measures

Health Monitoring

Set up automated health checks:

# Create monitoring script
cat > /usr/local/bin/check-querri-health.sh << 'EOF'
#!/bin/bash
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8180/healthz)
if [ "$STATUS" -ne 200 ]; then
    echo "Health check failed: HTTP $STATUS"
    # Send alert
    curl -X POST $SLACK_WEBHOOK -d '{"text":"Querri health check failed"}'
fi
EOF

chmod +x /usr/local/bin/check-querri-health.sh

# Add to crontab
echo "*/5 * * * * /usr/local/bin/check-querri-health.sh" | crontab -

Log Rotation

Prevent disk space issues:

/etc/docker/daemon.json

# Configure Docker log rotation
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

sudo systemctl restart docker

Regular Maintenance

# Weekly cleanup script
cat > /usr/local/bin/querri-maintenance.sh << 'EOF'
#!/bin/bash

# Clean Docker system
docker system prune -f

# Check disk space
DISK_USAGE=$(df -h / | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $DISK_USAGE -gt 80 ]; then
    echo "WARNING: Disk usage at ${DISK_USAGE}%"
fi

# Backup database
/path/to/backup_mongodb.sh
EOF

chmod +x /usr/local/bin/querri-maintenance.sh

# Schedule weekly
echo "0 2 * * 0 /usr/local/bin/querri-maintenance.sh" | crontab -

Quick Reference

Common Commands

# View all service logs
docker compose logs -f

# Restart all services
docker compose restart

# Rebuild and restart specific service
docker compose up -d --build server-api

# Check service health
docker compose ps

# View resource usage
docker stats

# Clean up
docker system prune -a -f

# Backup database
docker compose exec -T mongo mongodump ...

# Restore database
docker compose exec -T mongo mongorestore ...

Environment Variables Check

# Required variables checklist
for var in MONGO_INITDB_ROOT_USERNAME MONGO_INITDB_ROOT_PASSWORD WORKOS_API_KEY WORKOS_CLIENT_ID JWT_PRIVATE_KEY; do
    if grep -q "^${var}=" .env-prod; then
        echo "✓ $var is set"
    else
        echo "✗ $var is MISSING"
    fi
done

Next Steps

• Monitoring & Usage - Set up proactive monitoring
• Backup & Maintenance - Regular maintenance procedures
• Installation & Deployment - Review deployment setup
• Security & Permissions - Security troubleshooting