237899745/ystp

Fork 0

Files

yuyx df9c40e456 Parallelize worker batch processing

2025-12-20 18:37:27 +08:00

15 KiB

Raw Blame History

部署指南

环境准备

系统要求

Linux (Ubuntu 22.04+ / Debian 12+ 推荐)
2+ CPU 核心（启用独立 Worker 建议 4+）
4GB+ 内存
50GB+ 磁盘空间

依赖安装

# Ubuntu/Debian
sudo apt update
sudo apt install -y \
    build-essential \
    pkg-config \
    libssl-dev \
    libpq-dev \
    cmake \
    nasm \
    libjpeg-dev \
    libpng-dev \
    libwebp-dev

# 安装 Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env

# 初始化数据库会用到 psql（建议安装 PostgreSQL client）
sudo apt install -y postgresql-client

# 安装 Node.js (前端构建)
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs

本地开发

1. 启动数据库服务

# 使用 Docker Compose 启动 PostgreSQL 和 Redis
docker-compose -f docker/docker-compose.dev.yml up -d

docker/docker-compose.dev.yml:

version: '3.8'

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: imageforge
      POSTGRES_PASSWORD: devpassword
      POSTGRES_DB: imageforge
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

  # 可选：MinIO（S3 兼容，本地开发更接近生产）
  # minio:
  #   image: minio/minio:RELEASE.2024-01-28T20-20-01Z
  #   command: server /data --console-address ":9001"
  #   environment:
  #     MINIO_ROOT_USER: minioadmin
  #     MINIO_ROOT_PASSWORD: minioadmin
  #   ports:
  #     - "9000:9000"
  #     - "9001:9001"
  #   volumes:
  #     - minio_data:/data

volumes:
  postgres_data:
  redis_data:
  minio_data:

2. 配置环境变量

cp .env.example .env

.env.example:

# 运行模式：建议将 API 与 Worker 分开运行
IMAGEFORGE_ROLE=api # api | worker

# 服务配置
HOST=0.0.0.0
PORT=8080
PUBLIC_BASE_URL=http://localhost:8080
RUST_LOG=info,imageforge=debug

# 数据库
DATABASE_URL=postgres://imageforge:devpassword@localhost:5432/imageforge

# Redis
REDIS_URL=redis://localhost:6379

# Worker 并发（每个批量任务内同时处理的文件数）
WORKER_CONCURRENCY=4

# JWT（网站/管理后台）
JWT_SECRET=your-super-secret-key-change-in-production
JWT_EXPIRY_HOURS=168

# API Key
API_KEY_PEPPER=please-change-this-in-production

# 存储（生产建议 S3/MinIO + 预签名 URL）
STORAGE_TYPE=local  # local | s3
STORAGE_PATH=./uploads
# 预签名下载链接过期（分钟）
SIGNED_URL_TTL_MINUTES=60

# S3 配置（如果使用 S3/MinIO）
# S3_ENDPOINT=http://localhost:9000
# S3_BUCKET=your-bucket
# S3_REGION=us-east-1
# S3_ACCESS_KEY=xxx
# S3_SECRET_KEY=xxx

# 计费（已确认：Stripe）
BILLING_PROVIDER=stripe
STRIPE_SECRET_KEY=sk_test_xxx
STRIPE_WEBHOOK_SECRET=whsec_xxx

# 限制（默认值；最终以套餐/用户覆盖为准）
ALLOW_ANONYMOUS_UPLOAD=true
ANON_MAX_FILE_SIZE_MB=5
ANON_MAX_FILES_PER_BATCH=5
ANON_DAILY_UNITS=10
MAX_IMAGE_PIXELS=40000000
IDEMPOTENCY_TTL_HOURS=24

# 结果保留（匿名默认；登录用户按套餐 retention_days）
ANON_RETENTION_HOURS=24

# 管理员初始账户
ADMIN_EMAIL=admin@example.com
ADMIN_PASSWORD=changeme123

3. 初始化数据库

# 运行首期迁移（可重复执行）
psql "$DATABASE_URL" -f migrations/001_init.sql

4. 启动开发服务器

# 后端 API (热重载)
cargo install cargo-watch
IMAGEFORGE_ROLE=api cargo watch -x run

# 后端 Worker（另一个终端，处理异步/批量任务）
IMAGEFORGE_ROLE=worker cargo watch -x run

# 前端 (另一个终端)
cd frontend
npm install
npm run dev

5. Stripe Webhook 本地调试（可选）

本地调试 Stripe 订阅/支付状态，通常需要将 Stripe Webhook 转发到本机：

# 1) 安装并登录 Stripe CLI（按官方文档）
# 2) 监听并转发到你的后端回调地址
stripe listen --forward-to http://localhost:8080/api/v1/webhooks/stripe

# CLI 会输出一个 whsec_...，写入 .env 的 STRIPE_WEBHOOK_SECRET

生产部署

注意：以下内容为生产部署模板示例；仓库当前首期仅提供开发用 docker/docker-compose.dev.yml，生产 compose/Dockerfile/nginx/k8s 等可在开工阶段按需落地并调整。

方案一：Docker Compose（推荐小规模）

docker/docker-compose.prod.yml:

version: '3.8'

services:
  api:
    build:
      context: ..
      dockerfile: docker/Dockerfile
    environment:
      - IMAGEFORGE_ROLE=api
      - BILLING_PROVIDER=stripe
      - PUBLIC_BASE_URL=https://your-domain.com
      - STRIPE_SECRET_KEY=${STRIPE_SECRET_KEY}
      - STRIPE_WEBHOOK_SECRET=${STRIPE_WEBHOOK_SECRET}
      - DATABASE_URL=postgres://imageforge:${DB_PASSWORD}@postgres:5432/imageforge
      - REDIS_URL=redis://redis:6379
      - JWT_SECRET=${JWT_SECRET}
      - STORAGE_TYPE=local
      - STORAGE_PATH=/app/uploads
    ports:
      - "8080:8080"
    volumes:
      - uploads:/app/uploads
    depends_on:
      - postgres
      - redis
    restart: unless-stopped

  worker:
    build:
      context: ..
      dockerfile: docker/Dockerfile
    environment:
      - IMAGEFORGE_ROLE=worker
      - DATABASE_URL=postgres://imageforge:${DB_PASSWORD}@postgres:5432/imageforge
      - REDIS_URL=redis://redis:6379
      - JWT_SECRET=${JWT_SECRET}
      - STORAGE_TYPE=local
      - STORAGE_PATH=/app/uploads
    volumes:
      - uploads:/app/uploads
    depends_on:
      - postgres
      - redis
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: imageforge
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: imageforge
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro
    depends_on:
      - api
    restart: unless-stopped

volumes:
  uploads:
  postgres_data:
  redis_data:

docker/Dockerfile:

FROM rust:1.92-bookworm AS builder

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    cmake \
    nasm \
    pkg-config \
  && rm -rf /var/lib/apt/lists/*

COPY Cargo.toml Cargo.lock ./
COPY src ./src
COPY migrations ./migrations
COPY templates ./templates

RUN cargo build --release

# 前端构建阶段
FROM node:20-alpine AS frontend-builder

WORKDIR /app/frontend
COPY frontend/package*.json ./
RUN npm ci
COPY frontend ./
RUN npm run build

# 运行阶段
FROM debian:bookworm-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY --from=builder /app/target/release/imageforge ./imageforge
COPY --from=frontend-builder /app/frontend/dist ./static
COPY migrations ./migrations

RUN mkdir -p uploads

ENV HOST=0.0.0.0
ENV PORT=8080

EXPOSE 8080

CMD ["./imageforge"]

docker/nginx.conf:

events {
    worker_connections 1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    # 日志
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    # 文件上传大小限制
    client_max_body_size 100M;

    # Gzip
    gzip on;
    gzip_types text/plain text/css application/json application/javascript;

    upstream backend {
        server api:8080;
    }

    server {
        listen 80;
        server_name your-domain.com;
        return 301 https://$server_name$request_uri;
    }

    server {
        listen 443 ssl http2;
        server_name your-domain.com;

        ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;

        # SSL 配置
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
        ssl_prefer_server_ciphers off;

        # 静态文件
        location /static/ {
            proxy_pass http://backend;
            expires 30d;
            add_header Cache-Control "public, immutable";
        }

        # WebSocket
        location /ws/ {
            proxy_pass http://backend;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header Host $host;
            proxy_read_timeout 86400;
        }

        # API 和其他请求
        location / {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

部署步骤

# 1. 创建 .env 文件
cat > .env << EOF
DB_PASSWORD=your-secure-db-password
JWT_SECRET=your-very-long-random-jwt-secret-at-least-32-chars
STRIPE_SECRET_KEY=sk_live_xxx
STRIPE_WEBHOOK_SECRET=whsec_xxx
EOF

# 2. 获取 SSL 证书
sudo certbot certonly --standalone -d your-domain.com

# 3. 构建并启动
docker-compose -f docker/docker-compose.prod.yml up -d --build

# 4. 查看日志
docker-compose -f docker/docker-compose.prod.yml logs -f api

方案二：Kubernetes（大规模）

k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: imageforge
spec:
  replicas: 3
  selector:
    matchLabels:
      app: imageforge
  template:
    metadata:
      labels:
        app: imageforge
    spec:
      containers:
      - name: imageforge
        image: your-registry/imageforge:latest
        ports:
        - containerPort: 8080
        env:
        - name: IMAGEFORGE_ROLE
          value: api
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: imageforge-secrets
              key: database-url
        - name: REDIS_URL
          valueFrom:
            secretKeyRef:
              name: imageforge-secrets
              key: redis-url
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: imageforge-secrets
              key: jwt-secret
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2000m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: imageforge
spec:
  selector:
    app: imageforge
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: imageforge
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - your-domain.com
    secretName: imageforge-tls
  rules:
  - host: your-domain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: imageforge
            port:
              number: 80

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: imageforge-worker
spec:
  replicas: 2
  selector:
    matchLabels:
      app: imageforge-worker
  template:
    metadata:
      labels:
        app: imageforge-worker
    spec:
      containers:
      - name: imageforge-worker
        image: your-registry/imageforge:latest
        env:
        - name: IMAGEFORGE_ROLE
          value: worker
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: imageforge-secrets
              key: database-url
        - name: REDIS_URL
          valueFrom:
            secretKeyRef:
              name: imageforge-secrets
              key: redis-url
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: imageforge-secrets
              key: jwt-secret
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "4Gi"
            cpu: "4000m"

监控与日志

Prometheus 指标

应用暴露 /metrics 端点：

// 在代码中添加指标
use prometheus::{Counter, Histogram};

lazy_static! {
    static ref COMPRESSION_REQUESTS: Counter = Counter::new(
        "imageforge_compression_requests_total",
        "Total number of compression requests"
    ).unwrap();

    static ref COMPRESSION_DURATION: Histogram = Histogram::with_opts(
        HistogramOpts::new(
            "imageforge_compression_duration_seconds",
            "Time spent compressing images"
        )
    ).unwrap();
}

Grafana 仪表板

监控项目：

请求量 / QPS
响应时间 P50/P95/P99
错误率
压缩任务队列长度
CPU / 内存使用率
磁盘使用率

日志聚合

使用 ELK Stack 或 Loki：

# docker-compose 添加 Loki
loki:
  image: grafana/loki:2.9.0
  ports:
    - "3100:3100"
  command: -config.file=/etc/loki/local-config.yaml

promtail:
  image: grafana/promtail:2.9.0
  volumes:
    - /var/log:/var/log
    - ./promtail-config.yml:/etc/promtail/config.yml
  command: -config.file=/etc/promtail/config.yml

备份策略

数据库备份

#!/bin/bash
# backup.sh

DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR=/backups

# PostgreSQL 备份
docker exec postgres pg_dump -U imageforge imageforge | gzip > $BACKUP_DIR/db_$DATE.sql.gz

# 保留最近 7 天的备份
find $BACKUP_DIR -name "db_*.sql.gz" -mtime +7 -delete

# 可选：上传到 S3
# aws s3 cp $BACKUP_DIR/db_$DATE.sql.gz s3://your-bucket/backups/

添加到 crontab：

0 3 * * * /path/to/backup.sh

上传文件备份

如果使用本地存储，定期同步到 S3：

aws s3 sync /app/uploads s3://your-bucket/uploads --delete

故障排查

常见问题

1. 数据库连接失败

# 检查 PostgreSQL 状态
docker-compose logs postgres

# 测试连接
docker exec -it postgres psql -U imageforge -d imageforge -c "SELECT 1"

2. 压缩失败

# 检查应用日志
docker-compose logs api | grep ERROR
docker-compose logs worker | grep ERROR

# 检查磁盘空间
df -h

3. 内存不足

# 查看内存使用
docker stats

# 调整容器内存限制

4. 上传超时

# 检查 Nginx 配置
# client_max_body_size 和 proxy_read_timeout

健康检查端点

GET /health
{
  "status": "healthy",
  "database": "connected",
  "redis": "connected",
  "storage": "available",
  "uptime": 3600
}

15 KiB Raw Blame History Unescape Escape