This commit is contained in:
@@ -1,16 +1,15 @@
|
||||
# 可观测性设计(日志/指标/追踪)- ImageForge
|
||||
# 可观测性与告警 - ImageForge
|
||||
|
||||
目标:让“压缩效果、性能瓶颈、队列健康、计费正确性、滥用风险”都能被观测与告警,便于商用运营。
|
||||
目标:让“压缩效果、性能瓶颈、队列健康、计费正确性、滥用风险”都能被观测与告警,便于商用运营。下列请求标识和基础 Prometheus 指标已经实现;OpenTelemetry 和业务仪表板仍属于后续增强项。
|
||||
|
||||
---
|
||||
|
||||
## 1. 统一规范
|
||||
|
||||
### 1.1 请求标识
|
||||
- 每个 HTTP 请求生成 `request_id`(或从网关透传),写入:
|
||||
- 响应头:`X-Request-Id`
|
||||
- 日志字段:`request_id`
|
||||
- Trace:`trace_id/span_id`(如启用 OpenTelemetry)
|
||||
- API 会生成 `req_<uuid>`,也会接受由可信网关透传的安全 `X-Request-Id`。
|
||||
- 请求 ID 会写入全部响应的 `X-Request-Id`、成功/失败请求日志和 JSON 错误体。
|
||||
- 传入值仅允许 1-128 个 ASCII 字母、数字、点、下划线、冒号和连字符,避免日志注入。
|
||||
|
||||
### 1.2 日志格式
|
||||
- 结构化日志(JSON)优先,便于 Loki/ELK 聚合。
|
||||
@@ -25,40 +24,49 @@
|
||||
|
||||
---
|
||||
|
||||
## 2. 指标(Prometheus)
|
||||
## 2. 指标(Prometheus,已实现)
|
||||
|
||||
API 在 `/metrics` 暴露 Prometheus 文本格式。生产环境只应从宿主机或监控私网抓取,不要通过公开域名开放该路径:
|
||||
|
||||
```bash
|
||||
curl --fail http://127.0.0.1:18180/metrics
|
||||
```
|
||||
|
||||
压缩、S3 回退和死信累计值存放在 Redis Hash `metrics:imageforge`,因此 API 与独立 Worker 的事件会汇总到同一组指标。HTTP 请求与错误指标是 API 进程级指标,重启后归零。
|
||||
|
||||
### 2.1 API 服务指标
|
||||
请求类:
|
||||
- `http_requests_total{route,method,status}`
|
||||
- `http_request_duration_seconds_bucket{route,method}`
|
||||
- `imageforge_http_requests_total{method,status_class}`
|
||||
- `imageforge_http_request_duration_seconds_bucket`
|
||||
|
||||
鉴权与风控:
|
||||
- `auth_fail_total{reason}`
|
||||
- `rate_limited_total{scope}`(anonymous/user/api_key)
|
||||
- `quota_exceeded_total{plan}`
|
||||
错误与风控:
|
||||
- `imageforge_errors_total{code}`,包含 `RATE_LIMITED`、`QUOTA_EXCEEDED` 等业务错误码
|
||||
|
||||
计费链路:
|
||||
- `billing_webhook_total{provider,event_type,result}`
|
||||
- `subscription_state_total{state}`
|
||||
- `invoice_total{status}`
|
||||
依赖与队列:
|
||||
- `imageforge_dependency_up{dependency="database|redis"}`
|
||||
- `imageforge_active_tasks`
|
||||
- `imageforge_queue_messages{state="stream|pending|dead_letter"}`
|
||||
|
||||
### 2.2 Worker 指标
|
||||
队列与吞吐:
|
||||
- `jobs_received_total`
|
||||
- `jobs_inflight`
|
||||
- `jobs_completed_total{result}`
|
||||
- `job_duration_seconds_bucket{format,level}`
|
||||
- `imageforge_compressions_total{result}`
|
||||
- `imageforge_compression_duration_seconds_sum/count`
|
||||
|
||||
压缩效果:
|
||||
- `bytes_in_total`、`bytes_out_total`、`bytes_saved_total`
|
||||
- `compression_ratio_bucket{format,level}`
|
||||
- `imageforge_compression_bytes_total{direction="input|output"}`
|
||||
|
||||
资源与异常:
|
||||
- `decode_failed_total{reason}`
|
||||
- `pixel_limit_hit_total`
|
||||
- `imageforge_storage_fallbacks_total`
|
||||
- `imageforge_dead_letters_total`
|
||||
|
||||
### 2.3 Redis/队列指标(可选)
|
||||
- Streams 消费延迟、pending 数量、dead-letter 数量(如实现)。
|
||||
Prometheus 抓取示例:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: imageforge
|
||||
static_configs:
|
||||
- targets: ['127.0.0.1:18180']
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user