# Prometheus配置示例
scrape_configs:
  - job_name: 'user-service'
    static_configs:
      - targets: ['user-service:8080']
    metrics_path: '/actuator/prometheus'
    
  - job_name: 'order-service'
    static_configs:
      - targets: ['order-service:8080']
    metrics_path: '/actuator/prometheus'
    
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

数据存储层

负责存储和管理监控数据。

# Prometheus存储配置
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  
rule_files:
  - "alert.rules.yml"
  
storage:
  tsdb:
    path: /prometheus/data
    retention: 30d

数据展示层

负责将监控数据以可视化的方式展示。

{
  "dashboard": {
    "title": "微服务监控面板",
    "panels": [
      {
        "title": "服务响应时间",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))",
            "legendFormat": "{{service}}"
          }
        ]
      },
      {
        "title": "服务错误率",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service)",
            "legendFormat": "{{service}}"
          }
        ]
      }
    ]
  }
}

监控数据流

建立清晰的监控数据流可以确保监控体系的有效运行。

数据采集

Pull模式：监控系统主动拉取指标数据
Push模式：应用主动推送指标数据
事件驱动：基于事件的监控数据收集

数据处理

数据清洗：过滤无效和异常数据
数据聚合：对原始数据进行聚合计算
数据转换：将数据转换为标准格式

数据存储

时序数据库：存储时间序列数据
日志存储：存储结构化和非结构化日志
追踪存储：存储分布式追踪数据

使用 Prometheus 与 Grafana 进行性能监控

Prometheus和Grafana是目前最流行的开源监控解决方案组合，它们为微服务监控提供了强大的功能。

1. Prometheus核心特性

多维数据模型

Prometheus使用多维数据模型，通过标签（labels）来标识时间序列。

http_requests_total{method="POST", handler="/api/users", status="200"} 1234
http_requests_total{method="GET", handler="/api/orders", status="500"} 5

强大的查询语言

Prometheus提供了强大的查询语言PromQL，可以进行复杂的数据分析。

# 计算HTTP请求的95%分位数响应时间
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# 计算服务的错误率
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# 计算CPU使用率
100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

服务发现

Prometheus支持多种服务发现机制，可以自动发现监控目标。

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true

2. Grafana可视化

Grafana提供了丰富的可视化功能，可以将监控数据以图表的形式展示。

仪表板设计

系统概览面板

{
  "title": "系统概览",
  "panels": [
    {
      "title": "CPU使用率",
      "type": "gauge",
      "targets": [
        {
          "expr": "100 - (avg(irate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
          "legendFormat": "CPU使用率"
        }
      ]
    },
    {
      "title": "内存使用率",
      "type": "gauge",
      "targets": [
        {
          "expr": "(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100",
          "legendFormat": "内存使用率"
        }
      ]
    },
    {
      "title": "磁盘使用率",
      "type": "gauge",
      "targets": [
        {
          "expr": "(node_filesystem_size_bytes{mountpoint=\"/\"} - node_filesystem_free_bytes{mountpoint=\"/\"}) / node_filesystem_size_bytes{mountpoint=\"/\"} * 100",
          "legendFormat": "磁盘使用率"
        }
      ]
    }
  ]
}

应用性能面板

{
  "title": "应用性能",
  "panels": [
    {
      "title": "请求速率",
      "type": "graph",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total[5m])) by (service)",
          "legendFormat": "{{service}}"
        }
      ]
    },
    {
      "title": "错误率",
      "type": "graph",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service)",
          "legendFormat": "{{service}}"
        }
      ]
    },
    {
      "title": "95%响应时间",
      "type": "graph",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))",
          "legendFormat": "{{service}}"
        }
      ]
    }
  ]
}

3. 告警规则配置

Prometheus支持基于指标的告警规则配置。

# alert.rules.yml
groups:
- name: example
  rules:
  - alert: HighCPUUsage
    expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage detected"
      description: "CPU usage is above 80% for more than 5 minutes"

  - alert: HighMemoryUsage
    expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 85
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High memory usage detected"
      description: "Memory usage is above 85% for more than 5 minutes"

  - alert: ServiceDown
    expr: up == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Service is down"
      description: "Service {{ $labels.instance }} is down"

分布式日志与追踪的监控

在微服务架构中，分布式日志和追踪是理解系统行为和排查问题的重要手段。

1. 分布式日志管理

结构化日志

使用结构化日志格式便于日志的收集、存储和分析。

{
  "timestamp": "2025-08-31T10:30:45.123Z",
  "level": "INFO",
  "service": "user-service",
  "traceId": "abc123def456",
  "spanId": "789ghi012",
  "message": "User login successful",
  "userId": "12345",
  "ip": "192.168.1.100"
}

日志收集架构

建立统一的日志收集架构可以有效管理分布式日志。

EFK Stack

EFK（Elasticsearch + Fluentd + Kibana）是流行的日志解决方案。

Fluentd配置

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  <parse>
    @type json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

<filter kubernetes.**>
  @type kubernetes_metadata
</filter>

<match kubernetes.**>
  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true
</match>

Elasticsearch索引模板

{
  "index_patterns": ["logstash-*"],
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "timestamp": { "type": "date" },
      "level": { "type": "keyword" },
      "service": { "type": "keyword" },
      "traceId": { "type": "keyword" },
      "message": { "type": "text" }
    }
  }
}

日志分析

通过日志分析可以发现系统中的异常和问题。

错误日志分析

GET /logstash-*/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "level": "ERROR" } },
        { "range": { "timestamp": { "gte": "now-1h" } } }
      ]
    }
  },
  "aggs": {
    "by_service": {
      "terms": { "field": "service" }
    }
  }
}

性能日志分析

GET /logstash-*/_search
{
  "query": {
    "bool": {
      "must": [
        { "range": { "duration": { "gte": 1000 } } },
        { "range": { "timestamp": { "gte": "now-1h" } } }
      ]
    }
  },
  "aggs": {
    "avg_duration": {
      "avg": { "field": "duration" }
    }
  }
}

2. 分布式追踪

分布式追踪用于跟踪请求在微服务系统中的流转过程。

OpenTelemetry

OpenTelemetry是云原生基金会的可观测性框架。

SDK集成

// Java应用集成示例
@Configuration
public class TracingConfiguration {
    
    @Bean
    public OpenTelemetry openTelemetry() {
        return OpenTelemetrySdk.builder()
            .setTracerProvider(SdkTracerProvider.builder()
                .addSpanProcessor(BatchSpanProcessor.builder(
                    OtlpGrpcSpanExporter.builder()
                        .setEndpoint("http://otel-collector:4317")
                        .build())
                    .build())
                .build())
            .buildAndRegisterGlobal();
    }
    
    @Bean
    public Tracer tracer(OpenTelemetry openTelemetry) {
        return openTelemetry.getTracer("user-service");
    }
}

自动 instrumentation

# Kubernetes部署配置
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
spec:
  template:
    spec:
      containers:
      - name: user-service
        image: user-service:latest
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://otel-collector:4317"
        - name: OTEL_SERVICE_NAME
          value: "user-service"

Jaeger

Jaeger是流行的分布式追踪系统。

架构组件

Jaeger Agent：接收客户端发送的追踪数据
Jaeger Collector：收集和处理追踪数据
Jaeger Query：提供查询接口和UI
存储后端：存储追踪数据（Cassandra、Elasticsearch等）

追踪数据结构

{
  "traceId": "abc123def456",
  "spans": [
    {
      "spanId": "789ghi012",
      "operationName": "getUser",
      "startTime": "2025-08-31T10:30:45.123Z",
      "duration": 150,
      "tags": [
        { "key": "http.status_code", "value": "200" },
        { "key": "userId", "value": "12345" }
      ],
      "logs": [
        { "timestamp": "2025-08-31T10:30:45.150Z", "fields": [{ "key": "event", "value": "user found" }] }
      ]
    }
  ]
}

微服务中的智能告警与异常检测

智能告警和异常检测可以帮助我们及时发现系统中的问题，减少故障响应时间。

1. 告警策略设计

告警级别

建立多级告警机制可以区分问题的严重程度。

紧急告警（Critical）

系统宕机：核心服务不可用
数据丢失：关键数据丢失或损坏
安全事件：安全漏洞或攻击事件

重要告警（High）

性能下降：关键性能指标严重下降
资源耗尽：CPU、内存、磁盘等资源即将耗尽
错误率激增：服务错误率异常升高

警告告警（Warning）

阈值接近：监控指标接近预设阈值
异常波动：指标出现异常波动
潜在问题：可能影响系统稳定的潜在问题

告警规则

建立合理的告警规则可以减少误报和漏报。

静态阈值告警

# CPU使用率超过80%告警
ALERT HighCPUUsage
  IF 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
  FOR 5m
  LABELS { severity = "warning" }
  ANNOTATIONS {
    summary = "High CPU usage detected",
    description = "CPU usage is above 80% for more than 5 minutes"
  }

动态阈值告警

# 基于历史数据的动态阈值告警
ALERT AnomalousRequestRate
  IF rate(http_requests_total[5m]) > (avg_over_time(rate(http_requests_total[1h])[1d:1h]) * 2)
  FOR 5m
  LABELS { severity = "warning" }
  ANNOTATIONS {
    summary = "Anomalous request rate detected",
    description = "Request rate is 2x higher than historical average"
  }

2. 异常检测算法

统计学方法

基于统计学的异常检测方法简单有效。

3-Sigma规则

import numpy as np

def detect_anomalies_3sigma(data):
    mean = np.mean(data)
    std = np.std(data)
    threshold_upper = mean + 3 * std
    threshold_lower = mean - 3 * std
    
    anomalies = []
    for i, value in enumerate(data):
        if value > threshold_upper or value < threshold_lower:
            anomalies.append((i, value))
    
    return anomalies

四分位数方法

def detect_anomalies_iqr(data):
    q1 = np.percentile(data, 25)
    q3 = np.percentile(data, 75)
    iqr = q3 - q1
    lower_bound = q1 - 1.5 * iqr
    upper_bound = q3 + 1.5 * iqr
    
    anomalies = []
    for i, value in enumerate(data):
        if value < lower_bound or value > upper_bound:
            anomalies.append((i, value))
    
    return anomalies

机器学习方法

基于机器学习的异常检测可以处理更复杂的情况。

孤立森林

from sklearn.ensemble import IsolationForest

def detect_anomalies_isolation_forest(data):
    clf = IsolationForest(contamination=0.1)
    anomalies = clf.fit_predict(data)
    
    result = []
    for i, label in enumerate(anomalies):
        if label == -1:  # -1表示异常点
            result.append((i, data[i]))
    
    return result

LSTM异常检测

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense

def build_lstm_model(sequence_length, n_features):
    model = Sequential([
        LSTM(50, activation='relu', input_shape=(sequence_length, n_features)),
        Dense(1)
    ])
    model.compile(optimizer='adam', loss='mse')
    return model

def detect_anomalies_lstm(model, data, threshold):
    predictions = model.predict(data)
    mse = np.mean(np.power(data - predictions, 2), axis=1)
    anomalies = mse > threshold
    return anomalies

3. 告警通知与处理

通知渠道

建立多元化的告警通知渠道确保告警及时送达。

邮件通知

# Alertmanager配置
receivers:
- name: email-receiver
  email_configs:
  - to: 'ops-team@example.com'
    from: 'alertmanager@example.com'
    smarthost: 'smtp.example.com:587'
    auth_username: 'alertmanager'
    auth_password: 'password'

Slack通知

receivers:
- name: slack-receiver
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
    channel: '#alerts'
    send_resolved: true

Webhook通知

receivers:
- name: webhook-receiver
  webhook_configs:
  - url: 'http://internal-system/webhook'
    send_resolved: true

告警抑制

建立告警抑制机制可以减少告警噪音。

# Alertmanager抑制规则
inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'
  equal: ['alertname', 'service']

告警分组

通过告警分组可以将相关的告警合并处理。

# Alertmanager路由配置
route:
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: 'default-receiver'