Prometheus与Kubernetes中的监控：云原生环境下的可观测性实践

老马啸西风2025/8/31大约 8 分钟

在前一篇文章中，我们介绍了微服务监控工具与技术栈的整体情况。本文将深入探讨Prometheus在Kubernetes环境中的监控实践，包括服务发现、指标收集、告警配置以及与Kubernetes原生组件的集成。

Prometheus在Kubernetes中的部署

Helm部署方式

使用Helm Chart部署Prometheus是最常见的方式：

# 添加Prometheus社区Helm仓库
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 安装Prometheus
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false

自定义Values配置

# prometheus-values.yaml
prometheus:
  prometheusSpec:
    # 启用服务发现
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}
    podMonitorSelector: {}
    podMonitorNamespaceSelector: {}
    
    # 资源限制
    resources:
      limits:
        cpu: 1000m
        memory: 2Gi
      requests:
        cpu: 500m
        memory: 1Gi
    
    # 存储配置
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          resources:
            requests:
              storage: 50Gi
    
    # 保留策略
    retention: 30d
    retentionSize: 30GB
    
    # 外部标签
    externalLabels:
      cluster: production
      region: us-west-2

# Alertmanager配置
alertmanager:
  config:
    global:
      resolve_timeout: 5m
      smtp_smarthost: 'smtp.gmail.com:587'
      smtp_from: 'alerts@example.com'
      smtp_auth_username: 'alerts@example.com'
      smtp_auth_password: 'password'
    
    route:
      group_by: ['alertname', 'job']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'slack-notifications'
    
    receivers:
      - name: 'slack-notifications'
        slack_configs:
          - api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
            channel: '#alerts'
            send_resolved: true

grafana:
  adminPassword: admin123
  persistence:
    enabled: true
    size: 10Gi
  plugins:
    - grafana-piechart-panel
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: 'default'
          orgId: 1
          folder: ''
          type: file
          disableDeletion: false
          editable: true
          options:
            path: /var/lib/grafana/dashboards/default

Kubernetes服务发现集成

基于注解的服务发现

通过Pod注解自动发现监控目标：

# deployment-with-monitoring.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
  template:
    metadata:
      labels:
        app: user-service
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/actuator/prometheus"
        prometheus.io/port: "8080"
    spec:
      containers:
      - name: user-service
        image: user-service:1.2.3
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /actuator/health
            port: 8080
        livenessProbe:
          httpGet:
            path: /actuator/health
            port: 8080

ServiceMonitor配置

使用Prometheus Operator的ServiceMonitor资源：

# service-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: user-service-monitor
  namespace: monitoring
  labels:
    app: user-service
spec:
  selector:
    matchLabels:
      app: user-service
  namespaceSelector:
    matchNames:
      - default
  endpoints:
  - port: http
    path: /actuator/prometheus
    interval: 30s
    scrapeTimeout: 10s
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_name]
      targetLabel: pod
    - sourceLabels: [__meta_kubernetes_pod_container_name]
      targetLabel: container
    - sourceLabels: [__meta_kubernetes_namespace]
      targetLabel: namespace
    metricRelabelings:
    - sourceLabels: [__name__]
      regex: 'jvm_gc_collection_seconds.*'
      action: drop

PodMonitor配置

直接监控Pod指标：

# pod-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: user-service-pod-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: user-service
  namespaceSelector:
    matchNames:
      - default
  podMetricsEndpoints:
  - port: http
    path: /actuator/prometheus
    interval: 30s
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_name]
      targetLabel: pod
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      targetLabel: node

Kubernetes原生组件监控

kubelet监控

# kubelet-service-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubelet
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: https-metrics
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    path: /metrics/cadvisor
    port: https-metrics
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      k8s-app: kubelet

API Server监控

# apiserver-service-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-apiserver
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: https
    scheme: https
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      serverName: kubernetes
  jobLabel: component
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      component: apiserver
      provider: kubernetes

CoreDNS监控

# coredns-service-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: coredns
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 15s
    port: metrics
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      k8s-app: kube-dns

自定义指标收集

应用指标暴露

@RestController
public class KubernetesMetricsController {
    
    private final MeterRegistry meterRegistry;
    private final Counter kubernetesPodRestarts;
    private final Gauge kubernetesPodStatus;
    private final Timer kubernetesApiOperationDuration;
    
    public KubernetesMetricsController(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
        
        // Pod重启计数
        this.kubernetesPodRestarts = Counter.builder("kubernetes.pod.restarts")
            .description("Number of pod restarts")
            .tags("service", "user-service")
            .register(meterRegistry);
            
        // Pod状态
        this.kubernetesPodStatus = Gauge.builder("kubernetes.pod.status")
            .description("Current pod status")
            .tags("service", "user-service")
            .register(meterRegistry, this, KubernetesMetricsController::getPodStatus);
            
        // API操作耗时
        this.kubernetesApiOperationDuration = Timer.builder("kubernetes.api.operation.duration")
            .description("Duration of Kubernetes API operations")
            .tags("service", "user-service")
            .register(meterRegistry);
    }
    
    @GetMapping("/metrics/kubernetes")
    public String getKubernetesMetrics() {
        // 收集Kubernetes相关指标
        collectKubernetesMetrics();
        
        // 返回Prometheus格式的指标
        return scrapePrometheusMetrics();
    }
    
    private void collectKubernetesMetrics() {
        try {
            // 获取Pod信息
            String podName = System.getenv("HOSTNAME");
            String namespace = System.getenv("NAMESPACE");
            
            // 查询Pod状态
            V1Pod pod = kubernetesClient.pods()
                .inNamespace(namespace)
                .withName(podName)
                .get();
                
            if (pod != null) {
                // 记录重启次数
                Integer restartCount = pod.getStatus().getContainerStatuses()
                    .stream()
                    .mapToInt(V1ContainerStatus::getRestartCount)
                    .sum();
                    
                // 如果重启次数增加，记录指标
                if (restartCount > 0) {
                    kubernetesPodRestarts.increment(restartCount);
                }
            }
            
            // 记录API操作耗时
            Timer.Sample sample = Timer.start(meterRegistry);
            kubernetesClient.pods().list();
            sample.stop(kubernetesApiOperationDuration);
            
        } catch (Exception e) {
            log.warn("Failed to collect Kubernetes metrics", e);
        }
    }
    
    private double getPodStatus() {
        try {
            String podName = System.getenv("HOSTNAME");
            String namespace = System.getenv("NAMESPACE");
            
            V1Pod pod = kubernetesClient.pods()
                .inNamespace(namespace)
                .withName(podName)
                .get();
                
            if (pod != null && "Running".equals(pod.getStatus().getPhase())) {
                return 1.0; // 正常运行
            } else {
                return 0.0; // 异常状态
            }
        } catch (Exception e) {
            return -1.0; // 无法获取状态
        }
    }
}

Sidecar模式指标收集

# deployment-with-sidecar.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
  template:
    metadata:
      labels:
        app: user-service
    spec:
      containers:
      - name: user-service
        image: user-service:1.2.3
        ports:
        - containerPort: 8080
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
      - name: prometheus-exporter
        image: prometheus-exporter:1.0.0
        ports:
        - containerPort: 9100
        env:
        - name: TARGET_SERVICE
          value: "localhost:8080"
        - name: SCRAPE_INTERVAL
          value: "30s"
        volumeMounts:
        - name: metrics-config
          mountPath: /etc/prometheus-exporter
      volumes:
      - name: metrics-config
        configMap:
          name: user-service-metrics-config

告警规则配置

Kubernetes特定告警

# kubernetes-alerts.yaml
groups:
  - name: kubernetes-alerts
    rules:
      # Pod重启告警
      - alert: KubernetesPodRestarting
        expr: increase(kubernetes_pod_restarts[10m]) > 2
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Pod restarting frequently"
          description: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} has restarted {{ $value }} times in the last 10 minutes"
          
      # Pod状态异常告警
      - alert: KubernetesPodNotReady
        expr: kube_pod_status_ready{condition="true"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Pod not ready"
          description: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is not ready"
          
      # 节点不可用告警
      - alert: KubernetesNodeNotReady
        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Node not ready"
          description: "Node {{ $labels.node }} is not ready"
          
      # 资源不足告警
      - alert: KubernetesMemoryUsageHigh
        expr: (kube_pod_container_resource_requests{resource="memory"} / kube_node_status_allocatable{resource="memory"}) > 0.9
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage"
          description: "Memory usage on node {{ $labels.node }} is above 90%"
          
      # 磁盘空间不足告警
      - alert: KubernetesDiskSpaceLow
        expr: kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space"
          description: "Disk space on volume {{ $labels.persistentvolumeclaim }} is below 10%"

应用级告警

# application-alerts.yaml
groups:
  - name: application-alerts
    rules:
      # 高错误率告警
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          description: "Error rate for service {{ $labels.service }} is {{ $value | humanizePercentage }}"
          
      # 高延迟告警
      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "High latency detected"
          description: "95th percentile latency for service {{ $labels.service }} is {{ $value }} seconds"
          
      # 低可用性告警
      - alert: LowAvailability
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service is down"
          description: "Service {{ $labels.job }} is not responding"
          
      # 业务指标告警
      - alert: LowUserActivity
        expr: rate(user_activity_total[1h]) < 10
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Low user activity"
          description: "User activity rate is below 10 per hour"

Grafana仪表板配置

Kubernetes集群仪表板

{
  "dashboard": {
    "id": null,
    "title": "Kubernetes Cluster Monitoring",
    "tags": ["kubernetes", "cluster"],
    "timezone": "browser",
    "schemaVersion": 16,
    "version": 0,
    "panels": [
      {
        "id": 1,
        "type": "stat",
        "title": "Cluster Status",
        "gridPos": {
          "x": 0,
          "y": 0,
          "w": 6,
          "h": 3
        },
        "targets": [
          {
            "expr": "sum(up{job=~\"kubernetes-.*\"})",
            "refId": "A"
          }
        ],
        "thresholds": "0,1",
        "colorBackground": true,
        "colors": ["#d44a3a", "rgba(237, 129, 40, 0.89)", "#299c46"]
      },
      {
        "id": 2,
        "type": "graph",
        "title": "Node CPU Usage",
        "gridPos": {
          "x": 6,
          "y": 0,
          "w": 18,
          "h": 6
        },
        "targets": [
          {
            "expr": "100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
            "legendFormat": "{{instance}}",
            "refId": "A"
          }
        ],
        "xaxis": {
          "mode": "time"
        },
        "yaxes": [
          {
            "format": "percent",
            "label": "CPU Usage %"
          },
          {
            "format": "short"
          }
        ]
      },
      {
        "id": 3,
        "type": "graph",
        "title": "Node Memory Usage",
        "gridPos": {
          "x": 0,
          "y": 6,
          "w": 12,
          "h": 6
        },
        "targets": [
          {
            "expr": "(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100",
            "legendFormat": "{{instance}}",
            "refId": "A"
          }
        ],
        "xaxis": {
          "mode": "time"
        },
        "yaxes": [
          {
            "format": "percent",
            "label": "Memory Usage %"
          },
          {
            "format": "short"
          }
        ]
      },
      {
        "id": 4,
        "type": "graph",
        "title": "Pod Status",
        "gridPos": {
          "x": 12,
          "y": 6,
          "w": 12,
          "h": 6
        },
        "targets": [
          {
            "expr": "sum by (phase) (kube_pod_status_phase)",
            "legendFormat": "{{phase}}",
            "refId": "A"
          }
        ],
        "xaxis": {
          "mode": "time"
        },
        "yaxes": [
          {
            "format": "short",
            "label": "Pod Count"
          },
          {
            "format": "short"
          }
        ]
      }
    ]
  }
}

应用服务仪表板

{
  "dashboard": {
    "id": null,
    "title": "Microservices Application Monitoring",
    "tags": ["microservices", "application"],
    "timezone": "browser",
    "schemaVersion": 16,
    "version": 0,
    "panels": [
      {
        "id": 1,
        "type": "graph",
        "title": "Request Rate by Service",
        "gridPos": {
          "x": 0,
          "y": 0,
          "w": 12,
          "h": 6
        },
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])",
            "legendFormat": "{{service}} - {{status}}",
            "refId": "A"
          }
        ],
        "xaxis": {
          "mode": "time"
        },
        "yaxes": [
          {
            "format": "reqps",
            "label": "Requests per second"
          },
          {
            "format": "short"
          }
        ]
      },
      {
        "id": 2,
        "type": "graph",
        "title": "Response Time Percentiles",
        "gridPos": {
          "x": 12,
          "y": 0,
          "w": 12,
          "h": 6
        },
        "targets": [
          {
            "expr": "histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "{{service}} - 50th percentile",
            "refId": "A"
          },
          {
            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "{{service}} - 95th percentile",
            "refId": "B"
          },
          {
            "expr": "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "{{service}} - 99th percentile",
            "refId": "C"
          }
        ],
        "xaxis": {
          "mode": "time"
        },
        "yaxes": [
          {
            "format": "s",
            "label": "Response time"
          },
          {
            "format": "short"
          }
        ]
      },
      {
        "id": 3,
        "type": "stat",
        "title": "Error Rate",
        "gridPos": {
          "x": 0,
          "y": 6,
          "w": 6,
          "h": 3
        },
        "targets": [
          {
            "expr": "rate(http_requests_total{status=~\"5..\"}[5m]) / rate(http_requests_total[5m]) * 100",
            "refId": "A"
          }
        ],
        "format": "percent",
        "thresholds": "1,5",
        "colorBackground": true,
        "colors": ["#299c46", "rgba(237, 129, 40, 0.89)", "#d44a3a"]
      },
      {
        "id": 4,
        "type": "graph",
        "title": "JVM Memory Usage",
        "gridPos": {
          "x": 6,
          "y": 6,
          "w": 18,
          "h": 6
        },
        "targets": [
          {
            "expr": "jvm_memory_bytes_used{area=\"heap\"} / jvm_memory_bytes_max{area=\"heap\"} * 100",
            "legendFormat": "{{service}} - Heap",
            "refId": "A"
          },
          {
            "expr": "jvm_memory_bytes_used{area=\"nonheap\"} / jvm_memory_bytes_max{area=\"nonheap\"} * 100",
            "legendFormat": "{{service}} - Non-Heap",
            "refId": "B"
          }
        ],
        "xaxis": {
          "mode": "time"
        },
        "yaxes": [
          {
            "format": "percent",
            "label": "Memory Usage %"
          },
          {
            "format": "short"
          }
        ]
      }
    ]
  }
}

性能优化与最佳实践

Prometheus性能调优

# prometheus-performance-tuning.yaml
prometheus:
  prometheusSpec:
    # 查询并发限制
    query:
      maxConcurrency: 20
      timeout: 2m
    
    # 存储优化
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          resources:
            requests:
              storage: 100Gi
    
    # 压缩策略
    enableFeatures:
      - "promql-at-modifier"
      - "promql-negative-offset"
    
    # 外部标签
    externalLabels:
      cluster: production
      region: us-west-2
    
    # 远程写配置（用于长期存储）
    remoteWrite:
      - url: http://remote-storage:9090/api/v1/write
        writeRelabelConfigs:
          - sourceLabels: [__name__]
            regex: 'up|scrape_.*'
            action: drop

监控数据治理

@Component
public class KubernetesMonitoringGovernance {
    
    @Scheduled(cron = "0 0 2 * * ?") // 每天凌晨2点执行
    public void enforceMonitoringGovernance() {
        // 清理过期的监控数据
        cleanupExpiredMetrics();
        
        // 优化存储性能
        optimizeStoragePerformance();
        
        // 验证监控配置
        validateMonitoringConfiguration();
        
        // 生成监控报告
        generateMonitoringReport();
    }
    
    private void cleanupExpiredMetrics() {
        // 删除超过保留期限的指标数据
        Duration retentionPeriod = Duration.ofDays(30);
        prometheusAdminClient.deleteMetricsOlderThan(retentionPeriod);
    }
    
    private void optimizeStoragePerformance() {
        // 执行存储优化操作
        prometheusAdminClient.compactStorage();
        prometheusAdminClient.rebuildIndex();
    }
    
    private void validateMonitoringConfiguration() {
        // 验证ServiceMonitor配置
        List<ServiceMonitor> serviceMonitors = kubernetesClient
            .resources(ServiceMonitor.class)
            .inNamespace("monitoring")
            .list()
            .getItems();
            
        for (ServiceMonitor monitor : serviceMonitors) {
            validateServiceMonitor(monitor);
        }
    }
    
    private void generateMonitoringReport() {
        // 生成监控健康报告
        MonitoringReport report = new MonitoringReport();
        report.setTimestamp(Instant.now());
        report.setClusterStatus(getClusterStatus());
        report.setServiceCoverage(getServiceCoverage());
        report.setAlertSummary(getAlertSummary());
        
        // 发送报告
        notificationService.sendMonitoringReport(report);
    }
}

故障排查与维护

常见问题诊断

# 检查Prometheus状态
kubectl -n monitoring get pods -l app=prometheus
kubectl -n monitoring logs -l app=prometheus

# 检查服务发现状态
kubectl -n monitoring exec -it prometheus-prometheus-0 -- wget -qO- http://localhost:9090/service-discovery

# 检查目标状态
kubectl -n monitoring exec -it prometheus-prometheus-0 -- wget -qO- http://localhost:9090/targets

# 检查告警状态
kubectl -n monitoring exec -it prometheus-prometheus-0 -- wget -qO- http://localhost:9090/alerts

监控系统维护

# maintenance-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: prometheus-maintenance
  namespace: monitoring
spec:
  schedule: "0 2 * * *" # 每天凌晨2点执行
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: maintenance
            image: prometheus-maintenance:1.0.0
            command:
            - /bin/sh
            - -c
            - |
              # 执行维护任务
              echo "Starting Prometheus maintenance..."
              
              # 清理过期数据
              promtool tsdb delete --min-time=24h /prometheus/data
              
              # 重建索引
              promtool tsdb create-blocks-from openmetrics /prometheus/data
              
              echo "Maintenance completed successfully"
          restartPolicy: OnFailure

总结

Prometheus与Kubernetes的集成提供了强大的云原生监控能力。通过合理配置服务发现、指标收集、告警规则和可视化仪表板，我们可以构建一个完整的Kubernetes监控解决方案。

关键要点包括：

使用Prometheus Operator简化部署和管理
实现自动化的服务发现机制
配置针对性的告警规则
创建直观的监控仪表板
实施性能优化和数据治理策略

在下一章中，我们将探讨分布式追踪与性能分析，包括OpenTracing、Jaeger和Zipkin的使用方法，以及如何通过追踪数据进行性能瓶颈分析。