Kubernetes Monitoring with Prometheus and Grafana

This project monitors a Kubernetes microservices architecture using Prometheus and Grafana. It covers microservices, Redis, and a Node.js application.

Table of Contents

• Introduction
• Technologies Used
• Prerequisites
• Installation
• Usage
• Project Description (Tasks)
  • Alerting System Testing
  • Redis Monitoring
  • Node.js App Monitoring

Introduction

I set up monitoring for a Kubernetes-based microservices application using Prometheus and Grafana. This included configuring alerts, monitoring a Redis instance, and tracking metrics from a Node.js application. The goal was to create a comprehensive monitoring solution for a typical Kubernetes environment.

Technologies Used

• Kubernetes: Orchestration.
• Prometheus: Metrics.
• Grafana: Visualization.
• Redis: Data store.
• Node.js: App runtime.
• Helm: Package manager.

Prerequisites

• AWS EKS Cluster
• Helm.

Installation

1. Create Namespace:

   kubectl create namespace monitoring
   

2. Install Prometheus Stack:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update
   helm install prometheus-monitoring prometheus-community/kube-prometheus-stack -n monitoring
   

Usage

1. Prometheus UI:

   kubectl port-forward svc/prometheus-monitoring-kube-prometheus 9090:9090 -n monitoring
   

   (http://localhost:9090).

2. Alertmanager UI:

   kubectl port-forward svc/prometheus-monitoring-kube-alertmanager 9093:9093 -n monitoring
   

   (http://localhost:9093).

3. Grafana Dashboard:

   kubectl port-forward -n monitoring svc/prometheus-monitoring-grafana 3000:80
   

   (http://localhost:3000).

Project Description (Tasks)

Alerting System Testing

• Description: Tested alert rules for CPU and pod failures, visualized in Grafana.

  • Goal: Verify Prometheus and Alertmanager setup.

• Procedure:

  1. Apply Configs:

     kubectl apply -f alert-rules.yaml -n monitoring
     kubectl apply -f alert-manager-config.yaml -n monitoring
     

     (Applies Prometheus rules and Alertmanager configuration)
  2. Simulate CPU Load: kubectl run cpu-test --image=containerstack/cpustress -- --cpu 4 --timeout 60s --metrics-brief. (Creates CPU load)
  3. Observe Alerts: In Prometheus UI, check the "Alerts" section. (Check for HostHighCpuLoad alert)

  Screenshot: Prometheus UI showing the HostHighCpuLoad alert.

  4. Simulate Pod Failure: Deploy a pod with an invalid image. (Causes pod crashloop)
  5. Observe Alerts: In Prometheus UI, check the "Alerts" section. (Check for PodCrashloop alert)
  6. Grafana: Create dashboard. (Visualize CPU, pod status)

  Screenshot: Grafana dashboard showing CPU usage and pod status.

Redis Monitoring

• Description: Monitored Redis, including alert rules, visualized in Grafana.

  • Goal: Verify Redis monitoring and alerting.

• Procedure:

  1. Install Exporter:

     helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
     helm repo update
     helm install redis-exporter prometheus-community/prometheus-redis-exporter -f redis-values.yaml -n monitoring
     

     (Deploys Redis Exporter)
  2. Apply Rules: kubectl apply -f redis-alert-rules.yaml -n monitoring. (Applies Prometheus rules)
  3. Verify Exporter: In Prometheus UI, check the "Targets" section. (Confirm Exporter target)

  Screenshot: Prometheus Targets with Redis Exporter

  4. Simulate Redis Down: kubectl scale deployment redis --replicas=0 -n monitoring. (Scale Redis to 0)
  5. Observe Alerts: In Prometheus UI, check the "Alerts" section. (Check for RedisDown alert)

  Screenshot: Prometheus UI showing the RedisDown alert.

  6. Grafana: Create dashboard. (Visualize Redis metrics)

  Screenshot: Grafana dashboard showing Redis metrics.

  7. Restore Redis: kubectl scale deployment redis --replicas=2 -n monitoring. (Restore Redis scale)

Node.js App Monitoring

• Description: Monitored a Node.js app with Prometheus and visualized in Grafana.

  • Goal: Verify custom application monitoring.

• Procedure:

  1. Deploy App: kubectl apply -f node-js-app-monitoring-k8s-config.yaml -n monitoring. (Deploys Node.js app)
  2. Access App: Determine IP/port. (kubectl get service)
  3. View Metrics: http://<ip>:<port>/metrics.
  4. Verify Exporter: In Prometheus UI, check the "Targets" section.
  5. Grafana: Create dashboard.

  Screenshot: Grafana dashboard showing Node.js application request duration. Screenshot: Grafana dashboard showing Node.js application request total.