Setting Up Load Testing and Monitoring with Docker, k6, Prometheus, Grafana, and InfluxDB

This guide covers configuring k6 for load testing, Prometheus and Grafana for monitoring, and InfluxDB for optional long-term metric storage using Docker. Each tool will be configured to work together in a containerized environment, allowing you to simulate high loads and monitor real-time performance metrics effectively.

Table of Contents

1. Load Testing with k6

• Setting up k6
• Creating a load test script

2. Prometheus and Grafana Monitoring Setup

• Configuring Prometheus
• Integrating Grafana for visualizations

3. Optional: Long-Term Storage with InfluxDB

4. Complete docker-compose.yml Setup

Step 1: Load Testing with k6 k6 is a popular tool for running load tests to simulate real-world usage and assess your application’s performance under pressure.

1.1 Setting Up k6 in Docker Compose Add a k6 service to your docker-compose.yml file:

YAML

services:
  k6:
    image: grafana/k6
    volumes:
      - type: bind
        source: ./load-testing
        target: /load-testing
    entrypoint: ["k6", "run", "/load-testing/load-test.js"]

Here:

• volumes: Mounts the load-testing folder from your host to /load-testing in the container, so k6 can access the load-test.js script.
• entrypoint: Defines the default command for k6 to run the test script.

1.2 Writing a k6 Load Test Script Create a load-test.js file inside a load-testing folder on your host machine:

JavaScript

import http from 'k6/http';
import { sleep } from 'k6';
export let options = {
  stages: [
    { duration: '1m', target: 200 },
    { duration: '3m', target: 200 },
    { duration: '1m', target: 0 },
  ],
};
export default function () {
  http.get('http://host.docker.internal:3000'); // Adjust if needed
  sleep(1);
}

This script simulates a ramp-up to 200 requests per second, maintains that load for 3 minutes, and then ramps down.

Step 2: Prometheus and Grafana Monitoring Setup Prometheus and Grafana allow you to monitor and visualize metrics like response times, request rates, and system load in real-time.

2.1 Configuring Prometheus Create a prometheus.yml file for Prometheus:

YAML

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'kafka'
    static_configs:
      - targets: ['kafka:9092']
- job_name: 'express-app'
  static_configs:
    - targets: ['host.docker.internal:3000'] # No "http://"

In this configuration:

• scrape_interval: Determines how often Prometheus collects metrics.
• scrape_configs: Specifies targets, such as Kafka on 9092 and your application on 3000.

Add the Prometheus service to your docker-compose.yml file:

YAML

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

This setup maps your configuration file to Prometheus and exposes it on port 9090 for access.

2.2 Configuring Grafana Add Grafana to your docker-compose.yml file:

YAML

services:
  grafana:
    image: grafana/grafana
    ports:
      - "3001:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - prometheus

To set up Grafana:

• Open Grafana at http://localhost:3001.
• Add Prometheus as a data source:
• Navigate to Configuration > Data Sources.
• Select Prometheus.
• Set the URL to http://prometheus:9090.
• Save the data source.
• Import or create dashboards to visualize metrics, such as request rates and response times.

Step 3: Optional Long-Term Storage with InfluxDB InfluxDB is a time-series database that stores metrics for historical analysis. Add InfluxDB to your docker-compose.yml file if you need extended metric retention:

YAML

services:
  influxdb:
    image: influxdb:latest
    ports:
      - "8086:8086"

Integrating InfluxDB with Grafana

• In Grafana, go to Configuration > Data Sources and add InfluxDB.
• Set the URL to http://influxdb:8086.

Complete docker-compose.yml Setup Here’s the full docker-compose.yml file with all services configured:

YAML

version: '3.8'
services:
  your-app-service:
    image: your-app-image
    ports:
      - "3000:3000"
  kafka:
    image: bitnami/kafka:latest
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
  zookeeper:
    image: bitnami/zookeeper:latest
    ports:
      - "2181:2181"
  k6:
    image: grafana/k6
    volumes:
      - type: bind
        source: ./load-testing
        target: /load-testing
    entrypoint: ["k6", "run", "/load-testing/load-test.js"]
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana
    ports:
      - "3001:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - prometheus
  influxdb:
    image: influxdb:latest
    ports:
      - "8086:8086"

Running the Setup

• Start the Services:
• Bash
• docker-compose up -d
• Access Grafana at http://localhost:3001 and add Prometheus and InfluxDB as data sources.
• Run the k6 Load Test: If k6 is configured to run automatically, it will begin the load test upon starting.

Monitoring Your Application

• Prometheus: Access Prometheus at http://localhost:9090 to view raw metrics.
• Grafana: Use Grafana dashboards for real-time insights and to visualize performance metrics such as HTTP request rates, response times, and Kafka throughput.

With this setup, you have a complete environment for load testing, monitoring, and analyzing your application’s performance under load.

Next we will see how to create producers and topic consumers and monitor some transactions from a real api also we will be saving this data to an sql database