Compare commits

..
10 Commits
Author SHA1 Message Date
Your Name 1eaa67fc62 more alloy annoying 2025-03-19 11:19:53 +00:00
Your Name 32b4d8ba43 remove node exporter and add alloy for metrics 2025-03-19 10:06:48 +00:00
Your Name 0046ff0f07 alloy works 2025-03-18 16:04:23 +00:00
Your Name 753fb48b20 add alloy 2025-03-18 14:06:29 +00:00
Your Name a0f337e08d add visuals 2025-03-18 11:26:53 +00:00
Bartal Læarsson 27ee758ee1 fix container surveillance 2025-03-18 09:33:02 +00:00
Your Name 581f35cad3 add alerting 2025-03-17 16:06:35 +00:00
Bartal Læarsson dea9cdc0fe fix container surveillance 2025-03-17 14:27:38 +00:00
Bartal Læarsson d0ca70c546 config readme 2025-03-11 14:27:52 +00:00
Bartal Læarsson a89c3ad3cc config readme 2025-03-11 14:26:35 +00:00
10 changed files with 330 additions and 138 deletions
+119 -63
View File
@@ -1,95 +1,151 @@
# Monitoring Stack with Grafana, Prometheus, Loki & Alloy # LGP Monitoring Stack
This repository contains a complete monitoring stack for visualizing metrics and logs from containers and host systems using Docker Compose. A production-grade monitoring solution for Linux servers using Prometheus, Grafana, Loki, and related components.
## Components ## Architecture
- **Grafana**: Visualization and dashboarding (port 3000) This stack implements a comprehensive monitoring solution with the following components:
- **Prometheus**: Metrics collection and storage (port 9090)
- **Loki**: Log aggregation system (port 3100) - **Prometheus**: Time series database and metrics collector with alerting capabilities
- **Alloy**: Grafana complementary observability system (port 12345) - **Grafana**: Visualization platform for metrics, logs, and alerting
- **Node Exporter**: Host metrics exporter (port 9100) - **Loki**: Log aggregation system optimized for Prometheus and Grafana
- **cAdvisor**: Container metrics exporter (port 8080) - **Alloy**: Grafana's observability correlation engine
- **cAdvisor**: Container resource usage and performance monitoring
- **Node Exporter**: Physical host metrics collector (CPU, memory, disk, network)
## Component Details
| Component | Purpose | Default Port | Data Retention |
|-----------|---------|--------------|----------------|
| Prometheus | Metrics collection and storage | 3004 | 15d / 10GB |
| Grafana | Visualization and dashboarding | 3003 | Persistent |
| Loki | Log aggregation | 9020 | Configured via volume |
| Alloy | Correlation engine | 9030 | Persistent |
| cAdvisor | Container metrics | 9050 | N/A (scrape only) |
| Node Exporter | Host metrics | 9060 | N/A (scrape only) |
## Quick Start ## Quick Start
```bash ```shell
# Optional: Customize configuration
# edit config/prometheus.yml, docker-compose.yml, etc.
# Start the monitoring stack # Start the monitoring stack
docker-compose up -d docker-compose up -d
# View logs # Verify all components are running
docker-compose logs -f docker-compose ps
# Stop the stack # Stop the stack
docker-compose down docker-compose down
``` ```
## Services Access ## Detailed Configuration
- **Grafana**: http://localhost:9010 (user: admin, password: password) ### Prometheus
- **Prometheus**: http://localhost:9040
- **Loki**: http://localhost:9020
- **cAdvisor**: http://localhost:9050
- **Node Exporter**: http://localhost:9060
- **Alloy**: http://localhost:9030
## Directory Structure The Prometheus configuration (`config/prometheus.yml`) includes:
``` - 60s scrape interval for most targets
. - Sample limits to prevent overload (1000 global, 50000 per job)
├── config/ - Alert rules in `alerts.yml`
│ ├── alloy.conf # Alloy configuration - Job configurations for all monitoring components
│ ├── loki.yml # Loki configuration
│ ├── prometheus.yml # Prometheus configuration
│ ├── prometheus-alerts.yml # Prometheus alerting rules
│ └── provisioning/
│ ├── dashboards/ # Grafana dashboard provisioning
│ │ ├── dashboard-provider.yml # Dashboard provider config
│ │ └── node-exporter-full.json # Node Exporter dashboard
│ └── datasources/ # Grafana datasource provisioning
│ └── grafana-datasources.yml # Datasource config
├── docker-compose.yml # Docker Compose configuration
└── README.md # This file
```
## Preconfigured Features To customize:
- Adjust `scrape_interval` for different reporting frequencies
- Modify `sample_limit` based on your server capacity
- Configure alerting rules in `alerts.yml`
- **Dashboards**: Node Exporter Full dashboard pre-installed ### Multi-Server Monitoring
- **Datasources**: Prometheus and Loki automatically configured
- **Alerts**: Basic host and container alerts configured in Prometheus
- **Metrics**: Host and container metrics automatically collected
- **Persistent Storage**: All data stored in Docker volumes
## Configuration For monitoring multiple servers:
### Modifying Prometheus Targets 1. **Static Configuration**:
```yaml
# In prometheus.yml
- job_name: 'remote-node'
static_configs:
- targets: ['server1:9100', 'server2:9100', 'server3:9100']
labels:
env: production
```
Edit `config/prometheus.yml` to add additional scrape targets. 2. **Service Discovery**:
- Consider using DNS or file-based discovery for dynamic environments
### Adding Grafana Dashboards ### Resource Allocation
Place JSON dashboard files in `config/provisioning/dashboards/` and restart the stack. The stack is configured with reasonable defaults:
- Prometheus: 1-2GB memory reservation
- Individual exporters: minimal overhead (~50-100MB each)
### Alerting Rules For high-traffic environments, adjust the `deploy.resources` section in `docker-compose.yml`.
Edit `config/prometheus-alerts.yml` to modify alerting rules. ## Security Considerations
This setup includes basic security measures:
1. Container isolation via Docker networks
2. Resource limits to prevent DoS
3. Basic authentication for Grafana
**Production Hardening Steps:**
- Enable TLS for all components
- Use secrets management for credentials
- Implement network policies/firewall rules
- Set up proper authentication for Prometheus
## Alerting Setup
1. **Define alerts** in `config/prometheus-alerts.yml`:
```yaml
groups:
- name: host_alerts
rules:
- alert: HighCPULoad
expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU load on {{ $labels.instance }}"
description: "CPU load is > 80% for 5 minutes"
```
2. **Configure Alertmanager** to handle notifications (email, Slack, PagerDuty, etc.)
## Performance Tuning
- **Prometheus retention**: Adjust `--storage.tsdb.retention.time` and `--storage.tsdb.retention.size`
- **Scrape interval**: Balance between granularity and performance
- **Query optimization**: Use recording rules for complex queries
- **Resource allocation**: Adjust container memory limits based on load
## Troubleshooting ## Troubleshooting
**Dashboards not showing up:** Common issues and solutions:
- Check Grafana logs: `docker-compose logs grafana`
- Verify directory structure matches documentation
- Ensure dashboard JSON files are valid
**Service won't start:** 1. **Prometheus not starting**: Check volume permissions and storage capacity
- Check for port conflicts 2. **Missing metrics**: Verify scrape configurations and target accessibility
- Verify all required files exist in the correct locations 3. **High memory usage**: Adjust retention settings and sample limits
- Check individual service logs: `docker-compose logs [service_name]` 4. **Dashboard errors**: Check data source configuration in Grafana
## Security Notes Debug commands:
```shell
# Check Prometheus targets
curl -s http://localhost:3004/api/v1/targets | jq .
This setup uses default credentials and is intended for local development or testing. For production use: # Verify scrape metrics
- Change default Grafana password curl -s http://localhost:9060/metrics | grep node_
- Set up proper authentication for all services
- Use secrets management for credentials # Check Loki logs
- Configure TLS for all exposed endpoints docker-compose logs loki
```
## Dashboard Examples
The stack comes with pre-configured dashboards for:
- Node Exporter metrics (CPU, memory, disk, network)
- Container performance monitoring
- System overview metrics
- Log visualization and analysis
+22
View File
@@ -0,0 +1,22 @@
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'job']
group_wait: 30s
group_interval: 5m
repeat_interval: 3h
receiver: 'teams'
receivers:
- name: 'teams'
webhook_configs:
- url: 'http://webhook-adapter:8080/alert'
send_resolved: true
inhibit_rules:
- source_matchers:
- severity = 'critical'
target_matchers:
- severity = 'warning'
equal: ['alertname']
-4
View File
@@ -1,4 +0,0 @@
logging {
level = "info"
format = "logfmt"
}
+38
View File
@@ -0,0 +1,38 @@
prometheus.exporter.unix "host_metrics" {
// Enable specific collectors that would provide the missing metrics
enable_collectors = [
"cpu", "pressure", "netdev", "netstat", "diskstats",
"meminfo", "loadavg", "softnet", "stat", "thermal_zone",
]
// Set paths for accessing host metrics when running in a container
procfs_path = "/host/proc"
sysfs_path = "/host/sys"
rootfs_path = "/rootfs"
// Configure filesystem collector
filesystem {
mount_points_exclude = "^/(sys|proc|dev|host|etc)($$|/)"
}
// Configure netdev collector
netdev {
// Include all network devices for network traffic metrics
device_exclude = ""
}
}
// Make sure we scrape the host metrics
prometheus.scrape "host_metrics" {
targets = prometheus.exporter.unix.host_metrics.targets
forward_to = [prometheus.remote_write.prometheus.receiver]
scrape_interval = "15s"
scrape_timeout = "10s"
job_name = "node"
}
prometheus.remote_write "prometheus" {
endpoint {
url = "http://prometheus:9090/api/v1/write"
}
}
+14 -4
View File
@@ -40,14 +40,23 @@ groups:
description: "Container {{ $labels.name }} CPU usage is > 80% for 5 minutes" description: "Container {{ $labels.name }} CPU usage is > 80% for 5 minutes"
- alert: ContainerHighMemory - alert: ContainerHighMemory
expr: (container_memory_usage_bytes{name!=""} / container_spec_memory_limit_bytes{name!=""} * 100) > 80 # First filter out containers with memory limits, then calculate percentage
for: 5m expr: |
(
container_memory_usage_bytes{name!=""}
/
container_spec_memory_limit_bytes{name!=""}
* 100 > 90
)
and
container_spec_memory_limit_bytes{name!=""} > 0
for: 10m
labels: labels:
severity: warning severity: warning
annotations: annotations:
summary: "High memory usage in container {{ $labels.name }}" summary: "High memory usage in container {{ $labels.name }}"
description: "Container {{ $labels.name }} memory usage is > 80% for 5 minutes" description: "Container {{ $labels.name }} ({{ $labels.image }}) memory usage is {{ printf \"%.1f\" $value }}% of its limit for 10 minutes"
- alert: ContainerDown - alert: ContainerDown
expr: absent(container_last_seen{name!=""}) expr: absent(container_last_seen{name!=""})
for: 1m for: 1m
@@ -65,3 +74,4 @@ groups:
annotations: annotations:
summary: "Container {{ $labels.name }} restarting frequently" summary: "Container {{ $labels.name }} restarting frequently"
description: "Container {{ $labels.name }} has restarted more than 2 times in the last 15 minutes" description: "Container {{ $labels.name }} has restarted more than 2 times in the last 15 minutes"
+16 -18
View File
@@ -2,7 +2,7 @@ global:
scrape_interval: 60s scrape_interval: 60s
evaluation_interval: 60s evaluation_interval: 60s
scrape_timeout: 10s scrape_timeout: 10s
sample_limit: 1000 # Limit samples per scrape to prevent overload sample_limit: 1000
# Rule files # Rule files
rule_files: rule_files:
@@ -14,26 +14,12 @@ alerting:
- targets: - targets:
scrape_configs: scrape_configs:
# Prometheus itself # Prometheus
- job_name: 'prometheus' - job_name: 'prometheus'
sample_limit: 50000 sample_limit: 50000
static_configs: static_configs:
- targets: ['prometheus:9090'] - targets: ['prometheus:9090']
# Container metrics via cAdvisor
- job_name: 'cadvisor'
sample_limit: 50000
static_configs:
- targets: ['cadvisor:8080']
metrics_path: /metrics
scheme: http
# Host metrics via node-exporter
- job_name: 'node-exporter'
sample_limit: 50000
static_configs:
- targets: ['node-exporter:9100']
# Loki metrics # Loki metrics
- job_name: 'loki' - job_name: 'loki'
sample_limit: 50000 sample_limit: 50000
@@ -46,8 +32,20 @@ scrape_configs:
static_configs: static_configs:
- targets: ['grafana:3000'] - targets: ['grafana:3000']
# Alloy metrics # Node metrics from Alloy - with custom instance name
- job_name: 'alloy' - job_name: 'node'
sample_limit: 50000 sample_limit: 50000
static_configs: static_configs:
- targets: ['alloy:12345'] - targets: ['alloy:12345']
labels:
instance: 'main-server' # Use any name you want
metrics_path: /metrics
# Remote node for the second server
- job_name: 'node-app-server01'
sample_limit: 50000
static_configs:
- targets: ['10.249.84.41:12345']
labels:
instance: 'app-server-01' # Use any name you want
metrics_path: /metrics
@@ -0,0 +1,21 @@
apiVersion: 1
contactPoints:
- name: ms-teams
receivers:
- uid: teams-notifications
type: teams
settings:
url: https://prod-152.westeurope.logic.azure.com:443/workflows/6040474cfb314615aabd47694804d7e7/triggers/manual/paths/invoke?api-version=2016-06-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=ehfajgttd-2NC7rdEL5z9rylE4IiyVpgPjiOfDqF_z0
title: "{{ if eq .Status \"firing\" }}🔥{{ else }}✅{{ end }} [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}"
message: |-
{{ if ne .Status "firing" }}Resolved{{ else }}{{ end }}: {{ .CommonAnnotations.summary }}
{{ .CommonAnnotations.description }}
{{- if gt (len .Alerts.Firing) 0 }}
Alerts:
{{- range .Alerts.Firing }}
- {{ .Labels.instance }} {{ .Labels.job }}: {{ .Annotations.summary }}
{{- end }}
{{- end }}
@@ -0,0 +1,12 @@
apiVersion: 1
policies:
- receiver: ms-teams
group_by: ['alertname', 'job']
repeat_interval: 3h
group_interval: 5m
routes:
- receiver: ms-teams
matchers:
- severity =~ "critical|warning|info"
continue: true
+51
View File
@@ -0,0 +1,51 @@
{
"type": "message",
"attachments": [
{
"contentType": "application/vnd.microsoft.card.adaptive",
"content": {
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"type": "AdaptiveCard",
"version": "1.3",
"body": [
{
"type": "TextBlock",
"text": "{{ if eq .Status "firing" }}🔥{{ else }}✅{{ end }} {{ .Status | title }}: {{ .CommonLabels.alertname }}",
"weight": "Bolder",
"size": "Large"
},
{
"type": "TextBlock",
"text": "{{ .CommonAnnotations.summary }}",
"wrap": true
},
{
"type": "FactSet",
"facts": [
{
"title": "Status",
"value": "{{ .Status | title }}"
},
{
"title": "Severity",
"value": "{{ .CommonLabels.severity | title }}"
}
]
},
{
"type": "TextBlock",
"text": "{{ .CommonAnnotations.description }}",
"wrap": true
}
],
"actions": [
{
"type": "Action.OpenUrl",
"title": "View in Alertmanager",
"url": "{{ .ExternalURL }}"
}
]
}
}
]
}
+37 -49
View File
@@ -1,10 +1,9 @@
services: services:
# Grafana for visualization grafana:
grafanao:
image: grafana/grafana:latest image: grafana/grafana:latest
container_name: grafana container_name: grafana
healthcheck: healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:3003"] test: ["CMD", "wget", "-q", "--spider", "http://localhost:3000"]
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 3 retries: 3
@@ -14,14 +13,16 @@ services:
- GF_SECURITY_ADMIN_USER=admin - GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=password # Change this in production - GF_SECURITY_ADMIN_PASSWORD=password # Change this in production
- GF_USERS_ALLOW_SIGN_UP=false - GF_USERS_ALLOW_SIGN_UP=false
- GF_UNIFIED_ALERTING_ENABLED=true
- GF_ALERTING_ENABLED=false
- GF_UNIFIED_ALERTING_PROMETHEUS_ALERTS_ENABLED=true
volumes: volumes:
- grafana-data:/var/lib/grafana - grafana-data:/var/lib/grafana
- ./config/provisioning/dashboards:/etc/grafana/provisioning/dashboards - ./config/provisioning/dashboards:/etc/grafana/provisioning/dashboards
- ./config/provisioning/datasources:/etc/grafana/provisioning/datasources - ./config/provisioning/datasources:/etc/grafana/provisioning/datasources
# Add this new mount to store the actual dashboard JSON - ./config/provisioning/alerting:/etc/grafana/provisioning/alerting
- ./config/provisioning/dashboards:/var/lib/grafana/dashboards - ./config/dashboards:/var/lib/grafana/dashboards
depends_on: depends_on:
- loki
- prometheus - prometheus
restart: unless-stopped restart: unless-stopped
networks: networks:
@@ -37,20 +38,15 @@ services:
networks: networks:
- monitoring - monitoring
alloy: # AlertManager for handling alerts
image: grafana/alloy:latest alertmanager:
container_name: grafana-alloy image: quay.io/prometheus/alertmanager:latest
container_name: alertmanager
ports: ports:
- "9030:12345" - "9093:9093"
volumes: volumes:
- ./config/alloy.conf:/etc/alloy/config.alloy - ./config/alertmanager.yml:/etc/alertmanager/config.yml
- alloy-data:/var/lib/alloy/data restart: unless-stopped
command:
- run
- --server.http.listen-addr=0.0.0.0:12345
- --storage.path=/var/lib/alloy/data
- /etc/alloy/config.alloy
restart: unless-stopped
networks: networks:
- monitoring - monitoring
@@ -72,6 +68,7 @@ services:
- '--web.console.libraries=/etc/prometheus/console_libraries' - '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles' - '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle' - '--web.enable-lifecycle'
- "--web.enable-remote-write-receiver"
restart: unless-stopped restart: unless-stopped
networks: networks:
- monitoring - monitoring
@@ -87,48 +84,39 @@ services:
max_attempts: 3 max_attempts: 3
window: 120s window: 120s
# cAdvisor for container metrics alloy:
cadvisor: image: grafana/alloy:latest
image: gcr.io/cadvisor/cadvisor:v0.49.1 container_name: alloy
container_name: cadvisor hostname: localhost # Set a proper hostname
volumes: volumes:
- /:/rootfs:ro - ./config/config.alloy:/etc/alloy/config.alloy
- /var/run:/var/run:ro - /proc:/host/proc:ro # CRITICAL - must be mounted at /host/proc
- /sys:/sys:ro - /sys:/host/sys:ro # CRITICAL - must be mounted at /host/sys
- /var/lib/docker/:/var/lib/docker:ro - /:/rootfs:ro # CRITICAL - must be mounted at /rootfs
- /dev/disk/:/dev/disk:ro
ports:
- "9050:8080"
restart: unless-stopped
networks:
- monitoring
privileged: true
devices:
- /dev/kmsg:/dev/kmsg
# Node Exporter for host metrics
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command: command:
- '--path.procfs=/host/proc' - run
- '--path.sysfs=/host/sys' - --server.http.listen-addr=0.0.0.0:12345
- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' - --storage.path=/var/lib/alloy/data
- /etc/alloy/config.alloy
ports: ports:
- "9060:9100" - "12345:12345"
restart: unless-stopped privileged: true # CRITICAL for accessing system metrics
pid: "host" # CRITICAL for accessing process metrics
environment:
- HOSTNAME=localhost # Explicitly set the hostname for metrics
networks: networks:
- monitoring - monitoring
volumes: volumes:
grafana-data: grafana-data:
prometheus-data: prometheus-data:
alloy-data: alloy-data:
driver: local driver: local
loki-data:
driver: local
alertmanager-data:
driver: local
networks: networks:
monitoring: monitoring: