Compare commits
10
Commits
cf3968ac2c
...
1eaa67fc62
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1eaa67fc62 | ||
|
|
32b4d8ba43 | ||
|
|
0046ff0f07 | ||
|
|
753fb48b20 | ||
|
|
a0f337e08d | ||
|
|
27ee758ee1 | ||
|
|
581f35cad3 | ||
|
|
dea9cdc0fe | ||
|
|
d0ca70c546 | ||
|
|
a89c3ad3cc |
@@ -1,95 +1,151 @@
|
|||||||
# Monitoring Stack with Grafana, Prometheus, Loki & Alloy
|
# LGP Monitoring Stack
|
||||||
|
|
||||||
This repository contains a complete monitoring stack for visualizing metrics and logs from containers and host systems using Docker Compose.
|
A production-grade monitoring solution for Linux servers using Prometheus, Grafana, Loki, and related components.
|
||||||
|
|
||||||
## Components
|
## Architecture
|
||||||
|
|
||||||
- **Grafana**: Visualization and dashboarding (port 3000)
|
This stack implements a comprehensive monitoring solution with the following components:
|
||||||
- **Prometheus**: Metrics collection and storage (port 9090)
|
|
||||||
- **Loki**: Log aggregation system (port 3100)
|
- **Prometheus**: Time series database and metrics collector with alerting capabilities
|
||||||
- **Alloy**: Grafana complementary observability system (port 12345)
|
- **Grafana**: Visualization platform for metrics, logs, and alerting
|
||||||
- **Node Exporter**: Host metrics exporter (port 9100)
|
- **Loki**: Log aggregation system optimized for Prometheus and Grafana
|
||||||
- **cAdvisor**: Container metrics exporter (port 8080)
|
- **Alloy**: Grafana's observability correlation engine
|
||||||
|
- **cAdvisor**: Container resource usage and performance monitoring
|
||||||
|
- **Node Exporter**: Physical host metrics collector (CPU, memory, disk, network)
|
||||||
|
|
||||||
|
## Component Details
|
||||||
|
|
||||||
|
| Component | Purpose | Default Port | Data Retention |
|
||||||
|
|-----------|---------|--------------|----------------|
|
||||||
|
| Prometheus | Metrics collection and storage | 3004 | 15d / 10GB |
|
||||||
|
| Grafana | Visualization and dashboarding | 3003 | Persistent |
|
||||||
|
| Loki | Log aggregation | 9020 | Configured via volume |
|
||||||
|
| Alloy | Correlation engine | 9030 | Persistent |
|
||||||
|
| cAdvisor | Container metrics | 9050 | N/A (scrape only) |
|
||||||
|
| Node Exporter | Host metrics | 9060 | N/A (scrape only) |
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
```bash
|
```shell
|
||||||
|
# Optional: Customize configuration
|
||||||
|
# edit config/prometheus.yml, docker-compose.yml, etc.
|
||||||
|
|
||||||
# Start the monitoring stack
|
# Start the monitoring stack
|
||||||
docker-compose up -d
|
docker-compose up -d
|
||||||
|
|
||||||
# View logs
|
# Verify all components are running
|
||||||
docker-compose logs -f
|
docker-compose ps
|
||||||
|
|
||||||
# Stop the stack
|
# Stop the stack
|
||||||
docker-compose down
|
docker-compose down
|
||||||
```
|
```
|
||||||
|
|
||||||
## Services Access
|
## Detailed Configuration
|
||||||
|
|
||||||
- **Grafana**: http://localhost:9010 (user: admin, password: password)
|
### Prometheus
|
||||||
- **Prometheus**: http://localhost:9040
|
|
||||||
- **Loki**: http://localhost:9020
|
|
||||||
- **cAdvisor**: http://localhost:9050
|
|
||||||
- **Node Exporter**: http://localhost:9060
|
|
||||||
- **Alloy**: http://localhost:9030
|
|
||||||
|
|
||||||
## Directory Structure
|
The Prometheus configuration (`config/prometheus.yml`) includes:
|
||||||
|
|
||||||
```
|
- 60s scrape interval for most targets
|
||||||
.
|
- Sample limits to prevent overload (1000 global, 50000 per job)
|
||||||
├── config/
|
- Alert rules in `alerts.yml`
|
||||||
│ ├── alloy.conf # Alloy configuration
|
- Job configurations for all monitoring components
|
||||||
│ ├── loki.yml # Loki configuration
|
|
||||||
│ ├── prometheus.yml # Prometheus configuration
|
|
||||||
│ ├── prometheus-alerts.yml # Prometheus alerting rules
|
|
||||||
│ └── provisioning/
|
|
||||||
│ ├── dashboards/ # Grafana dashboard provisioning
|
|
||||||
│ │ ├── dashboard-provider.yml # Dashboard provider config
|
|
||||||
│ │ └── node-exporter-full.json # Node Exporter dashboard
|
|
||||||
│ └── datasources/ # Grafana datasource provisioning
|
|
||||||
│ └── grafana-datasources.yml # Datasource config
|
|
||||||
├── docker-compose.yml # Docker Compose configuration
|
|
||||||
└── README.md # This file
|
|
||||||
```
|
|
||||||
|
|
||||||
## Preconfigured Features
|
To customize:
|
||||||
|
- Adjust `scrape_interval` for different reporting frequencies
|
||||||
|
- Modify `sample_limit` based on your server capacity
|
||||||
|
- Configure alerting rules in `alerts.yml`
|
||||||
|
|
||||||
- **Dashboards**: Node Exporter Full dashboard pre-installed
|
### Multi-Server Monitoring
|
||||||
- **Datasources**: Prometheus and Loki automatically configured
|
|
||||||
- **Alerts**: Basic host and container alerts configured in Prometheus
|
|
||||||
- **Metrics**: Host and container metrics automatically collected
|
|
||||||
- **Persistent Storage**: All data stored in Docker volumes
|
|
||||||
|
|
||||||
## Configuration
|
For monitoring multiple servers:
|
||||||
|
|
||||||
### Modifying Prometheus Targets
|
1. **Static Configuration**:
|
||||||
|
```yaml
|
||||||
|
# In prometheus.yml
|
||||||
|
- job_name: 'remote-node'
|
||||||
|
static_configs:
|
||||||
|
- targets: ['server1:9100', 'server2:9100', 'server3:9100']
|
||||||
|
labels:
|
||||||
|
env: production
|
||||||
|
```
|
||||||
|
|
||||||
Edit `config/prometheus.yml` to add additional scrape targets.
|
2. **Service Discovery**:
|
||||||
|
- Consider using DNS or file-based discovery for dynamic environments
|
||||||
|
|
||||||
### Adding Grafana Dashboards
|
### Resource Allocation
|
||||||
|
|
||||||
Place JSON dashboard files in `config/provisioning/dashboards/` and restart the stack.
|
The stack is configured with reasonable defaults:
|
||||||
|
- Prometheus: 1-2GB memory reservation
|
||||||
|
- Individual exporters: minimal overhead (~50-100MB each)
|
||||||
|
|
||||||
### Alerting Rules
|
For high-traffic environments, adjust the `deploy.resources` section in `docker-compose.yml`.
|
||||||
|
|
||||||
Edit `config/prometheus-alerts.yml` to modify alerting rules.
|
## Security Considerations
|
||||||
|
|
||||||
|
This setup includes basic security measures:
|
||||||
|
|
||||||
|
1. Container isolation via Docker networks
|
||||||
|
2. Resource limits to prevent DoS
|
||||||
|
3. Basic authentication for Grafana
|
||||||
|
|
||||||
|
**Production Hardening Steps:**
|
||||||
|
- Enable TLS for all components
|
||||||
|
- Use secrets management for credentials
|
||||||
|
- Implement network policies/firewall rules
|
||||||
|
- Set up proper authentication for Prometheus
|
||||||
|
|
||||||
|
## Alerting Setup
|
||||||
|
|
||||||
|
1. **Define alerts** in `config/prometheus-alerts.yml`:
|
||||||
|
```yaml
|
||||||
|
groups:
|
||||||
|
- name: host_alerts
|
||||||
|
rules:
|
||||||
|
- alert: HighCPULoad
|
||||||
|
expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
|
||||||
|
for: 5m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "High CPU load on {{ $labels.instance }}"
|
||||||
|
description: "CPU load is > 80% for 5 minutes"
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Configure Alertmanager** to handle notifications (email, Slack, PagerDuty, etc.)
|
||||||
|
|
||||||
|
## Performance Tuning
|
||||||
|
|
||||||
|
- **Prometheus retention**: Adjust `--storage.tsdb.retention.time` and `--storage.tsdb.retention.size`
|
||||||
|
- **Scrape interval**: Balance between granularity and performance
|
||||||
|
- **Query optimization**: Use recording rules for complex queries
|
||||||
|
- **Resource allocation**: Adjust container memory limits based on load
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
**Dashboards not showing up:**
|
Common issues and solutions:
|
||||||
- Check Grafana logs: `docker-compose logs grafana`
|
|
||||||
- Verify directory structure matches documentation
|
|
||||||
- Ensure dashboard JSON files are valid
|
|
||||||
|
|
||||||
**Service won't start:**
|
1. **Prometheus not starting**: Check volume permissions and storage capacity
|
||||||
- Check for port conflicts
|
2. **Missing metrics**: Verify scrape configurations and target accessibility
|
||||||
- Verify all required files exist in the correct locations
|
3. **High memory usage**: Adjust retention settings and sample limits
|
||||||
- Check individual service logs: `docker-compose logs [service_name]`
|
4. **Dashboard errors**: Check data source configuration in Grafana
|
||||||
|
|
||||||
## Security Notes
|
Debug commands:
|
||||||
|
```shell
|
||||||
|
# Check Prometheus targets
|
||||||
|
curl -s http://localhost:3004/api/v1/targets | jq .
|
||||||
|
|
||||||
This setup uses default credentials and is intended for local development or testing. For production use:
|
# Verify scrape metrics
|
||||||
- Change default Grafana password
|
curl -s http://localhost:9060/metrics | grep node_
|
||||||
- Set up proper authentication for all services
|
|
||||||
- Use secrets management for credentials
|
# Check Loki logs
|
||||||
- Configure TLS for all exposed endpoints
|
docker-compose logs loki
|
||||||
|
```
|
||||||
|
|
||||||
|
## Dashboard Examples
|
||||||
|
|
||||||
|
The stack comes with pre-configured dashboards for:
|
||||||
|
- Node Exporter metrics (CPU, memory, disk, network)
|
||||||
|
- Container performance monitoring
|
||||||
|
- System overview metrics
|
||||||
|
- Log visualization and analysis
|
||||||
|
|||||||
@@ -0,0 +1,22 @@
|
|||||||
|
global:
|
||||||
|
resolve_timeout: 5m
|
||||||
|
|
||||||
|
route:
|
||||||
|
group_by: ['alertname', 'job']
|
||||||
|
group_wait: 30s
|
||||||
|
group_interval: 5m
|
||||||
|
repeat_interval: 3h
|
||||||
|
receiver: 'teams'
|
||||||
|
|
||||||
|
receivers:
|
||||||
|
- name: 'teams'
|
||||||
|
webhook_configs:
|
||||||
|
- url: 'http://webhook-adapter:8080/alert'
|
||||||
|
send_resolved: true
|
||||||
|
|
||||||
|
inhibit_rules:
|
||||||
|
- source_matchers:
|
||||||
|
- severity = 'critical'
|
||||||
|
target_matchers:
|
||||||
|
- severity = 'warning'
|
||||||
|
equal: ['alertname']
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
logging {
|
|
||||||
level = "info"
|
|
||||||
format = "logfmt"
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
prometheus.exporter.unix "host_metrics" {
|
||||||
|
// Enable specific collectors that would provide the missing metrics
|
||||||
|
enable_collectors = [
|
||||||
|
"cpu", "pressure", "netdev", "netstat", "diskstats",
|
||||||
|
"meminfo", "loadavg", "softnet", "stat", "thermal_zone",
|
||||||
|
]
|
||||||
|
|
||||||
|
// Set paths for accessing host metrics when running in a container
|
||||||
|
procfs_path = "/host/proc"
|
||||||
|
sysfs_path = "/host/sys"
|
||||||
|
rootfs_path = "/rootfs"
|
||||||
|
|
||||||
|
// Configure filesystem collector
|
||||||
|
filesystem {
|
||||||
|
mount_points_exclude = "^/(sys|proc|dev|host|etc)($$|/)"
|
||||||
|
}
|
||||||
|
|
||||||
|
// Configure netdev collector
|
||||||
|
netdev {
|
||||||
|
// Include all network devices for network traffic metrics
|
||||||
|
device_exclude = ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Make sure we scrape the host metrics
|
||||||
|
prometheus.scrape "host_metrics" {
|
||||||
|
targets = prometheus.exporter.unix.host_metrics.targets
|
||||||
|
forward_to = [prometheus.remote_write.prometheus.receiver]
|
||||||
|
scrape_interval = "15s"
|
||||||
|
scrape_timeout = "10s"
|
||||||
|
job_name = "node"
|
||||||
|
}
|
||||||
|
|
||||||
|
prometheus.remote_write "prometheus" {
|
||||||
|
endpoint {
|
||||||
|
url = "http://prometheus:9090/api/v1/write"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -40,14 +40,23 @@ groups:
|
|||||||
description: "Container {{ $labels.name }} CPU usage is > 80% for 5 minutes"
|
description: "Container {{ $labels.name }} CPU usage is > 80% for 5 minutes"
|
||||||
|
|
||||||
- alert: ContainerHighMemory
|
- alert: ContainerHighMemory
|
||||||
expr: (container_memory_usage_bytes{name!=""} / container_spec_memory_limit_bytes{name!=""} * 100) > 80
|
# First filter out containers with memory limits, then calculate percentage
|
||||||
for: 5m
|
expr: |
|
||||||
|
(
|
||||||
|
container_memory_usage_bytes{name!=""}
|
||||||
|
/
|
||||||
|
container_spec_memory_limit_bytes{name!=""}
|
||||||
|
* 100 > 90
|
||||||
|
)
|
||||||
|
and
|
||||||
|
container_spec_memory_limit_bytes{name!=""} > 0
|
||||||
|
for: 10m
|
||||||
labels:
|
labels:
|
||||||
severity: warning
|
severity: warning
|
||||||
annotations:
|
annotations:
|
||||||
summary: "High memory usage in container {{ $labels.name }}"
|
summary: "High memory usage in container {{ $labels.name }}"
|
||||||
description: "Container {{ $labels.name }} memory usage is > 80% for 5 minutes"
|
description: "Container {{ $labels.name }} ({{ $labels.image }}) memory usage is {{ printf \"%.1f\" $value }}% of its limit for 10 minutes"
|
||||||
|
|
||||||
- alert: ContainerDown
|
- alert: ContainerDown
|
||||||
expr: absent(container_last_seen{name!=""})
|
expr: absent(container_last_seen{name!=""})
|
||||||
for: 1m
|
for: 1m
|
||||||
@@ -65,3 +74,4 @@ groups:
|
|||||||
annotations:
|
annotations:
|
||||||
summary: "Container {{ $labels.name }} restarting frequently"
|
summary: "Container {{ $labels.name }} restarting frequently"
|
||||||
description: "Container {{ $labels.name }} has restarted more than 2 times in the last 15 minutes"
|
description: "Container {{ $labels.name }} has restarted more than 2 times in the last 15 minutes"
|
||||||
|
|
||||||
|
|||||||
+16
-18
@@ -2,7 +2,7 @@ global:
|
|||||||
scrape_interval: 60s
|
scrape_interval: 60s
|
||||||
evaluation_interval: 60s
|
evaluation_interval: 60s
|
||||||
scrape_timeout: 10s
|
scrape_timeout: 10s
|
||||||
sample_limit: 1000 # Limit samples per scrape to prevent overload
|
sample_limit: 1000
|
||||||
|
|
||||||
# Rule files
|
# Rule files
|
||||||
rule_files:
|
rule_files:
|
||||||
@@ -14,26 +14,12 @@ alerting:
|
|||||||
- targets:
|
- targets:
|
||||||
|
|
||||||
scrape_configs:
|
scrape_configs:
|
||||||
# Prometheus itself
|
# Prometheus
|
||||||
- job_name: 'prometheus'
|
- job_name: 'prometheus'
|
||||||
sample_limit: 50000
|
sample_limit: 50000
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets: ['prometheus:9090']
|
- targets: ['prometheus:9090']
|
||||||
|
|
||||||
# Container metrics via cAdvisor
|
|
||||||
- job_name: 'cadvisor'
|
|
||||||
sample_limit: 50000
|
|
||||||
static_configs:
|
|
||||||
- targets: ['cadvisor:8080']
|
|
||||||
metrics_path: /metrics
|
|
||||||
scheme: http
|
|
||||||
|
|
||||||
# Host metrics via node-exporter
|
|
||||||
- job_name: 'node-exporter'
|
|
||||||
sample_limit: 50000
|
|
||||||
static_configs:
|
|
||||||
- targets: ['node-exporter:9100']
|
|
||||||
|
|
||||||
# Loki metrics
|
# Loki metrics
|
||||||
- job_name: 'loki'
|
- job_name: 'loki'
|
||||||
sample_limit: 50000
|
sample_limit: 50000
|
||||||
@@ -46,8 +32,20 @@ scrape_configs:
|
|||||||
static_configs:
|
static_configs:
|
||||||
- targets: ['grafana:3000']
|
- targets: ['grafana:3000']
|
||||||
|
|
||||||
# Alloy metrics
|
# Node metrics from Alloy - with custom instance name
|
||||||
- job_name: 'alloy'
|
- job_name: 'node'
|
||||||
sample_limit: 50000
|
sample_limit: 50000
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets: ['alloy:12345']
|
- targets: ['alloy:12345']
|
||||||
|
labels:
|
||||||
|
instance: 'main-server' # Use any name you want
|
||||||
|
metrics_path: /metrics
|
||||||
|
|
||||||
|
# Remote node for the second server
|
||||||
|
- job_name: 'node-app-server01'
|
||||||
|
sample_limit: 50000
|
||||||
|
static_configs:
|
||||||
|
- targets: ['10.249.84.41:12345']
|
||||||
|
labels:
|
||||||
|
instance: 'app-server-01' # Use any name you want
|
||||||
|
metrics_path: /metrics
|
||||||
|
|||||||
@@ -0,0 +1,21 @@
|
|||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
contactPoints:
|
||||||
|
- name: ms-teams
|
||||||
|
receivers:
|
||||||
|
- uid: teams-notifications
|
||||||
|
type: teams
|
||||||
|
settings:
|
||||||
|
url: https://prod-152.westeurope.logic.azure.com:443/workflows/6040474cfb314615aabd47694804d7e7/triggers/manual/paths/invoke?api-version=2016-06-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=ehfajgttd-2NC7rdEL5z9rylE4IiyVpgPjiOfDqF_z0
|
||||||
|
title: "{{ if eq .Status \"firing\" }}🔥{{ else }}✅{{ end }} [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}"
|
||||||
|
message: |-
|
||||||
|
{{ if ne .Status "firing" }}Resolved{{ else }}{{ end }}: {{ .CommonAnnotations.summary }}
|
||||||
|
|
||||||
|
{{ .CommonAnnotations.description }}
|
||||||
|
|
||||||
|
{{- if gt (len .Alerts.Firing) 0 }}
|
||||||
|
Alerts:
|
||||||
|
{{- range .Alerts.Firing }}
|
||||||
|
- {{ .Labels.instance }} {{ .Labels.job }}: {{ .Annotations.summary }}
|
||||||
|
{{- end }}
|
||||||
|
{{- end }}
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
policies:
|
||||||
|
- receiver: ms-teams
|
||||||
|
group_by: ['alertname', 'job']
|
||||||
|
repeat_interval: 3h
|
||||||
|
group_interval: 5m
|
||||||
|
routes:
|
||||||
|
- receiver: ms-teams
|
||||||
|
matchers:
|
||||||
|
- severity =~ "critical|warning|info"
|
||||||
|
continue: true
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
{
|
||||||
|
"type": "message",
|
||||||
|
"attachments": [
|
||||||
|
{
|
||||||
|
"contentType": "application/vnd.microsoft.card.adaptive",
|
||||||
|
"content": {
|
||||||
|
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
|
||||||
|
"type": "AdaptiveCard",
|
||||||
|
"version": "1.3",
|
||||||
|
"body": [
|
||||||
|
{
|
||||||
|
"type": "TextBlock",
|
||||||
|
"text": "{{ if eq .Status "firing" }}🔥{{ else }}✅{{ end }} {{ .Status | title }}: {{ .CommonLabels.alertname }}",
|
||||||
|
"weight": "Bolder",
|
||||||
|
"size": "Large"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "TextBlock",
|
||||||
|
"text": "{{ .CommonAnnotations.summary }}",
|
||||||
|
"wrap": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "FactSet",
|
||||||
|
"facts": [
|
||||||
|
{
|
||||||
|
"title": "Status",
|
||||||
|
"value": "{{ .Status | title }}"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"title": "Severity",
|
||||||
|
"value": "{{ .CommonLabels.severity | title }}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "TextBlock",
|
||||||
|
"text": "{{ .CommonAnnotations.description }}",
|
||||||
|
"wrap": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"actions": [
|
||||||
|
{
|
||||||
|
"type": "Action.OpenUrl",
|
||||||
|
"title": "View in Alertmanager",
|
||||||
|
"url": "{{ .ExternalURL }}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
+37
-49
@@ -1,10 +1,9 @@
|
|||||||
services:
|
services:
|
||||||
# Grafana for visualization
|
grafana:
|
||||||
grafanao:
|
|
||||||
image: grafana/grafana:latest
|
image: grafana/grafana:latest
|
||||||
container_name: grafana
|
container_name: grafana
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "wget", "-q", "--spider", "http://localhost:3003"]
|
test: ["CMD", "wget", "-q", "--spider", "http://localhost:3000"]
|
||||||
interval: 30s
|
interval: 30s
|
||||||
timeout: 10s
|
timeout: 10s
|
||||||
retries: 3
|
retries: 3
|
||||||
@@ -14,14 +13,16 @@ services:
|
|||||||
- GF_SECURITY_ADMIN_USER=admin
|
- GF_SECURITY_ADMIN_USER=admin
|
||||||
- GF_SECURITY_ADMIN_PASSWORD=password # Change this in production
|
- GF_SECURITY_ADMIN_PASSWORD=password # Change this in production
|
||||||
- GF_USERS_ALLOW_SIGN_UP=false
|
- GF_USERS_ALLOW_SIGN_UP=false
|
||||||
|
- GF_UNIFIED_ALERTING_ENABLED=true
|
||||||
|
- GF_ALERTING_ENABLED=false
|
||||||
|
- GF_UNIFIED_ALERTING_PROMETHEUS_ALERTS_ENABLED=true
|
||||||
volumes:
|
volumes:
|
||||||
- grafana-data:/var/lib/grafana
|
- grafana-data:/var/lib/grafana
|
||||||
- ./config/provisioning/dashboards:/etc/grafana/provisioning/dashboards
|
- ./config/provisioning/dashboards:/etc/grafana/provisioning/dashboards
|
||||||
- ./config/provisioning/datasources:/etc/grafana/provisioning/datasources
|
- ./config/provisioning/datasources:/etc/grafana/provisioning/datasources
|
||||||
# Add this new mount to store the actual dashboard JSON
|
- ./config/provisioning/alerting:/etc/grafana/provisioning/alerting
|
||||||
- ./config/provisioning/dashboards:/var/lib/grafana/dashboards
|
- ./config/dashboards:/var/lib/grafana/dashboards
|
||||||
depends_on:
|
depends_on:
|
||||||
- loki
|
|
||||||
- prometheus
|
- prometheus
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
networks:
|
networks:
|
||||||
@@ -37,20 +38,15 @@ services:
|
|||||||
networks:
|
networks:
|
||||||
- monitoring
|
- monitoring
|
||||||
|
|
||||||
alloy:
|
# AlertManager for handling alerts
|
||||||
image: grafana/alloy:latest
|
alertmanager:
|
||||||
container_name: grafana-alloy
|
image: quay.io/prometheus/alertmanager:latest
|
||||||
|
container_name: alertmanager
|
||||||
ports:
|
ports:
|
||||||
- "9030:12345"
|
- "9093:9093"
|
||||||
volumes:
|
volumes:
|
||||||
- ./config/alloy.conf:/etc/alloy/config.alloy
|
- ./config/alertmanager.yml:/etc/alertmanager/config.yml
|
||||||
- alloy-data:/var/lib/alloy/data
|
restart: unless-stopped
|
||||||
command:
|
|
||||||
- run
|
|
||||||
- --server.http.listen-addr=0.0.0.0:12345
|
|
||||||
- --storage.path=/var/lib/alloy/data
|
|
||||||
- /etc/alloy/config.alloy
|
|
||||||
restart: unless-stopped
|
|
||||||
networks:
|
networks:
|
||||||
- monitoring
|
- monitoring
|
||||||
|
|
||||||
@@ -72,6 +68,7 @@ services:
|
|||||||
- '--web.console.libraries=/etc/prometheus/console_libraries'
|
- '--web.console.libraries=/etc/prometheus/console_libraries'
|
||||||
- '--web.console.templates=/etc/prometheus/consoles'
|
- '--web.console.templates=/etc/prometheus/consoles'
|
||||||
- '--web.enable-lifecycle'
|
- '--web.enable-lifecycle'
|
||||||
|
- "--web.enable-remote-write-receiver"
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
networks:
|
networks:
|
||||||
- monitoring
|
- monitoring
|
||||||
@@ -87,48 +84,39 @@ services:
|
|||||||
max_attempts: 3
|
max_attempts: 3
|
||||||
window: 120s
|
window: 120s
|
||||||
|
|
||||||
# cAdvisor for container metrics
|
alloy:
|
||||||
cadvisor:
|
image: grafana/alloy:latest
|
||||||
image: gcr.io/cadvisor/cadvisor:v0.49.1
|
container_name: alloy
|
||||||
container_name: cadvisor
|
hostname: localhost # Set a proper hostname
|
||||||
volumes:
|
volumes:
|
||||||
- /:/rootfs:ro
|
- ./config/config.alloy:/etc/alloy/config.alloy
|
||||||
- /var/run:/var/run:ro
|
- /proc:/host/proc:ro # CRITICAL - must be mounted at /host/proc
|
||||||
- /sys:/sys:ro
|
- /sys:/host/sys:ro # CRITICAL - must be mounted at /host/sys
|
||||||
- /var/lib/docker/:/var/lib/docker:ro
|
- /:/rootfs:ro # CRITICAL - must be mounted at /rootfs
|
||||||
- /dev/disk/:/dev/disk:ro
|
|
||||||
ports:
|
|
||||||
- "9050:8080"
|
|
||||||
restart: unless-stopped
|
|
||||||
networks:
|
|
||||||
- monitoring
|
|
||||||
privileged: true
|
|
||||||
devices:
|
|
||||||
- /dev/kmsg:/dev/kmsg
|
|
||||||
|
|
||||||
# Node Exporter for host metrics
|
|
||||||
node-exporter:
|
|
||||||
image: prom/node-exporter:latest
|
|
||||||
container_name: node-exporter
|
|
||||||
volumes:
|
|
||||||
- /proc:/host/proc:ro
|
|
||||||
- /sys:/host/sys:ro
|
|
||||||
- /:/rootfs:ro
|
|
||||||
command:
|
command:
|
||||||
- '--path.procfs=/host/proc'
|
- run
|
||||||
- '--path.sysfs=/host/sys'
|
- --server.http.listen-addr=0.0.0.0:12345
|
||||||
- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
|
- --storage.path=/var/lib/alloy/data
|
||||||
|
- /etc/alloy/config.alloy
|
||||||
ports:
|
ports:
|
||||||
- "9060:9100"
|
- "12345:12345"
|
||||||
restart: unless-stopped
|
privileged: true # CRITICAL for accessing system metrics
|
||||||
|
pid: "host" # CRITICAL for accessing process metrics
|
||||||
|
environment:
|
||||||
|
- HOSTNAME=localhost # Explicitly set the hostname for metrics
|
||||||
networks:
|
networks:
|
||||||
- monitoring
|
- monitoring
|
||||||
|
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
grafana-data:
|
grafana-data:
|
||||||
prometheus-data:
|
prometheus-data:
|
||||||
alloy-data:
|
alloy-data:
|
||||||
driver: local
|
driver: local
|
||||||
|
loki-data:
|
||||||
|
driver: local
|
||||||
|
alertmanager-data:
|
||||||
|
driver: local
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
monitoring:
|
monitoring:
|
||||||
|
|||||||
Reference in New Issue
Block a user