config readme

This commit is contained in:
Bartal Læarsson
2025-03-11 14:26:35 +00:00
parent cf3968ac2c
commit a89c3ad3cc
+120 -62
View File
@@ -1,95 +1,153 @@
# Monitoring Stack with Grafana, Prometheus, Loki & Alloy # LGP Monitoring Stack
This repository contains a complete monitoring stack for visualizing metrics and logs from containers and host systems using Docker Compose. A production-grade monitoring solution for Linux servers using Prometheus, Grafana, Loki, and related components.
## Components ## Architecture
- **Grafana**: Visualization and dashboarding (port 3000) This stack implements a comprehensive monitoring solution with the following components:
- **Prometheus**: Metrics collection and storage (port 9090)
- **Loki**: Log aggregation system (port 3100) - **Prometheus**: Time series database and metrics collector with alerting capabilities
- **Alloy**: Grafana complementary observability system (port 12345) - **Grafana**: Visualization platform for metrics, logs, and alerting
- **Node Exporter**: Host metrics exporter (port 9100) - **Loki**: Log aggregation system optimized for Prometheus and Grafana
- **cAdvisor**: Container metrics exporter (port 8080) - **Alloy**: Grafana's observability correlation engine
- **cAdvisor**: Container resource usage and performance monitoring
- **Node Exporter**: Physical host metrics collector (CPU, memory, disk, network)
![Monitoring Stack Architecture](https://via.placeholder.com/800x400?text=Monitoring+Stack+Architecture)
## Component Details
| Component | Purpose | Default Port | Data Retention |
|-----------|---------|--------------|----------------|
| Prometheus | Metrics collection and storage | 3004 | 15d / 10GB |
| Grafana | Visualization and dashboarding | 3003 | Persistent |
| Loki | Log aggregation | 9020 | Configured via volume |
| Alloy | Correlation engine | 9030 | Persistent |
| cAdvisor | Container metrics | 9050 | N/A (scrape only) |
| Node Exporter | Host metrics | 9060 | N/A (scrape only) |
## Quick Start ## Quick Start
```bash ```shell
# Optional: Customize configuration
# edit config/prometheus.yml, docker-compose.yml, etc.
# Start the monitoring stack # Start the monitoring stack
docker-compose up -d docker-compose up -d
# View logs # Verify all components are running
docker-compose logs -f docker-compose ps
# Stop the stack # Stop the stack
docker-compose down docker-compose down
``` ```
## Services Access ## Detailed Configuration
- **Grafana**: http://localhost:9010 (user: admin, password: password) ### Prometheus
- **Prometheus**: http://localhost:9040
- **Loki**: http://localhost:9020
- **cAdvisor**: http://localhost:9050
- **Node Exporter**: http://localhost:9060
- **Alloy**: http://localhost:9030
## Directory Structure The Prometheus configuration (`config/prometheus.yml`) includes:
``` - 60s scrape interval for most targets
. - Sample limits to prevent overload (1000 global, 50000 per job)
├── config/ - Alert rules in `alerts.yml`
│ ├── alloy.conf # Alloy configuration - Job configurations for all monitoring components
│ ├── loki.yml # Loki configuration
│ ├── prometheus.yml # Prometheus configuration To customize:
│ ├── prometheus-alerts.yml # Prometheus alerting rules - Adjust `scrape_interval` for different reporting frequencies
│ └── provisioning/ - Modify `sample_limit` based on your server capacity
│ ├── dashboards/ # Grafana dashboard provisioning - Configure alerting rules in `alerts.yml`
│ │ ├── dashboard-provider.yml # Dashboard provider config
│ │ └── node-exporter-full.json # Node Exporter dashboard ### Multi-Server Monitoring
│ └── datasources/ # Grafana datasource provisioning
│ └── grafana-datasources.yml # Datasource config For monitoring multiple servers:
├── docker-compose.yml # Docker Compose configuration
└── README.md # This file 1. **Static Configuration**:
```yaml
# In prometheus.yml
- job_name: 'remote-node'
static_configs:
- targets: ['server1:9100', 'server2:9100', 'server3:9100']
labels:
env: production
``` ```
## Preconfigured Features 2. **Service Discovery**:
- Consider using DNS or file-based discovery for dynamic environments
- **Dashboards**: Node Exporter Full dashboard pre-installed ### Resource Allocation
- **Datasources**: Prometheus and Loki automatically configured
- **Alerts**: Basic host and container alerts configured in Prometheus
- **Metrics**: Host and container metrics automatically collected
- **Persistent Storage**: All data stored in Docker volumes
## Configuration The stack is configured with reasonable defaults:
- Prometheus: 1-2GB memory reservation
- Individual exporters: minimal overhead (~50-100MB each)
### Modifying Prometheus Targets For high-traffic environments, adjust the `deploy.resources` section in `docker-compose.yml`.
Edit `config/prometheus.yml` to add additional scrape targets. ## Security Considerations
### Adding Grafana Dashboards This setup includes basic security measures:
Place JSON dashboard files in `config/provisioning/dashboards/` and restart the stack. 1. Container isolation via Docker networks
2. Resource limits to prevent DoS
3. Basic authentication for Grafana
### Alerting Rules **Production Hardening Steps:**
- Enable TLS for all components
- Use secrets management for credentials
- Implement network policies/firewall rules
- Set up proper authentication for Prometheus
Edit `config/prometheus-alerts.yml` to modify alerting rules. ## Alerting Setup
1. **Define alerts** in `config/prometheus-alerts.yml`:
```yaml
groups:
- name: host_alerts
rules:
- alert: HighCPULoad
expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU load on {{ $labels.instance }}"
description: "CPU load is > 80% for 5 minutes"
```
2. **Configure Alertmanager** to handle notifications (email, Slack, PagerDuty, etc.)
## Performance Tuning
- **Prometheus retention**: Adjust `--storage.tsdb.retention.time` and `--storage.tsdb.retention.size`
- **Scrape interval**: Balance between granularity and performance
- **Query optimization**: Use recording rules for complex queries
- **Resource allocation**: Adjust container memory limits based on load
## Troubleshooting ## Troubleshooting
**Dashboards not showing up:** Common issues and solutions:
- Check Grafana logs: `docker-compose logs grafana`
- Verify directory structure matches documentation
- Ensure dashboard JSON files are valid
**Service won't start:** 1. **Prometheus not starting**: Check volume permissions and storage capacity
- Check for port conflicts 2. **Missing metrics**: Verify scrape configurations and target accessibility
- Verify all required files exist in the correct locations 3. **High memory usage**: Adjust retention settings and sample limits
- Check individual service logs: `docker-compose logs [service_name]` 4. **Dashboard errors**: Check data source configuration in Grafana
## Security Notes Debug commands:
```shell
# Check Prometheus targets
curl -s http://localhost:3004/api/v1/targets | jq .
This setup uses default credentials and is intended for local development or testing. For production use: # Verify scrape metrics
- Change default Grafana password curl -s http://localhost:9060/metrics | grep node_
- Set up proper authentication for all services
- Use secrets management for credentials # Check Loki logs
- Configure TLS for all exposed endpoints docker-compose logs loki
```
## Dashboard Examples
The stack comes with pre-configured dashboards for:
- Node Exporter metrics (CPU, memory, disk, network)
- Container performance monitoring
- System overview metrics
- Log visualization and analysis