config readme
This commit is contained in:
@@ -1,95 +1,153 @@
|
|||||||
# Monitoring Stack with Grafana, Prometheus, Loki & Alloy
|
# LGP Monitoring Stack
|
||||||
|
|
||||||
This repository contains a complete monitoring stack for visualizing metrics and logs from containers and host systems using Docker Compose.
|
A production-grade monitoring solution for Linux servers using Prometheus, Grafana, Loki, and related components.
|
||||||
|
|
||||||
## Components
|
## Architecture
|
||||||
|
|
||||||
- **Grafana**: Visualization and dashboarding (port 3000)
|
This stack implements a comprehensive monitoring solution with the following components:
|
||||||
- **Prometheus**: Metrics collection and storage (port 9090)
|
|
||||||
- **Loki**: Log aggregation system (port 3100)
|
- **Prometheus**: Time series database and metrics collector with alerting capabilities
|
||||||
- **Alloy**: Grafana complementary observability system (port 12345)
|
- **Grafana**: Visualization platform for metrics, logs, and alerting
|
||||||
- **Node Exporter**: Host metrics exporter (port 9100)
|
- **Loki**: Log aggregation system optimized for Prometheus and Grafana
|
||||||
- **cAdvisor**: Container metrics exporter (port 8080)
|
- **Alloy**: Grafana's observability correlation engine
|
||||||
|
- **cAdvisor**: Container resource usage and performance monitoring
|
||||||
|
- **Node Exporter**: Physical host metrics collector (CPU, memory, disk, network)
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
## Component Details
|
||||||
|
|
||||||
|
| Component | Purpose | Default Port | Data Retention |
|
||||||
|
|-----------|---------|--------------|----------------|
|
||||||
|
| Prometheus | Metrics collection and storage | 3004 | 15d / 10GB |
|
||||||
|
| Grafana | Visualization and dashboarding | 3003 | Persistent |
|
||||||
|
| Loki | Log aggregation | 9020 | Configured via volume |
|
||||||
|
| Alloy | Correlation engine | 9030 | Persistent |
|
||||||
|
| cAdvisor | Container metrics | 9050 | N/A (scrape only) |
|
||||||
|
| Node Exporter | Host metrics | 9060 | N/A (scrape only) |
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
```bash
|
```shell
|
||||||
|
# Optional: Customize configuration
|
||||||
|
# edit config/prometheus.yml, docker-compose.yml, etc.
|
||||||
|
|
||||||
# Start the monitoring stack
|
# Start the monitoring stack
|
||||||
docker-compose up -d
|
docker-compose up -d
|
||||||
|
|
||||||
# View logs
|
# Verify all components are running
|
||||||
docker-compose logs -f
|
docker-compose ps
|
||||||
|
|
||||||
# Stop the stack
|
# Stop the stack
|
||||||
docker-compose down
|
docker-compose down
|
||||||
```
|
```
|
||||||
|
|
||||||
## Services Access
|
## Detailed Configuration
|
||||||
|
|
||||||
- **Grafana**: http://localhost:9010 (user: admin, password: password)
|
### Prometheus
|
||||||
- **Prometheus**: http://localhost:9040
|
|
||||||
- **Loki**: http://localhost:9020
|
|
||||||
- **cAdvisor**: http://localhost:9050
|
|
||||||
- **Node Exporter**: http://localhost:9060
|
|
||||||
- **Alloy**: http://localhost:9030
|
|
||||||
|
|
||||||
## Directory Structure
|
The Prometheus configuration (`config/prometheus.yml`) includes:
|
||||||
|
|
||||||
```
|
- 60s scrape interval for most targets
|
||||||
.
|
- Sample limits to prevent overload (1000 global, 50000 per job)
|
||||||
├── config/
|
- Alert rules in `alerts.yml`
|
||||||
│ ├── alloy.conf # Alloy configuration
|
- Job configurations for all monitoring components
|
||||||
│ ├── loki.yml # Loki configuration
|
|
||||||
│ ├── prometheus.yml # Prometheus configuration
|
|
||||||
│ ├── prometheus-alerts.yml # Prometheus alerting rules
|
|
||||||
│ └── provisioning/
|
|
||||||
│ ├── dashboards/ # Grafana dashboard provisioning
|
|
||||||
│ │ ├── dashboard-provider.yml # Dashboard provider config
|
|
||||||
│ │ └── node-exporter-full.json # Node Exporter dashboard
|
|
||||||
│ └── datasources/ # Grafana datasource provisioning
|
|
||||||
│ └── grafana-datasources.yml # Datasource config
|
|
||||||
├── docker-compose.yml # Docker Compose configuration
|
|
||||||
└── README.md # This file
|
|
||||||
```
|
|
||||||
|
|
||||||
## Preconfigured Features
|
To customize:
|
||||||
|
- Adjust `scrape_interval` for different reporting frequencies
|
||||||
|
- Modify `sample_limit` based on your server capacity
|
||||||
|
- Configure alerting rules in `alerts.yml`
|
||||||
|
|
||||||
- **Dashboards**: Node Exporter Full dashboard pre-installed
|
### Multi-Server Monitoring
|
||||||
- **Datasources**: Prometheus and Loki automatically configured
|
|
||||||
- **Alerts**: Basic host and container alerts configured in Prometheus
|
|
||||||
- **Metrics**: Host and container metrics automatically collected
|
|
||||||
- **Persistent Storage**: All data stored in Docker volumes
|
|
||||||
|
|
||||||
## Configuration
|
For monitoring multiple servers:
|
||||||
|
|
||||||
### Modifying Prometheus Targets
|
1. **Static Configuration**:
|
||||||
|
```yaml
|
||||||
|
# In prometheus.yml
|
||||||
|
- job_name: 'remote-node'
|
||||||
|
static_configs:
|
||||||
|
- targets: ['server1:9100', 'server2:9100', 'server3:9100']
|
||||||
|
labels:
|
||||||
|
env: production
|
||||||
|
```
|
||||||
|
|
||||||
Edit `config/prometheus.yml` to add additional scrape targets.
|
2. **Service Discovery**:
|
||||||
|
- Consider using DNS or file-based discovery for dynamic environments
|
||||||
|
|
||||||
### Adding Grafana Dashboards
|
### Resource Allocation
|
||||||
|
|
||||||
Place JSON dashboard files in `config/provisioning/dashboards/` and restart the stack.
|
The stack is configured with reasonable defaults:
|
||||||
|
- Prometheus: 1-2GB memory reservation
|
||||||
|
- Individual exporters: minimal overhead (~50-100MB each)
|
||||||
|
|
||||||
### Alerting Rules
|
For high-traffic environments, adjust the `deploy.resources` section in `docker-compose.yml`.
|
||||||
|
|
||||||
Edit `config/prometheus-alerts.yml` to modify alerting rules.
|
## Security Considerations
|
||||||
|
|
||||||
|
This setup includes basic security measures:
|
||||||
|
|
||||||
|
1. Container isolation via Docker networks
|
||||||
|
2. Resource limits to prevent DoS
|
||||||
|
3. Basic authentication for Grafana
|
||||||
|
|
||||||
|
**Production Hardening Steps:**
|
||||||
|
- Enable TLS for all components
|
||||||
|
- Use secrets management for credentials
|
||||||
|
- Implement network policies/firewall rules
|
||||||
|
- Set up proper authentication for Prometheus
|
||||||
|
|
||||||
|
## Alerting Setup
|
||||||
|
|
||||||
|
1. **Define alerts** in `config/prometheus-alerts.yml`:
|
||||||
|
```yaml
|
||||||
|
groups:
|
||||||
|
- name: host_alerts
|
||||||
|
rules:
|
||||||
|
- alert: HighCPULoad
|
||||||
|
expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
|
||||||
|
for: 5m
|
||||||
|
labels:
|
||||||
|
severity: warning
|
||||||
|
annotations:
|
||||||
|
summary: "High CPU load on {{ $labels.instance }}"
|
||||||
|
description: "CPU load is > 80% for 5 minutes"
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Configure Alertmanager** to handle notifications (email, Slack, PagerDuty, etc.)
|
||||||
|
|
||||||
|
## Performance Tuning
|
||||||
|
|
||||||
|
- **Prometheus retention**: Adjust `--storage.tsdb.retention.time` and `--storage.tsdb.retention.size`
|
||||||
|
- **Scrape interval**: Balance between granularity and performance
|
||||||
|
- **Query optimization**: Use recording rules for complex queries
|
||||||
|
- **Resource allocation**: Adjust container memory limits based on load
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
**Dashboards not showing up:**
|
Common issues and solutions:
|
||||||
- Check Grafana logs: `docker-compose logs grafana`
|
|
||||||
- Verify directory structure matches documentation
|
|
||||||
- Ensure dashboard JSON files are valid
|
|
||||||
|
|
||||||
**Service won't start:**
|
1. **Prometheus not starting**: Check volume permissions and storage capacity
|
||||||
- Check for port conflicts
|
2. **Missing metrics**: Verify scrape configurations and target accessibility
|
||||||
- Verify all required files exist in the correct locations
|
3. **High memory usage**: Adjust retention settings and sample limits
|
||||||
- Check individual service logs: `docker-compose logs [service_name]`
|
4. **Dashboard errors**: Check data source configuration in Grafana
|
||||||
|
|
||||||
## Security Notes
|
Debug commands:
|
||||||
|
```shell
|
||||||
|
# Check Prometheus targets
|
||||||
|
curl -s http://localhost:3004/api/v1/targets | jq .
|
||||||
|
|
||||||
This setup uses default credentials and is intended for local development or testing. For production use:
|
# Verify scrape metrics
|
||||||
- Change default Grafana password
|
curl -s http://localhost:9060/metrics | grep node_
|
||||||
- Set up proper authentication for all services
|
|
||||||
- Use secrets management for credentials
|
# Check Loki logs
|
||||||
- Configure TLS for all exposed endpoints
|
docker-compose logs loki
|
||||||
|
```
|
||||||
|
|
||||||
|
## Dashboard Examples
|
||||||
|
|
||||||
|
The stack comes with pre-configured dashboards for:
|
||||||
|
- Node Exporter metrics (CPU, memory, disk, network)
|
||||||
|
- Container performance monitoring
|
||||||
|
- System overview metrics
|
||||||
|
- Log visualization and analysis
|
||||||
|
|||||||
Reference in New Issue
Block a user