diff --git a/README.md b/README.md index 1681a44..8d5b614 100644 --- a/README.md +++ b/README.md @@ -1,95 +1,153 @@ -# Monitoring Stack with Grafana, Prometheus, Loki & Alloy +# LGP Monitoring Stack -This repository contains a complete monitoring stack for visualizing metrics and logs from containers and host systems using Docker Compose. +A production-grade monitoring solution for Linux servers using Prometheus, Grafana, Loki, and related components. -## Components +## Architecture -- **Grafana**: Visualization and dashboarding (port 3000) -- **Prometheus**: Metrics collection and storage (port 9090) -- **Loki**: Log aggregation system (port 3100) -- **Alloy**: Grafana complementary observability system (port 12345) -- **Node Exporter**: Host metrics exporter (port 9100) -- **cAdvisor**: Container metrics exporter (port 8080) +This stack implements a comprehensive monitoring solution with the following components: + +- **Prometheus**: Time series database and metrics collector with alerting capabilities +- **Grafana**: Visualization platform for metrics, logs, and alerting +- **Loki**: Log aggregation system optimized for Prometheus and Grafana +- **Alloy**: Grafana's observability correlation engine +- **cAdvisor**: Container resource usage and performance monitoring +- **Node Exporter**: Physical host metrics collector (CPU, memory, disk, network) + +![Monitoring Stack Architecture](https://via.placeholder.com/800x400?text=Monitoring+Stack+Architecture) + +## Component Details + +| Component | Purpose | Default Port | Data Retention | +|-----------|---------|--------------|----------------| +| Prometheus | Metrics collection and storage | 3004 | 15d / 10GB | +| Grafana | Visualization and dashboarding | 3003 | Persistent | +| Loki | Log aggregation | 9020 | Configured via volume | +| Alloy | Correlation engine | 9030 | Persistent | +| cAdvisor | Container metrics | 9050 | N/A (scrape only) | +| Node Exporter | Host metrics | 9060 | N/A (scrape only) | ## Quick Start -```bash +```shell +# Optional: Customize configuration +# edit config/prometheus.yml, docker-compose.yml, etc. + # Start the monitoring stack docker-compose up -d -# View logs -docker-compose logs -f +# Verify all components are running +docker-compose ps # Stop the stack docker-compose down ``` -## Services Access +## Detailed Configuration -- **Grafana**: http://localhost:9010 (user: admin, password: password) -- **Prometheus**: http://localhost:9040 -- **Loki**: http://localhost:9020 -- **cAdvisor**: http://localhost:9050 -- **Node Exporter**: http://localhost:9060 -- **Alloy**: http://localhost:9030 +### Prometheus -## Directory Structure +The Prometheus configuration (`config/prometheus.yml`) includes: -``` -. -├── config/ -│ ├── alloy.conf # Alloy configuration -│ ├── loki.yml # Loki configuration -│ ├── prometheus.yml # Prometheus configuration -│ ├── prometheus-alerts.yml # Prometheus alerting rules -│ └── provisioning/ -│ ├── dashboards/ # Grafana dashboard provisioning -│ │ ├── dashboard-provider.yml # Dashboard provider config -│ │ └── node-exporter-full.json # Node Exporter dashboard -│ └── datasources/ # Grafana datasource provisioning -│ └── grafana-datasources.yml # Datasource config -├── docker-compose.yml # Docker Compose configuration -└── README.md # This file -``` +- 60s scrape interval for most targets +- Sample limits to prevent overload (1000 global, 50000 per job) +- Alert rules in `alerts.yml` +- Job configurations for all monitoring components -## Preconfigured Features +To customize: +- Adjust `scrape_interval` for different reporting frequencies +- Modify `sample_limit` based on your server capacity +- Configure alerting rules in `alerts.yml` -- **Dashboards**: Node Exporter Full dashboard pre-installed -- **Datasources**: Prometheus and Loki automatically configured -- **Alerts**: Basic host and container alerts configured in Prometheus -- **Metrics**: Host and container metrics automatically collected -- **Persistent Storage**: All data stored in Docker volumes +### Multi-Server Monitoring -## Configuration +For monitoring multiple servers: -### Modifying Prometheus Targets +1. **Static Configuration**: + ```yaml + # In prometheus.yml + - job_name: 'remote-node' + static_configs: + - targets: ['server1:9100', 'server2:9100', 'server3:9100'] + labels: + env: production + ``` -Edit `config/prometheus.yml` to add additional scrape targets. +2. **Service Discovery**: + - Consider using DNS or file-based discovery for dynamic environments -### Adding Grafana Dashboards +### Resource Allocation -Place JSON dashboard files in `config/provisioning/dashboards/` and restart the stack. +The stack is configured with reasonable defaults: +- Prometheus: 1-2GB memory reservation +- Individual exporters: minimal overhead (~50-100MB each) -### Alerting Rules +For high-traffic environments, adjust the `deploy.resources` section in `docker-compose.yml`. -Edit `config/prometheus-alerts.yml` to modify alerting rules. +## Security Considerations + +This setup includes basic security measures: + +1. Container isolation via Docker networks +2. Resource limits to prevent DoS +3. Basic authentication for Grafana + +**Production Hardening Steps:** +- Enable TLS for all components +- Use secrets management for credentials +- Implement network policies/firewall rules +- Set up proper authentication for Prometheus + +## Alerting Setup + +1. **Define alerts** in `config/prometheus-alerts.yml`: + ```yaml + groups: + - name: host_alerts + rules: + - alert: HighCPULoad + expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 + for: 5m + labels: + severity: warning + annotations: + summary: "High CPU load on {{ $labels.instance }}" + description: "CPU load is > 80% for 5 minutes" + ``` + +2. **Configure Alertmanager** to handle notifications (email, Slack, PagerDuty, etc.) + +## Performance Tuning + +- **Prometheus retention**: Adjust `--storage.tsdb.retention.time` and `--storage.tsdb.retention.size` +- **Scrape interval**: Balance between granularity and performance +- **Query optimization**: Use recording rules for complex queries +- **Resource allocation**: Adjust container memory limits based on load ## Troubleshooting -**Dashboards not showing up:** -- Check Grafana logs: `docker-compose logs grafana` -- Verify directory structure matches documentation -- Ensure dashboard JSON files are valid +Common issues and solutions: -**Service won't start:** -- Check for port conflicts -- Verify all required files exist in the correct locations -- Check individual service logs: `docker-compose logs [service_name]` +1. **Prometheus not starting**: Check volume permissions and storage capacity +2. **Missing metrics**: Verify scrape configurations and target accessibility +3. **High memory usage**: Adjust retention settings and sample limits +4. **Dashboard errors**: Check data source configuration in Grafana -## Security Notes +Debug commands: +```shell +# Check Prometheus targets +curl -s http://localhost:3004/api/v1/targets | jq . -This setup uses default credentials and is intended for local development or testing. For production use: -- Change default Grafana password -- Set up proper authentication for all services -- Use secrets management for credentials -- Configure TLS for all exposed endpoints +# Verify scrape metrics +curl -s http://localhost:9060/metrics | grep node_ + +# Check Loki logs +docker-compose logs loki +``` + +## Dashboard Examples + +The stack comes with pre-configured dashboards for: +- Node Exporter metrics (CPU, memory, disk, network) +- Container performance monitoring +- System overview metrics +- Log visualization and analysis