93 lines
3.4 KiB
Markdown
93 lines
3.4 KiB
Markdown
# Hagfisk Faroese fisheries data pipeline and dashboard. Pulls monthly landing statistics from Hagstova Føroya's PX-Web API, stuffs them into a local DuckDB, and serves a filtered ECharts dashboard — all from a single Rust binary.
|
|
|
|
Named after the hagfish: a deep-sea creature famous for producing copious amounts of slime and being generally unpleasant. The name stuck anyway. The fish has no backbone. Neither does this README.
|
|
|
|
## What It Does
|
|
|
|
|
|
---
|
|
|
|
- Fetches fisheries data (mass + value by species, gear, zone, month) from `statbank.hagstova.fo`
|
|
- Stores everything in a local DuckDB file
|
|
- Serves a REST API with filtered queries, summaries, and Parquet export
|
|
- Renders an interactive dashboard with line, price-trend, and stacked bar charts
|
|
- Runs monthly ingestion automatically via systemd timer (28th of each month, 03:00)
|
|
|
|
## Tech Stack
|
|
|
|
| Thing | Choice | Why |
|
|
|---|---|---|
|
|
| Language | Rust 2024 | Fast, safe, single static binary. No runtime, no VM, no nonsense. |
|
|
| Web framework | Axum | Async, typed, plays well with Tokio. Doesn't fight you. |
|
|
| Database | DuckDB | Embedded OLAP. No server, no config, just a file. Perfect for analytics on a single machine. |
|
|
| Charts | ECharts | Powerful, flexible, no build step. Vanilla JS, no framework tax. |
|
|
| Static assets | rust-embed | Compiled into the binary. One artifact to deploy, nothing to forget. |
|
|
| Logging | tracing + tracing-subscriber | Structured logs to file and journal. |
|
|
| Deployment | Bare metal + systemd | No containers. No compose files. No port mappings. Just a binary and a `.service` unit. |
|
|
|
|
## Getting Started
|
|
|
|
```bash
|
|
cargo build --release
|
|
./target/release/hagfisk ingest --full
|
|
./target/release/hagfisk serve
|
|
```
|
|
|
|
Dashboard lives at `http://localhost:8090`.
|
|
|
|
## Configuration
|
|
|
|
`config.json` next to the binary:
|
|
|
|
```json
|
|
{
|
|
"duckdb_path": "hagfish.db",
|
|
"bind_address": "127.0.0.1:8090",
|
|
"data_source_url": "https://statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px",
|
|
"log_file_path": "hagfish.log",
|
|
"allowed_origins": []
|
|
}
|
|
```
|
|
|
|
## CLI
|
|
|
|
```
|
|
hagfisk serve Start the web server
|
|
hagfisk ingest [--full] Fetch data (incremental or full backfill)
|
|
hagfisk export -o FILE Export landings to Parquet
|
|
```
|
|
|
|
## Deployment
|
|
|
|
The `justfile` handles everything, but you'll need to [install](https://just.systems/man/en/installation.html) it on your machine.
|
|
|
|
```
|
|
just deploy # Build, rsync, restore SELinux contexts, restart
|
|
just deploy-reingest # Same + full re-ingest
|
|
just logs-app # Tail application logs
|
|
just ps-app # Check service status
|
|
```
|
|
|
|
Systemd units live in `systemd/` — drop them in `~/.config/systemd/user/` and enable the timer for monthly ingestion.
|
|
|
|
## API
|
|
|
|
| Endpoint | What |
|
|
|---|---|
|
|
| `GET /healthz` | Alive check |
|
|
| `GET /api/species` | Species codes + Faroese names |
|
|
| `GET /api/zones` | Economic zone codes |
|
|
| `GET /api/gear` | Fishing gear codes |
|
|
| `GET /api/landings` | Filtered landing records (month range, species, gear, zone, measure) |
|
|
| `GET /api/summary` | Monthly aggregates, top 10 species, price/kg trend |
|
|
| `GET /api/summary/monthly-breakdown` | Per-month per-species breakdown |
|
|
| `GET /api/available-filters` | Cascading filter options based on active selections |
|
|
| `GET /api/export.parquet` | Full dataset as Parquet download |
|
|
|
|
Full PX-Web API reference in `docs/API.md`.
|
|
|
|
---
|
|
|
|
Like the hagfish itself: not glamorous, but it works.
|
|
|