phase 1 complete
This commit is contained in:
+192
@@ -0,0 +1,192 @@
|
||||
|
||||
# API Documentation
|
||||
|
||||
This document describes the PX-Web API used by hagfish to fetch Faroese fisheries statistics from the official Statbank.
|
||||
|
||||
## EndpointBase URL: https://statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px
|
||||
- **GET**: Returns metadata (table structure, dimension codes, labels)
|
||||
- **POST**: Returns data in JSON-stat2 format
|
||||
|
||||
## Metadata Request (GET)
|
||||
|
||||
### Requestbash
|
||||
curl -s "https://statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px"
|
||||
### Response Structurejson
|
||||
{
|
||||
"title": "AVR01010 Avreiðingar í nøgd og virði...",
|
||||
"variables": [
|
||||
{
|
||||
"code": "measure",
|
||||
"text": "mát",
|
||||
"role": null,
|
||||
"values": ["MASS", "VALUE"],
|
||||
"valueTexts": ["Nøgd", "Virði"]
|
||||
}
|
||||
]
|
||||
}
|
||||
### Fields
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `title` | string | Table name in Faroese |
|
||||
| `variables[].code` | string | Dimension identifier (used in queries) |
|
||||
| `variables[].text` | string | Human-readable label in Faroese |
|
||||
| `variables[].role` | null/string | Classification (always `null` on this endpoint) |
|
||||
| `variables[].values` | string[] | Valid dimension codes |
|
||||
| `variables[].valueTexts` | string[] | Display labels (parallel with `values`) |
|
||||
|
||||
### Verified Dimension Codes
|
||||
|
||||
| Variable Code | Faroese Label | Values |
|
||||
|---------------|---------------|--------|
|
||||
| `measure` | mát | `MASS`, `VALUE` |
|
||||
| `Species (ASFIS2022)` | Fiskaslag (ASFIS2022) | `TOTAL`, `148XXXXXXX00000`, ... (72 total) |
|
||||
| `Fishing Gear (ISSCFG2016)` | Reiðskapur (ISSCFG2016) | `TOTAL`, ... (12 total) |
|
||||
| `Economic Zone (GEONOM2023)` | Búskapar øki (GEONOM2023) | `TOTAL`, ... (10 total) |
|
||||
| `Processing (EUMOFAPresentation)` | Virking (EUMOFAPresentation) | `TOTAL`, ... (6 total) |
|
||||
| `Preservation (EUMOFAPreservation)` | Viðgerð (EUMOFAPreservation) | `TOTAL`, ... (9 total) |
|
||||
| `Shipsize` | Skipastødd | `TOTAL`, ... (11 total) |
|
||||
| `month` | mánaður | `2015M01` through `2026M05` (137 months) |
|
||||
|
||||
**Note**: `role` is always `null`. Time dimension identification must use `code == "month"`.
|
||||
|
||||
---
|
||||
|
||||
## Data Request (POST)
|
||||
|
||||
### Request Body Formatjson
|
||||
{
|
||||
"query": [
|
||||
{
|
||||
"code": "dimension_code",
|
||||
"selection": {
|
||||
"filter": "item",
|
||||
"values": ["value1", "value2"]
|
||||
}
|
||||
}
|
||||
],
|
||||
"response": {
|
||||
"format": "json-stat2"
|
||||
}
|
||||
}
|
||||
### Filter Types
|
||||
|
||||
| Filter | Values | Description |
|
||||
|--------|--------|-------------|
|
||||
| `item` | explicit codes | Select specific values |
|
||||
| `all` | `["*"]` | Wildcard — select all values |
|
||||
| `top` | `["TOP_N"]` | Top N values by magnitude |
|
||||
|
||||
### Example Querybash
|
||||
curl -s -X POST
|
||||
-H "Content-Type: application/json"
|
||||
-d '{
|
||||
"query": [
|
||||
{"code": "month", "selection": {"filter": "item", "values": ["2024M01", "2024M02"]}},
|
||||
{"code": "Species (ASFIS2022)", "selection": {"filter": "all", "values": [""]}},
|
||||
{"code": "Fishing Gear (ISSCFG2016)", "selection": {"filter": "all", "values": [""]}},
|
||||
{"code": "Economic Zone (GEONOM2023)", "selection": {"filter": "all", "values": ["*"]}},
|
||||
{"code": "Processing (EUMOFAPresentation)", "selection": {"filter": "item", "values": ["TOTAL"]}},
|
||||
{"code": "Preservation (EUMOFAPreservation)", "selection": {"filter": "item", "values": ["TOTAL"]}},
|
||||
{"code": "Shipsize", "selection": {"filter": "item", "values": ["TOTAL"]}},
|
||||
{"code": "measure", "selection": {"filter": "item", "values": ["MASS", "VALUE"]}}
|
||||
],
|
||||
"response": {"format": "json-stat2"}
|
||||
}'
|
||||
"https://statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px"
|
||||
### Cell Limit
|
||||
|
||||
- Maximum cells per query: ~8,000,000
|
||||
- Recommended maximum: 1,000,000 for reliability
|
||||
- To fetch full dataset, paginate by month or use smaller dimension selections
|
||||
|
||||
---
|
||||
|
||||
## Response Format (JSON-stat2)
|
||||
|
||||
### Structurejson
|
||||
{
|
||||
"class": "dataset",
|
||||
"label": "...",
|
||||
"id": ["measure", "Species (ASFIS2022)", ..., "month"],
|
||||
"size": [2, 72, 12, 10, 1, 1, 1, 2],
|
||||
"dimension": {
|
||||
"measure": {
|
||||
"label": "mát",
|
||||
"category": {
|
||||
"index": {"MASS": 0, "VALUE": 1},
|
||||
"label": {"MASS": "Nøgd", "VALUE": "Virði"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"value": [73653738, 1234567, ...]
|
||||
}
|
||||
### Fields
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `id` | Array of dimension codes in order (defines cube layout) |
|
||||
| `size` | Cardinality per dimension (parallel with `id`) |
|
||||
| `dimension.<code>.category.index` | Maps value code → numeric position in dimension |
|
||||
| `dimension.<code>.category.label` | Maps value code → display text |
|
||||
| `value` | Flattened array of measurements (row-major order) |
|
||||
|
||||
### Row-Major Index Decoding
|
||||
|
||||
Values are stored in row-major order: the last dimension (`month`) varies fastest.
|
||||
|
||||
Given `size = [2, 72, 12, 10, 1, 1, 1, 2]`:
|
||||
|
||||
- Flat index `0` → `[0,0,0,0,0,0,0,0]` = measure=MASS, species=TOTAL, ..., month=2024M01
|
||||
- Flat index `1` → `[0,0,0,0,0,0,0,1]` = measure=MASS, species=TOTAL, ..., month=2024M02
|
||||
- Flat index `2` → `[0,0,0,0,0,0,1,0]` = measure=MASS, species=SPECIES_1, ..., month=2024M01
|
||||
|
||||
Decoding algorithm (reverse modulo):rust
|
||||
fn decode(flat_index: usize, sizes: &[usize]) -> Vec<usize> {
|
||||
let mut indices = Vec::new();
|
||||
let mut remaining = flat_index;
|
||||
for &size in sizes.iter().rev() {
|
||||
indices.push(remaining % size);
|
||||
remaining /= size;
|
||||
}
|
||||
indices.reverse();
|
||||
indices
|
||||
}
|
||||
### Sentinel Values
|
||||
|
||||
| Value | Meaning |
|
||||
|-------|---------|
|
||||
| `-1.0` | Missing/no data (treated as `NULL`) |
|
||||
| `null` | Missing/no data (already nullable in JSON) |
|
||||
|
||||
---
|
||||
|
||||
## Known Constraints
|
||||
|
||||
1. **Cell limit**: ~8 million cells per query
|
||||
2. **Rate limits**: Unknown — assume reasonable backoff for large downloads
|
||||
3. **Language**: API responds in requested language (`fo` or `en`)
|
||||
4. **Time zone**: No timezone specified — treat timestamps as local
|
||||
5. **Updates**: Data updated irregularly — metadata timestamp in response header
|
||||
|
||||
---
|
||||
|
||||
## Integration Checklist
|
||||
|
||||
- [x] Metadata endpoint reachable via GET
|
||||
- [x] POST queries return valid JSON-stat2
|
||||
- [x] Dimension codes match `variables[].code` from metadata
|
||||
- [x] Row-major index decoding verified
|
||||
- [x] Sentinel value coercion (`-1.0` → `None`) implemented
|
||||
- [x] Unicode (Faroese characters) handled correctly
|
||||
- [ ] Incremental ingestion logic tested
|
||||
- [ ] Parquet export tested
|
||||
- [ ] Error handling for network timeouts implemented
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Official PxWeb documentation: https://pxweb.github.io/docs/
|
||||
- JSON-stat2 specification: http://json-stat.org/format/
|
||||
- Hagstova Føroya: https://www.hagstova.fo/
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
# HAGFISH Project todo
|
||||
|
||||
`GET statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px — metadata`
|
||||
POST same URL — JSON-stat2 query
|
||||
8 dimensions, "-" → NULL
|
||||
|
||||
## Ingestion
|
||||
|
||||
Monthly run (cron or systemd timer, user's choice)
|
||||
Incremental with full backfill option
|
||||
GET metadata → build lookup maps → POST query → parse → insert into DuckDB
|
||||
Upsert semantics for revised months
|
||||
|
||||
## Storage
|
||||
DuckDB on disk
|
||||
|
||||
## Fact table
|
||||
landings(species_code, gear_code, zone_code, processing_code, preservation_code, shipsize_code, month, mass_kg, value_kr)
|
||||
Lookup tables: species, gear, zone, processing, preservation, shipsize
|
||||
Parquet export per run
|
||||
|
||||
## API (Axum)
|
||||
|
||||
GET /api/species — list species codes + Faroese names
|
||||
GET /api/landings — filtered query, JSON response
|
||||
GET /api/summary — aggregates
|
||||
GET /api/export.parquet — download Parquet
|
||||
Serve embedded static frontend
|
||||
|
||||
## Frontend
|
||||
JS + ECharts
|
||||
|
||||
Line chart, stacked bar, donut, dropdown filters
|
||||
Plain HTML/CSS/JS, no build step
|
||||
|
||||
## Deployment
|
||||
|
||||
Statically linked Rust binary
|
||||
config.json for settings (DuckDB path, bind addr, data source URL)
|
||||
Systemd timer for monthly ingestion (bare metal, no containers)
|
||||
|
||||
|
||||
## Project TODO
|
||||
|
||||
hagfish/
|
||||
├── Cargo.toml
|
||||
├── Taskfile.yml
|
||||
├── config.json
|
||||
├── static/
|
||||
│ ├── index.html
|
||||
│ ├── app.js
|
||||
│ └── style.css
|
||||
└── src/
|
||||
├── main.rs
|
||||
├── ingest.rs
|
||||
├── db.rs
|
||||
├── api.rs
|
||||
└── types.rs
|
||||
|
||||
### Phase 1: Types & Ingestion
|
||||
|
||||
- [x] 1.1 Define types in types.rs: MetadataResponse, VariableMeta, DataResponse, DataRow, Query, Selection, QueryItem, Config — all with serde derives
|
||||
- [x] 1.2 Implement ingest.rs::fetch_metadata(url) — GET request, parse JSON, return HashMap<(variable_code, value_code), faroese_label>
|
||||
- [x] 1.3 Implement ingest.rs::build_query(months: &[String]) — construct POST body with all species/gear/zones set to "*", processing/preservation/shipsize set to TOTAL, measure set to both MASS and VALUE
|
||||
- [x] 1.4 Implement ingest.rs::fetch_data(url, query) — POST request, parse JSON-stat2 response, return Vec<DataRow>
|
||||
- [x] 1.5 Implement ingest.rs::parse_row(row, lookup_maps) — decode key[] positions into labeled Landing struct. Handle "-" → None.
|
||||
- [x] 1.6 Write unit tests: mock JSON-stat2 response, verify key-to-label mapping, verify "-" handling, verify Faroese Unicode characters in species names (ð, á, í, ý, ø, ó)
|
||||
|
||||
### Phase 2: DuckDB Storage
|
||||
|
||||
- [ ] 2.1 Add duckdb crate dependency (bundled feature)
|
||||
- [ ] 2.2 Implement db.rs::init(path) — create tables: landings fact table + 6 lookup tables. Add indexes on month, species_code.
|
||||
- [ ] 2.3 Implement db.rs::upsert_landings(rows) — batch insert with delete+insert per month or INSERT OR REPLACE
|
||||
- [ ] 2.4 Implement db.rs::update_lookups(metadata) — populate lookup tables from metadata response
|
||||
- [ ] 2.5 Implement db.rs::get_last_month() — query max month from landings table for incremental ingestion
|
||||
- [ ] 2.6 Implement db.rs::export_parquet(path) — COPY landings TO 'path' (FORMAT PARQUET) partitioned by month
|
||||
- [ ] 2.7 Write integration tests: init in-memory DB, insert sample rows, query back, verify NULL handling
|
||||
|
||||
### Phase 3: API (Axum)
|
||||
|
||||
- [ ] 3.1 Set up Axum router in main.rs with Tokio runtime. AppState holds DuckDB connection wrapped in Mutex.
|
||||
- [ ] 3.2 Implement GET /api/species — query species lookup table, return JSON array
|
||||
- [ ] 3.3 Implement GET /api/landings — parse query params, build DuckDB SQL with WHERE clauses. Support: months, species, gear, zone, measure filters.
|
||||
- [ ] 3.4 Implement GET /api/summary — aggregate query: total mass + value by month, top 10 species by value, price/kg trend
|
||||
- [ ] 3.5 Implement GET /api/export.parquet — generate and stream Parquet via DuckDB COPY
|
||||
- [ ] 3.6 Implement GET /healthz
|
||||
- [ ] 3.7 Serve static files via rust-embed
|
||||
- [ ] 3.8 Write API tests
|
||||
|
||||
### Phase 4: Frontend
|
||||
|
||||
- [ ] 4.1 index.html — dropdown filters (species, zone, gear, month range) and 3 chart containers
|
||||
- [ ] 4.2 app.js — fetch species list on load, populate dropdowns, fetch /api/landings, render charts
|
||||
- [ ] 4.3 ECharts line chart: x=month, y=mass/value toggle
|
||||
- [ ] 4.4 ECharts stacked bar: x=month, y=value by species (top 10 + "other")
|
||||
- [ ] 4.5 ECharts donut: species distribution for selected month
|
||||
- [ ] 4.6 Loading states, error handling, empty state
|
||||
- [ ] 4.7 Responsive layout, plain CSS
|
||||
|
||||
### Phase 5: CLI & Scheduling
|
||||
|
||||
- [ ] 5.1 Add clap derive subcommands: hagfish ingest [--full] and hagfish serve
|
||||
- [ ] 5.2 Implement incremental logic: read get_last_month(), compute remaining months from metadata, fetch in batches if >12 months
|
||||
- [ ] 5.3 Log ingestion runs with slog (rows inserted, duration, errors)
|
||||
- [ ] 5.4 Add hagfish export --out /path/to/parquet subcommand
|
||||
- [ ] 5.5 Load config.json on startup (DuckDB path, bind address, data source URL, log file path)
|
||||
|
||||
### Phase 6: Bare Metal Deployment
|
||||
|
||||
- [ ] 6.1 Write systemd service unit file (hagfish.service) — ExecStart=/usr/local/bin/hagfish serve, restart policy
|
||||
- [ ] 6.2 Write systemd timer (hagfish-ingest.timer + hagfish-ingest.service) — monthly, runs hagfish ingest
|
||||
- [ ] 6.3 Taskfile: build (release, static), deploy (rsync binary + config + units, ssh reload)
|
||||
- [ ] 6.4 README with ELI5 Technology Choices section (why DuckDB, why Rust, why embedded static assets)
|
||||
Reference in New Issue
Block a user