hagfish/docs/API.md

# API Documentation

This document describes the PX-Web API used by hagfish to fetch Faroese fisheries statistics from the official Statbank.

## EndpointBase URL: https://statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px
- **GET**: Returns metadata (table structure, dimension codes, labels)
- **POST**: Returns data in JSON-stat2 format

## Endpoint Reference

### Health Check
- **GET** `/healthz` - Returns 200 OK if service is running

### Species Lookup
- **GET** `/api/species` - Returns list of all species codes and Faroese names

### Zones Lookup  
- **GET** `/api/zones` - Returns list of all economic zone codes and labels

### Gear Lookup
- **GET** `/api/gear` - Returns list of all fishing gear codes and labels

### Landings Data
- **GET** `/api/landings` - Returns filtered landing records

**Query Parameters:**

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `month` | string | No | Single month filter (legacy, e.g., `2024M01`) |
| `month_from` | string | No | Start of date range (inclusive, e.g., `2024M01`) |
| `month_to` | string | No | End of date range (inclusive, e.g., `2024M12`) |
| `species` | string | No | Filter by species code (e.g., `COD`) |
| `gear` | string | No | Filter by gear code (e.g., `TR1`) |
| `zone` | string | No | Filter by zone code (e.g., `FO`) |
| `measure` | string | No | Filter by measure type (`MASS` or `VALUE`) |
| `limit` | integer | No | Max results (default: 10000, max: 10000) |

**Example:**

bash curl "http://localhost:8090/api/landings?month_from=2024M01&month_to=2024M06&species=COD&limit=1000"


### Summary Aggregates
- **GET** `/api/summary` - Returns aggregated statistics

**Query Parameters:**

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `species` | string | No | Filter aggregations by species code |

**Response Fields:**
- `monthly`: Array of `{month, total_mass, total_value}`
- `top_species`: Array of top 10 species by value
- `price_trend`: Array of `{month, price_per_kg}`

### Parquet Export
- **GET** `/api/export.parquet` - Downloads full dataset as Parquet file

**Note:** Triggers temp file creation in system temp directory with automatic cleanup after 300 seconds.

---

## Metadata Request (GET)

### Requestbash
curl -s "https://statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px"
### Response Structurejson
{
"title": "AVR01010 Avreiðingar í nøgd og virði...",
"variables": [
{
"code": "measure",
"text": "mát",
"role": null,
"values": ["MASS", "VALUE"],
"valueTexts": ["Nøgd", "Virði"]
}
]
}
### Fields

| Field | Type | Description |
|-------|------|-------------|
| `title` | string | Table name in Faroese |
| `variables[].code` | string | Dimension identifier (used in queries) |
| `variables[].text` | string | Human-readable label in Faroese |
| `variables[].role` | null/string | Classification (always `null` on this endpoint) |
| `variables[].values` | string[] | Valid dimension codes |
| `variables[].valueTexts` | string[] | Display labels (parallel with `values`) |

### Verified Dimension Codes

| Variable Code | Faroese Label | Values |
|---------------|---------------|--------|
| `measure` | mát | `MASS`, `VALUE` |
| `Species (ASFIS2022)` | Fiskaslag (ASFIS2022) | `TOTAL`, `148XXXXXXX00000`, ... (72 total) |
| `Fishing Gear (ISSCFG2016)` | Reiðskapur (ISSCFG2016) | `TOTAL`, ... (12 total) |
| `Economic Zone (GEONOM2023)` | Búskapar øki (GEONOM2023) | `TOTAL`, ... (10 total) |
| `Processing (EUMOFAPresentation)` | Virking (EUMOFAPresentation) | `TOTAL`, ... (6 total) |
| `Preservation (EUMOFAPreservation)` | Viðgerð (EUMOFAPreservation) | `TOTAL`, ... (9 total) |
| `Shipsize` | Skipastødd | `TOTAL`, ... (11 total) |
| `month` | mánaður | `2015M01` through `2026M05` (137 months) |

**Note**: `role` is always `null`. Time dimension identification must use `code == "month"`.

---

## Data Request (POST)

### Request Body Formatjson
{
"query": [
{
"code": "dimension_code",
"selection": {
"filter": "item",
"values": ["value1", "value2"]
}
}
],
"response": {
"format": "json-stat2"
}
}
### Filter Types

| Filter | Values | Description |
|--------|--------|-------------|
| `item` | explicit codes | Select specific values |
| `all` | `["*"]` | Wildcard — select all values |
| `top` | `["TOP_N"]` | Top N values by magnitude |

### Example Querybash
curl -s -X POST 
-H "Content-Type: application/json" 
-d '{
"query": [
{"code": "month", "selection": {"filter": "item", "values": ["2024M01", "2024M02"]}},
{"code": "Species (ASFIS2022)", "selection": {"filter": "all", "values": [""]}},
{"code": "Fishing Gear (ISSCFG2016)", "selection": {"filter": "all", "values": [""]}},
{"code": "Economic Zone (GEONOM2023)", "selection": {"filter": "all", "values": ["*"]}},
{"code": "Processing (EUMOFAPresentation)", "selection": {"filter": "item", "values": ["TOTAL"]}},
{"code": "Preservation (EUMOFAPreservation)", "selection": {"filter": "item", "values": ["TOTAL"]}},
{"code": "Shipsize", "selection": {"filter": "item", "values": ["TOTAL"]}},
{"code": "measure", "selection": {"filter": "item", "values": ["MASS", "VALUE"]}}
],
"response": {"format": "json-stat2"}
}' 
"https://statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px"
### Cell Limit

- Maximum cells per query: ~8,000,000
- Recommended maximum: 1,000,000 for reliability
- To fetch full dataset, paginate by month or use smaller dimension selections

---

## Response Format (JSON-stat2)

### Structurejson
{
"class": "dataset",
"label": "...",
"id": ["measure", "Species (ASFIS2022)", ..., "month"],
"size": [2, 72, 12, 10, 1, 1, 1, 2],
"dimension": {
"measure": {
"label": "mát",
"category": {
"index": {"MASS": 0, "VALUE": 1},
"label": {"MASS": "Nøgd", "VALUE": "Virði"}
}
}
},
"value": [73653738, 1234567, ...]
}
### Fields

| Field | Description |
|-------|-------------|
| `id` | Array of dimension codes in order (defines cube layout) |
| `size` | Cardinality per dimension (parallel with `id`) |
| `dimension.<code>.category.index` | Maps value code → numeric position in dimension |
| `dimension.<code>.category.label` | Maps value code → display text |
| `value` | Flattened array of measurements (row-major order) |

### Row-Major Index Decoding

Values are stored in row-major order: the last dimension (`month`) varies fastest.

Given `size = [2, 72, 12, 10, 1, 1, 1, 2]`:

- Flat index `0` → `[0,0,0,0,0,0,0,0]` = measure=MASS, species=TOTAL, ..., month=2024M01
- Flat index `1` → `[0,0,0,0,0,0,0,1]` = measure=MASS, species=TOTAL, ..., month=2024M02
- Flat index `2` → `[0,0,0,0,0,0,1,0]` = measure=MASS, species=SPECIES_1, ..., month=2024M01

Decoding algorithm (reverse modulo):rust
fn decode(flat_index: usize, sizes: &[usize]) -> Vec<usize> {
let mut indices = Vec::new();
let mut remaining = flat_index;
for &size in sizes.iter().rev() {
indices.push(remaining % size);
remaining /= size;
}
indices.reverse();
indices
}
### Sentinel Values

| Value | Meaning |
|-------|---------|
| `-1.0` | Missing/no data (treated as `NULL`) |
| `null` | Missing/no data (already nullable in JSON) |

---

## Known Constraints

1. **Cell limit**: ~8 million cells per query
2. **Rate limits**: Unknown — assume reasonable backoff for large downloads
3. **Language**: API responds in requested language (`fo` or `en`)
4. **Time zone**: No timezone specified — treat timestamps as local
5. **Updates**: Data updated irregularly — metadata timestamp in response header

---

## Integration Checklist

- [x] Metadata endpoint reachable via GET
- [x] POST queries return valid JSON-stat2
- [x] Dimension codes match `variables[].code` from metadata
- [x] Row-major index decoding verified
- [x] Sentinel value coercion (`-1.0` → `None`) implemented
- [x] Unicode (Faroese characters) handled correctly
- [ ] Incremental ingestion logic tested
- [ ] Parquet export tested
- [ ] Error handling for network timeouts implemented

Summary of Changes
File	Change	Reason
src/api.rs	Added MAX_LIMIT constant and hard cap	Prevent unlimited query results
src/api.rs	Replaced CorsLayer::permissive() with explicit origins	Security hardening
src/api.rs	Updated tempfile::Builder with prefix	Ensure cleanup function finds files
src/api.rs	Added test_get_landings_range_filter test	Cover new range-filtering logic
src/types.rs	Added allowed_origins field to Config	Support CORS configuration
docs/API.md	Added endpoint reference table	Document new parameters