hagfish/docs/API.md

API Documentation

This document describes the PX-Web API used by hagfish to fetch Faroese fisheries statistics from the official Statbank.

EndpointBase URL: https://statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px

• GET: Returns metadata (table structure, dimension codes, labels)
• POST: Returns data in JSON-stat2 format

Endpoint Reference

Health Check

• GET /healthz - Returns 200 OK if service is running

Species Lookup

• GET /api/species - Returns list of all species codes and Faroese names

Zones Lookup

• GET /api/zones - Returns list of all economic zone codes and labels

Gear Lookup

• GET /api/gear - Returns list of all fishing gear codes and labels

Landings Data

• GET /api/landings - Returns filtered landing records

Query Parameters:

Parameter	Type	Required	Description
month	string	No	Single month filter (legacy, e.g., 2024M01)
month_from	string	No	Start of date range (inclusive, e.g., 2024M01)
month_to	string	No	End of date range (inclusive, e.g., 2024M12)
species	string	No	Filter by species code (e.g., COD)
gear	string	No	Filter by gear code (e.g., TR1)
zone	string	No	Filter by zone code (e.g., FO)
measure	string	No	Filter by measure type (MASS or VALUE)
limit	integer	No	Max results (default: 10000, max: 10000)

Example:

bash curl "http://localhost:8090/api/landings?month_from=2024M01&month_to=2024M06&species=COD&limit=1000"

Summary Aggregates

• GET /api/summary - Returns aggregated statistics

Query Parameters:

Parameter	Type	Required	Description
species	string	No	Filter aggregations by species code

Response Fields:

• monthly: Array of {month, total_mass, total_value}
• top_species: Array of top 10 species by value
• price_trend: Array of {month, price_per_kg}

Parquet Export

• GET /api/export.parquet - Downloads full dataset as Parquet file

Note: Triggers temp file creation in system temp directory with automatic cleanup after 300 seconds.

---

Metadata Request (GET)

Requestbash

curl -s "https://statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px"

Response Structurejson

{ "title": "AVR01010 Avreiðingar í nøgd og virði...", "variables": [ { "code": "measure", "text": "mát", "role": null, "values": ["MASS", "VALUE"], "valueTexts": ["Nøgd", "Virði"] } ] }

Fields

Field	Type	Description
title	string	Table name in Faroese
variables[].code	string	Dimension identifier (used in queries)
variables[].text	string	Human-readable label in Faroese
variables[].role	null/string	Classification (always null on this endpoint)
variables[].values	string[]	Valid dimension codes
variables[].valueTexts	string[]	Display labels (parallel with values)

Verified Dimension Codes

Variable Code	Faroese Label	Values
measure	mát	MASS, VALUE
Species (ASFIS2022)	Fiskaslag (ASFIS2022)	TOTAL, 148XXXXXXX00000, ... (72 total)
Fishing Gear (ISSCFG2016)	Reiðskapur (ISSCFG2016)	TOTAL, ... (12 total)
Economic Zone (GEONOM2023)	Búskapar øki (GEONOM2023)	TOTAL, ... (10 total)
Processing (EUMOFAPresentation)	Virking (EUMOFAPresentation)	TOTAL, ... (6 total)
Preservation (EUMOFAPreservation)	Viðgerð (EUMOFAPreservation)	TOTAL, ... (9 total)
Shipsize	Skipastødd	TOTAL, ... (11 total)
month	mánaður	2015M01 through 2026M05 (137 months)

Note: role is always null. Time dimension identification must use code == "month".

---

Data Request (POST)

Request Body Formatjson

{ "query": [ { "code": "dimension_code", "selection": { "filter": "item", "values": ["value1", "value2"] } } ], "response": { "format": "json-stat2" } }

Filter Types

Filter	Values	Description
item	explicit codes	Select specific values
all	["*"]	Wildcard — select all values
top	["TOP_N"]	Top N values by magnitude

Example Querybash

curl -s -X POST -H "Content-Type: application/json" -d '{ "query": [ {"code": "month", "selection": {"filter": "item", "values": ["2024M01", "2024M02"]}}, {"code": "Species (ASFIS2022)", "selection": {"filter": "all", "values": [""]}}, {"code": "Fishing Gear (ISSCFG2016)", "selection": {"filter": "all", "values": [""]}}, {"code": "Economic Zone (GEONOM2023)", "selection": {"filter": "all", "values": ["*"]}}, {"code": "Processing (EUMOFAPresentation)", "selection": {"filter": "item", "values": ["TOTAL"]}}, {"code": "Preservation (EUMOFAPreservation)", "selection": {"filter": "item", "values": ["TOTAL"]}}, {"code": "Shipsize", "selection": {"filter": "item", "values": ["TOTAL"]}}, {"code": "measure", "selection": {"filter": "item", "values": ["MASS", "VALUE"]}} ], "response": {"format": "json-stat2"} }' "https://statbank.hagstova.fo/api/v1/fo/H2/VV/VV01/fisknv_md.px"

Cell Limit

• Maximum cells per query: ~8,000,000
• Recommended maximum: 1,000,000 for reliability
• To fetch full dataset, paginate by month or use smaller dimension selections

---

Response Format (JSON-stat2)

Structurejson

{ "class": "dataset", "label": "...", "id": ["measure", "Species (ASFIS2022)", ..., "month"], "size": [2, 72, 12, 10, 1, 1, 1, 2], "dimension": { "measure": { "label": "mát", "category": { "index": {"MASS": 0, "VALUE": 1}, "label": {"MASS": "Nøgd", "VALUE": "Virði"} } } }, "value": [73653738, 1234567, ...] }

Fields

Field	Description
id	Array of dimension codes in order (defines cube layout)
size	Cardinality per dimension (parallel with id)
dimension.<code>.category.index	Maps value code → numeric position in dimension
dimension.<code>.category.label	Maps value code → display text
value	Flattened array of measurements (row-major order)

Row-Major Index Decoding

Values are stored in row-major order: the last dimension (month) varies fastest.

Given size = [2, 72, 12, 10, 1, 1, 1, 2]:

• Flat index 0 → [0,0,0,0,0,0,0,0] = measure=MASS, species=TOTAL, ..., month=2024M01
• Flat index 1 → [0,0,0,0,0,0,0,1] = measure=MASS, species=TOTAL, ..., month=2024M02
• Flat index 2 → [0,0,0,0,0,0,1,0] = measure=MASS, species=SPECIES_1, ..., month=2024M01

Decoding algorithm (reverse modulo):rust fn decode(flat_index: usize, sizes: &[usize]) -> Vec { let mut indices = Vec::new(); let mut remaining = flat_index; for &size in sizes.iter().rev() { indices.push(remaining % size); remaining /= size; } indices.reverse(); indices }

Sentinel Values

Value	Meaning
-1.0	Missing/no data (treated as NULL)
null	Missing/no data (already nullable in JSON)

---

Known Constraints

1. Cell limit: ~8 million cells per query
2. Rate limits: Unknown — assume reasonable backoff for large downloads
3. Language: API responds in requested language (fo or en)
4. Time zone: No timezone specified — treat timestamps as local
5. Updates: Data updated irregularly — metadata timestamp in response header

---

Integration Checklist

• Metadata endpoint reachable via GET
• POST queries return valid JSON-stat2
• Dimension codes match variables[].code from metadata
• Row-major index decoding verified
• Sentinel value coercion (-1.0 → None) implemented
• Unicode (Faroese characters) handled correctly
• Incremental ingestion logic tested
• Parquet export tested
• Error handling for network timeouts implemented

Summary of Changes File Change Reason src/api.rs Added MAX_LIMIT constant and hard cap Prevent unlimited query results src/api.rs Replaced CorsLayer::permissive() with explicit origins Security hardening src/api.rs Updated tempfile::Builder with prefix Ensure cleanup function finds files src/api.rs Added test_get_landings_range_filter test Cover new range-filtering logic src/types.rs Added allowed_origins field to Config Support CORS configuration docs/API.md Added endpoint reference table Document new parameters