finance-duck/OPERATIONS.txt

FINANCE DUCK — GO + REACT

Local build on NixOS
-------------------
nix-shell
npm --prefix web ci
npm --prefix web run build
go build -o bin/finance-duck ./cmd/finance-duck
./bin/finance-duck -data ./finance

Open http://localhost:8080. Create an account, then import its CSV statement
from Accounts and confirm the reviewed mapping. The application starts empty
except for expense/income fallback categories. Create your category tree, tags,
and merchants in the UI. Enable a merchant's default rule explicitly only when
its category/tags are reliable; leave it disabled for ambiguous merchants such
as Amazon.

Tests: go test ./...
The Go build embeds web/dist, so build React first. CGO and a C++ linker are
required by the native DuckDB driver. shell.nix supplies the toolchain. React
and CSS are served locally; there are no CDN requests or tracking scripts.

Private deployment
------------------
There is NO application login. Default bind: 127.0.0.1:8080.
For a VPN/reverse-proxy hostname, use an exact browser origin:
  ./bin/finance-duck -data /srv/finance -listen 127.0.0.1:8080 \
    -public-url https://finance.example.internal
Preserve the public Host header at the reverse proxy. The application rejects
other Hosts and cross-origin writes. Configure a long proxy request timeout
for bulk AI previews. Do not expose the reverse proxy to the public Internet.
A VPN-address bind can be used instead, with the corresponding -public-url.

Docker: docker compose up --build -d
Compose publishes only 127.0.0.1:8080. Set FINANCE_PUBLIC_URL when proxying it.
The container is unprivileged with a read-only root and a persistent named
volume at /data. A host bind mount must be writable by UID/GID 10001.
Never change the port mapping to public 0.0.0.0 without VPN/firewall isolation.

Native NixOS: the flake exports nixosModules.default. Import it into your host,
then set services.finance-duck.enable=true and publicURL to the exact browser
origin. Deploy with your normal nixos-rebuild switch. The module runs ONE
systemd service; React is embedded in the Go binary, not a separate service.
Defaults: listen 127.0.0.1:8080, service user finance-duck, persistent data at
/var/lib/finance-duck (0700). It does not configure networking or Tailscale.
Optional services.finance-duck.environmentFile names an existing runtime file
read by systemd for optional provider startup fallbacks. Both Enable Banking
and OpenRouter can be configured directly in Settings without that file.
Keep private keys outside the Nix store and readable only by the service.
See README's native NixOS section for optional environment-file permissions.

Deployed host: bender@100.87.224.3
--------------------------------
URL: https://nixos.taile9e6d9.ts.net:8444 (Tailscale only, no application login).
The backend binds 127.0.0.1:8080. Existing Funnel on 443 and OpenClaw on 8443
are separate. Never enable Funnel for Finance Duck.

Update from your workstation:
  ssh -t bender@100.87.224.3 'sudo finance-duck-update'
Or on the host:
  sudo finance-duck-update
  sudo finance-duck-update --no-pull   # deploy existing checkout edits

The updater serializes updates, pulls ~/projects/finance-duck as bender with
git pull --ff-only, creates a Git-filtered source snapshot, updates ONLY the
finance-duck input in /etc/nixos/flake.lock, runs nixos-rebuild switch, then
checks /api/health with the correct Host. The app has its own pinned Nixpkgs;
the host's Nixpkgs and NixVirt pins stay unchanged. A build failure leaves the
running service untouched; an activation/health failure exits nonzero.

Application and deployment source are tracked in the project repository.
Commit and push changes before running the normal updater. New local source
files must be tracked with git add before a --no-pull deployment. Pull refuses
to overwrite conflicting local edits; resolve them before retrying. No
automatic stashing, resetting, committing or pushing occurs. Keep credentials
and live finance data out of Git.

Host files:
  /etc/nixos/flake.nix                   application input and module import
  /etc/nixos/modules/finance-duck.nix    service, tailnet proxy, update command
  /var/lib/finance-duck-source           replaceable Git-filtered source snapshot
  /var/lib/finance-duck                  persistent data, finance-duck:finance-duck 0700
The source snapshot excludes .git, ignored files and untracked files. It is
not a data backup. The initial deployment starts empty without provider
credentials; it does not copy the workstation's finance/ directory.

Inspect on the host:
  systemctl status finance-duck finance-duck-tailscale-serve
  sudo journalctl -u finance-duck -f
  tailscale serve status
  curl -f https://nixos.taile9e6d9.ts.net:8444/api/health

Before significant upgrades, stop the service and back up the entire data
directory to a protected location (including hidden recovery files). Restart
after the backup, and protect any separate credential files as well:
  sudo systemctl stop finance-duck
  sudo install -d -m 0700 /var/backups/finance-duck
  sudo sh -c 'umask 077; tar -C /var/lib -czf /var/backups/finance-duck/data-$(date +%Y%m%dT%H%M%S).tar.gz finance-duck'
  sudo systemctl start finance-duck
Copy backups to secure off-host storage; a backup on this disk cannot protect
against disk loss.

Binary/system rollback:
  sudo nixos-rebuild switch --rollback
This switches the previous NixOS generation, NOT financial data or config
source. The generation before the first deployment has no Finance Duck service.
For a lasting rollback, restore the intended source/config/lock before the
next update. Do not remove or roll back live data blindly.

OpenRouter
----------
Open Settings -> OpenRouter credentials, paste the API key, and Save key.
Choose the exact provider/model identifier in Classification preferences and
save preferences. Replace key rotates it; Remove key disables AI. No SSH,
Nix rebuild or restart is needed. Changes affect future classification
requests; in-flight requests retain their original key.

"Classify newly imported transactions with AI" in Classification preferences
controls whether imports contact the provider at all. It applies to CSV imports
and bank synchronization, defaults to on, and is stored as classify_on_import
in config.toml. With it off, no import makes a provider request: enabled
merchant-default rules still classify, an alias match still attaches its
merchant, and everything else arrives on the editable fallback with no
provenance error. AI classification -> Analyse is unaffected by this preference.

The key is stored as private 0600 plaintext in state/openrouter.json under
the data directory, never in config.toml or browser storage. API responses
never return the saved key. Backups of the data directory contain this secret.
Only allow trusted users to reach the application; there is no separate
administrator login for changing credentials.

OPENROUTER_API_KEY remains a startup fallback only when the saved credential
file is absent. A saved key overrides it. Removing the key in Settings saves
an explicit disable, which also overrides the environment after restart.
Malformed saved credentials fail startup rather than reverting to another key.

Configured means present, not verified with OpenRouter. A successful
AI classification -> Analyse request verifies access with the selected model.
The model and endpoint must support strict structured outputs and the
configured privacy routing. Every classification request sets:
  provider.data_collection = deny
  provider.zdr = true
  provider.require_parameters = true
No retry relaxes these requirements. OpenRouter must also have prompt logging
disabled in your account settings. The underlying provider processes prompts;
this is not local AI and cannot promise that a remote provider honors policy.

Amounts and currency are omitted by default. Include Amount in Settings is
explicit opt-in. Local account/provider IDs, known counterparty names, banking
identifiers and recognizable references are stripped; candidate identifiers
are per-request opaque tokens. Categories/tags and candidate merchant names
are deliberately sent as classification context. Free-form text can contain
unknown personal names, so automatic sanitization is not an anonymity guarantee.
Conservative redaction can reduce recognition quality. Inspect your descriptions
and do not configure an API key if no financial text may leave the server.

Classification failures do not discard imports: facts are committed first and
failed enrichment stays unclassified with an error visible in Transactions.
Classification requests use one transaction at a time, not batches. Known
merchant defaults can classify without any configured AI key.

Provider rate limits
--------------------
Classification and Enable Banking use separate in-memory request gates. AI
imports and previews, including previews choosing another model, share the AI
gate. HTTP 429 retries retain the original request and privacy/authentication
controls. Other HTTP failures are not automatically retried.

Every outbound attempt is paced, including successful calls: AI request starts
are at least 3 seconds apart; bank starts at least 1 second apart. These are
conservative application defaults, not universal provider quota guarantees.
On 429, exponential fallback begins at 15 seconds for AI and 30 seconds for
banking. Consecutive failures escalate across operations, capped at 15 minutes;
success resets failure escalation but retains learned spacing up to 30 seconds.
Retry-After seconds and HTTP dates can extend delays, never shorten them.
Each operation makes at most four attempts and spends at most two minutes in
automatic retry waits. Normal pacing waits and queueing honor cancellation.
Network attempt timeouts remain separate (bank: at most 30 seconds; AI:
45 seconds by default).

Long hints, exhausted attempts, and canceled retries retain the provider's
cooldown. New calls fail without contacting that provider before its retry
time. Local merchant rules still work. Unrepresentably large positive hints
disable automatic retries rather than overflow into an early request.
These gates live in the running client, not the persisted financial journal.
Retries cannot lift provider quotas; wait until the reported time. Existing
failed classifications require another Analyse preview and explicit Apply.

Bank GET requests may retry 429. Authorization and once-only session exchange
POST requests never replay automatically, but still establish a cooldown.
Session status checks use the authorized UID list and expiry, without repeated
account-details requests; full account metadata remains saved from Exchange.
An unavailable session reports its original safe rate error once, without
additional unavailable-account errors or a false reconnection requirement.
Failed-account cursors and the last successful complete sync remain unchanged.

Manual sync, history import and balance HTTP requests forward actual PSU
metadata to account-data GET requests, consistently across retries/pages.
Only IP, User-Agent and available browser Accept headers are forwarded; never
cookies, authorization headers, referrer URLs, or arbitrary supplied Psu-* fields.
Scheduled sync contexts have no PSU metadata. User metadata is request-local,
never saved on a shared bank client or copied into classification requests.

With a public origin configured, a loopback reverse proxy must append its
observed client IP to X-Forwarded-For. Only its rightmost appended address is
trusted; direct/non-loopback callers cannot override their peer IP. Missing or
malformed proxy addresses remain unknown, not invented. Tailscale Serve uses
this deployment arrangement; other proxies must preserve this trust contract.

For a confirmed background ASPSP_RATE_LIMIT_EXCEEDED, defer that account and
endpoint at least six hours (longer provider hints take precedence), rather
than replaying every few seconds. This background-only quota does not suppress
an eligible genuine user request or another account/endpoint. Short generic
platform limits still use the shared bank controller for both request modes.
Only the bounded, exact error code is inspected; private provider messages are
not returned or logged. Quota state is client-local, finite entries expire,
and restart does not lift the upstream bank's actual quota.
Source: https://enablebanking.com/docs/faq/

Enable Banking
--------------
Register your application and public certificate with Enable Banking. For
personal production access follow its linked-own-accounts registration rules:
  https://enablebanking.com/docs/api/linked-accounts/
Configure the allowed redirect URL to the exact externally reachable URL:
  https://finance.example.internal/api/banking/callback
The browser must be able to reach this callback through your VPN.

Open Settings -> Enable Banking credentials. Enter the application ID, upload
its matching private-key PEM file, and Save configuration. Settings supplies
the exact browser-origin callback URL to register. Keys must be a single
unencrypted RSA PKCS#1 or PKCS#8 PEM, >=2048 bits and <=32 KiB.
No SSH, environment file, Nix rebuild or restart is needed.

Save validates local credentials without contacting Enable Banking. It does
not register/activate the app or authorize an account. Configured means local
setup is present; complete registration above and authorize under Accounts.

Leave the key upload empty when editing the same application to keep its key.
Same-app key/callback rotation preserves sessions and cursors. Changing the
application ID requires a new key upload and reconnecting your banks. Remove
configuration confirms before disabling banking and clearing local connection
state. Canonical accounts/history remain; upstream bank consent is not revoked.
Each successful change invalidates pending authorization flows.

Credentials persist privately (0600) in state/enablebanking.json beneath the
data directory. The API never returns the private key; browser storage never
contains it. Session state is bound to the application configuration generation,
so stale sessions cannot be revived by a restart after app changes/removal.
Back up the entire data directory securely, not isolated credential/state files.

Optional environment startup fallback, used ONLY when no saved config exists:
  ENABLEBANKING_APP_ID=<registered application ID>
  ENABLEBANKING_KEY_FILE=/run/secrets/enablebanking.key
  ENABLEBANKING_REDIRECT_URL=https://finance.example.internal/api/banking/callback
The referenced key must be readable by the service. Environment changes need
a restart. UI configuration or explicit removal overrides all three variables;
invalid saved configuration fails startup rather than using another key.

Accounts shows the configured callback URL. Register it exactly with Enable
Banking. Settings saves the current browser callback URL. The redirect carries a
one-time code and state, not a reusable API key. Go verifies state, exchanges the
code for a session_id, and persists session details locally with mode 0600.

Accounts -> Connect: enter the exact Enable Banking institution name and
country code (DE for Germany), then choose History to import (months).
The default is 12; whole numbers from 1 to 120 are accepted. Authorize through
the bank to save this choice with the consent. A successful callback immediately
wakes the synchronization worker; Sync now is also available.
The dashboard shows, for example, \"ING needs reconnection\" when consent expires
or is revoked. Reconnect ING starts the same approval flow with that bank and
country already selected. The new consent replaces the old account bindings
without duplicating local accounts or their financial history. Transient provider
errors are displayed separately from expired consent.

Only booked transactions are persisted. Each new account initially requests the
selected number of calendar months (12 by default); the bank may provide less.
Daily sync deliberately overlaps each account's last successful sync by 14 days.
Per-account cursors prevent another account's recent sync from skipping a new
account's history. Reconnection preserves the saved history choice and existing
cursors. Changing the choice or reconnecting does not backfill already-synced
accounts. Older records can be imported using CSV. Older saved consents without
a history choice use 12 months for accounts that have no successful-sync cursor.
A failed provider call retains local data. Failures whose cause Finance Duck
determines locally are named: a bank rate limit with its retry time, an expired
consent, an HTTP status, an unreachable provider, or a response the journal
cannot use (for example a booked transaction without a booking date). Only an
unrecognized cause falls back to "transaction retrieval failed". Provider
response text is never shown.

While every failing bank has named its own retry time, the account is reported
as waiting, not broken: the dashboard says synchronization retries by itself,
the account card shows a rate-limit badge with that time, and the scheduler
sleeps until the deadline instead of spending hourly session checks on a
refusal it already knows about. Sync now still tries immediately. Any failure
without such a deadline keeps the hourly retry. Retry times are persisted with
the sync state in whole seconds and rendered in the browser's time zone.
Sync now can retry sooner. Balances are fetched
on demand, with exact amount/currency/type values, rather than inferred from an
incomplete historical journal.

Historical imports for connected accounts
-----------------------------------------
Accounts -> account card -> Import older history opens a per-account dialog.
Choose Months back (12 by default, whole numbers 1-120) and confirm. The request
covers the selected past calendar months through today, subject to bank limits.
Only the chosen account is fetched. Existing transactions are deduplicated, and
new records follow normal transfer matching and classification. The dialog shows
the imported count, including zero for a repeated range with no new records.
Normal sync cursors, last-sync time and saved initial-history choices are not
changed. Inactive connected accounts may import history manually. Expired or
revoked bank authorization must be reconnected first. On failure, the UI refreshes
the journal to reveal any records already committed; it does not retry the import.

CSV import and identity
-----------------------
Import is a two-step flow: upload prepares a mapping and a preview, and nothing
reaches the journal until it is confirmed. Cancel, a browser reload, a failed
parse or a journal change between preview and confirmation all import nothing.
A prepared statement is held in memory only, expires after one hour, and at most
five may await confirmation. Uploads are limited to 2 MiB.

Recognized locally, without AI: N26 (English/German, old Date/Datum and new
Booking Date/Buchungsdatum schemas), ING Umsatzanzeige (Buchung,
Wertstellungsdatum, Auftraggeber/Empfaenger, Buchungstext, Verwendungszweck,
Betrag, Waehrung, below its metadata preamble), and Kontist's documented
transaction vocabulary (Payment Date/Buchungsdatum, Name, Amount/Betrag,
Purpose/Verwendungszweck). Accepted everywhere: comma/semicolon/tab separators,
UTF-8 with or without BOM, Windows-1252, CRLF, quoted multiline descriptions,
ISO and German dates, decimal point and comma, and thousands groups of exactly
three digits. Kontist date and decimal conventions are inferred from the file's
own first populated values; slash dates are read as month/day/year unless the
first component exceeds twelve. Foreign original amounts, exchange rates,
balances and categories are never used as account money. Currency comes from a
currency column, else an "Amount (EUR)"-style header, else the account; a
conflict with the account currency fails the whole statement. Use the original
export rather than spreadsheet-reformatted dates and numbers. Currency is
preserved but never converted or summed across currencies.

Any other layout requires a saved OpenRouter key and model: the columns are
mapped by the model, not the records. The request contains the delimiter, the
column names and up to four sample rows in which every letter is replaced by x
and every digit by 0, so descriptions, counterparties, references, IBANs and
amounts are not sent. The proposal is untrusted: each column must name a
supplied header exactly, money must come from one signed column or one
debit/credit pair, the date and decimal conventions must be from the supported
list, and no column may serve two fields. AI-mapped facts are recorded with
source "csv" and carry no transaction reference, because a repeating SEPA
mandate reference must never become a transaction identity. ING facts likewise
carry no reference. Review the previewed dates, amount signs and currency before
confirming; a wrong mapping is visible there, not after import.

Import sources: n26_csv, ing_csv, kontist_csv, csv (AI-mapped), enablebanking.

Stable provider entry references are scoped by account, source and debit/credit
direction: a debit and credit can share a reference without being collapsed.
Conflicting booking dates, amounts or currencies within one direction still
fail closed. Existing journal IDs and raw references are retained.
Enable Banking transaction_id is NOT guaranteed stable and is not used as the
primary identity. Fallback fingerprints retain identical-record occurrence
counts: two identical rows remain two transactions, and repeat imports do not
add two more. Without stable
IDs, identical records from separately truncated exports are intrinsically
ambiguous. Import consistent overlapping/full exports. Uncertain cross-source
collisions are rejected rather than silently double counted; retain the error
and reconcile the input locally before retrying. Facts are never silently
replaced when upstream descriptions or amounts change for an existing identity.

Transfers use reciprocal records from different owned accounts, equal/opposite
exact amounts and matching currency, with own-IBAN evidence and unambiguous
matching. Ambiguous pairs are not guessed. The linked records remain separate
immutable facts; analytical double-entry postings balance and transfers do not
count as income/spending. Populate local account IBANs to support recognition.

Canonical files and recovery
----------------------------
finance/
  config.toml                 model/amount/import opt-in only, no API keys
  accounts.finance
  categories.finance
  tags.finance
  merchants.finance
  journal/YYYY/YYYY-MM.finance
  state/sync-state.json       sensitive local consent/session metadata
  state/openrouter.json       sensitive UI-managed OpenRouter key or explicit disable
  state/enablebanking.json    sensitive UI-managed banking key/configuration or disable
  cache/finance.duckdb        disposable analytical projection

The custom grammar is deliberately small:
  category {
    id: "cat_example"
    name: "Groceries"
    parent_id: "cat_expenses"
    kind: "expense"
  }
A transaction block has facts: {...} and enrichment: {...} JSON-valued fields.
Financial amounts are quoted decimal strings, never binary floating point.
Up to four fractional digits are supported; arithmetic uses exact ten-thousandths
with explicit overflow checks. DuckDB stores DECIMAL(24,4).

Each block starts with account/category/tag/merchant/transaction and '{' on its
own line; fields use name: JSON. Strings use JSON escaping (including \n for
multiline descriptions). JSON values may span lines. Blank lines and full-line
# or // comments are accepted between fields/blocks. Unknown fields, duplicate
keys, malformed records, invalid references and taxonomy cycles are rejected.
The grammar is version-one strict: extension/split fields are not accepted yet.
Future format extensions require an explicit parser migration.

Stable category IDs survive renaming and moving; assigned categories must remain
leaves. Built-in roots and fallback leaves are protected. Move assigned records
to another leaf before adding children to their former category. Category
merges migrate referenced transactions/defaults; tag merges deduplicate links;
tag deletion removes all affected links after UI confirmation. Merchant merging
migrates transactions and retains source names/aliases on the target merchant.

Unchanged blocks and comments retain their text. UI enrichment edits do not
rewrite imported facts. File hashes detect external edits; invalid files stop
loading/indexing with a file/line error, not a partially refreshed dashboard.
A process lock prevents multiple app writers; use one instance per finance dir.
Revision conflicts require refreshing/re-previewing, not blind overwriting.

Multi-file writes are recoverable and logically atomic within the application.
Do not run external writers during a commit; external tools do not participate
in the app's process lock. Stop the service for manual bulk edits or backups.
If a write is interrupted, keep all state files and restart for recovery before
editing the journal manually. Keep backups of the entire canonical directory,
including hidden/state recovery files, plus the separately stored secrets.
The cache can be excluded. Avoid exposing any financial directory through a
static file server, Git public remote, or unencrypted shared backup.

Rebuild from text:
  ./bin/finance-duck -data ./finance -rebuild
Stop the running app before using that command (single writer lock), or use
Settings -> Rebuild index while it runs. A broken/deleted DuckDB file can be
removed while stopped and regenerated; it never contains the only copy of
financial records. When an index rebuild fails, UI mutations still preserve
canonical data and the index error is surfaced rather than serving stale totals.

Reclassification
----------------
AI / Classification: choose dates, model and independent Merchant/Category/Tags
fields. Analyse produces a read-only preview. Apply all/selected writes all
selected changes in one canonical commit; financial facts never change. A
manual edit, external journal change or taxonomy change invalidates old previews.
Previews are kept in memory for up to one hour and disappear on restart. Cancel
writes nothing. Transfers are skipped, and unselected fields are preserved.
Failed rows remain unchanged and are listed separately from proposed changes.

Boundaries and verification
---------------------------
There are no splits, budgets, investments, tax/invoice/receipt processing,
login/multi-user support, arbitrary SQL or natural-language query execution.
Natural-language query DSL and Sankey exploration remain explicitly later work.
There is no browser-to-bank credential handling or payment initiation.

Automated tests exercise deterministic financial invariants and mock remote
provider HTTP behavior. They do not replace testing real consent renewals and
real booked transaction samples for your banks. No real user credentials or
financial history are included in the repository. Test fixtures are synthetic.

API documentation sources:
  https://enablebanking.com/docs/api/reference/
  https://openrouter.ai/docs/guides/features/structured-outputs
  https://openrouter.ai/docs/guides/features/zdr
  https://openrouter.ai/docs/guides/routing/provider-selection