lars/finance-duck
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0fd3c5c0dc8ac198caec9efbb3afc818805fc489
finance-duck/OPERATIONS.txt
T
Lars Nolden 0fd3c5c0dc Filter transactions by classification status 
The Transactions toolbar gains a status filter over classification
provenance — manual, AI, merchant rule, transfer match, unclassified —
alongside the existing Needs review toggle. The Source column now
renders the same human labels the filter options use instead of raw
provenance keys, so "openrouter" reads as AI and both fallback shapes
read as Unclassified.
2026-09-14 12:39:11 +02:00

797 lines
47 KiB
Plaintext
Raw Blame History

FINANCE DUCK — GO + REACT
Local build on NixOS
-------------------
nix-shell
npm --prefix web ci
npm --prefix web run build
go build -o bin/finance-duck ./cmd/finance-duck
./bin/finance-duck -data ./finance
Open http://localhost:8080. Create an account, then import its CSV statement
from Accounts and confirm the reviewed mapping. The application starts empty
except for expense/income fallback categories. Create your category tree, tags,
and merchants in the UI. Enable a merchant's default rule explicitly only when
its category/tags are reliable; leave it disabled for ambiguous merchants such
as Amazon. Category and tag pickers create in place: type an unknown name in
a category picker and choose "Create … in …" (a bare name lands under the
kind's root; "Parent / Name" targets that parent), or type a new tag next to
the tag checkboxes. Assignment pickers offer leaf categories only, matching
what the server accepts; a name that already exists is selected, never
duplicated.
Tests: go test ./...
The Go build embeds web/dist, so build React first. CGO and a C++ linker are
required by the native DuckDB driver. shell.nix supplies the toolchain. React
and CSS are served locally; there are no CDN requests or tracking scripts.
Private deployment
------------------
There is NO application login. Default bind: 127.0.0.1:8080.
For a VPN/reverse-proxy hostname, use an exact browser origin:
./bin/finance-duck -data /srv/finance -listen 127.0.0.1:8080 \
-public-url https://finance.example.internal
Preserve the public Host header at the reverse proxy. The application rejects
other Hosts and cross-origin writes. Configure a long proxy request timeout
for bulk AI previews. Do not expose the reverse proxy to the public Internet.
A VPN-address bind can be used instead, with the corresponding -public-url.
Docker: docker compose up --build -d
Compose publishes only 127.0.0.1:8080. Set FINANCE_PUBLIC_URL when proxying it.
The container is unprivileged with a read-only root and a persistent named
volume at /data. A host bind mount must be writable by UID/GID 10001.
Never change the port mapping to public 0.0.0.0 without VPN/firewall isolation.
Native NixOS: the flake exports nixosModules.default. Import it into your host,
then set services.finance-duck.enable=true and publicURL to the exact browser
origin. Deploy with your normal nixos-rebuild switch. The module runs ONE
systemd service; React is embedded in the Go binary, not a separate service.
Defaults: listen 127.0.0.1:8080, service user finance-duck, persistent data at
/var/lib/finance-duck (0700). It does not configure networking or Tailscale.
Optional services.finance-duck.environmentFile names an existing runtime file
read by systemd for optional provider startup fallbacks. Both Enable Banking
and OpenRouter can be configured directly in Settings without that file.
Keep private keys outside the Nix store and readable only by the service.
See README's native NixOS section for optional environment-file permissions.
Deployed host: bender@100.87.224.3
--------------------------------
URL: https://nixos.taile9e6d9.ts.net:8444 (Tailscale only, no application login).
The backend binds 127.0.0.1:8080. Existing Funnel on 443 and OpenClaw on 8443
are separate. Never enable Funnel for Finance Duck.
Update from your workstation:
ssh -t bender@100.87.224.3 'sudo finance-duck-update'
Or on the host:
sudo finance-duck-update
sudo finance-duck-update --no-pull # deploy existing checkout edits
The updater serializes updates, pulls ~/projects/finance-duck as bender with
git pull --ff-only, creates a Git-filtered source snapshot, updates ONLY the
finance-duck input in /etc/nixos/flake.lock, runs nixos-rebuild switch, then
checks /api/health with the correct Host. The app has its own pinned Nixpkgs;
the host's Nixpkgs and NixVirt pins stay unchanged. A build failure leaves the
running service untouched; an activation/health failure exits nonzero.
Application and deployment source are tracked in the project repository.
Commit and push changes before running the normal updater. New local source
files must be tracked with git add before a --no-pull deployment. Pull refuses
to overwrite conflicting local edits; resolve them before retrying. No
automatic stashing, resetting, committing or pushing occurs. Keep credentials
and live finance data out of Git.
Host files:
/etc/nixos/flake.nix application input and module import
/etc/nixos/modules/finance-duck.nix service, tailnet proxy, update command
/var/lib/finance-duck-source replaceable Git-filtered source snapshot
/var/lib/finance-duck persistent data, finance-duck:finance-duck 0700
The source snapshot excludes .git, ignored files and untracked files. It is
not a data backup. The initial deployment starts empty without provider
credentials; it does not copy the workstation's finance/ directory.
Inspect on the host:
systemctl status finance-duck finance-duck-tailscale-serve
sudo journalctl -u finance-duck -f
tailscale serve status
curl -f https://nixos.taile9e6d9.ts.net:8444/api/health
Before significant upgrades, stop the service and back up the entire data
directory to a protected location (including hidden recovery files). Restart
after the backup, and protect any separate credential files as well:
sudo systemctl stop finance-duck
sudo install -d -m 0700 /var/backups/finance-duck
sudo sh -c 'umask 077; tar -C /var/lib -czf /var/backups/finance-duck/data-$(date +%Y%m%dT%H%M%S).tar.gz finance-duck'
sudo systemctl start finance-duck
Copy backups to secure off-host storage; a backup on this disk cannot protect
against disk loss.
Binary/system rollback:
sudo nixos-rebuild switch --rollback
This switches the previous NixOS generation, NOT financial data or config
source. The generation before the first deployment has no Finance Duck service.
For a lasting rollback, restore the intended source/config/lock before the
next update. Do not remove or roll back live data blindly.
OpenRouter
----------
Open Settings -> OpenRouter credentials, paste the API key, and Save key.
Choose the exact provider/model identifier in Classification preferences and
save preferences. Replace key rotates it; Remove key disables AI. No SSH,
Nix rebuild or restart is needed. Changes affect future classification
requests; in-flight requests retain their original key.
"Classify newly imported transactions with AI" in Classification preferences
controls whether imports contact the provider at all. It applies to CSV imports
and bank synchronization, defaults to on, and is stored as classify_on_import
in config.toml. With it off, no import makes a provider request: enabled
merchant-default rules still classify, an alias match still attaches its
merchant, and everything else arrives on the editable fallback with no
provenance error. AI classification -> Analyse is unaffected by this preference.
The key is stored as private 0600 plaintext in state/openrouter.json under
the data directory, never in config.toml or browser storage. API responses
never return the saved key. Backups of the data directory contain this secret.
Only allow trusted users to reach the application; there is no separate
administrator login for changing credentials.
OPENROUTER_API_KEY remains a startup fallback only when the saved credential
file is absent. A saved key overrides it. Removing the key in Settings saves
an explicit disable, which also overrides the environment after restart.
Malformed saved credentials fail startup rather than reverting to another key.
Configured means present, not verified with OpenRouter. A successful
AI classification -> Analyse request verifies access with the selected model.
The model and endpoint must support strict structured outputs and the
configured privacy routing. Every classification request sets:
provider.data_collection = deny
provider.zdr = true
provider.require_parameters = true
No retry relaxes these requirements. OpenRouter must also have prompt logging
disabled in your account settings. The underlying provider processes prompts;
this is not local AI and cannot promise that a remote provider honors policy.
Each classification sends the transaction date, signed amount, currency,
merchant and counterparty text, account institution/currency, the complete
leaf-category registry for the transaction kind, all tags and all merchants.
Names, paths, hints and aliases remain available, but registry IDs use short
request-local references (c1, m1, t1), including merchant usual categories and
history. History includes only categories offered for that transaction kind.
Responses are mapped back to canonical IDs and validated locally; canonical
IDs are not accepted as alternative response references. This keeps the full
registry without the long-ID schema overhead that providers can reject.
Identifier-only redaction removes IBANs (with a
directly attached BIC), labeled BIC/SWIFT references, UUIDs, URLs/emails,
labeled payment or customer references, card fragments, long digit-bearing
tokens, the row's own IDs, account labels and configured private names. A
bare eight- or eleven-letter word is never treated as a BIC: that shape
matches ordinary payee names, and a bank code alone reveals no more than the
institution field already sent. Counterparty text is intentionally retained
unless it is in Private names; this is the accepted recognition trade-off,
not an anonymity guarantee. There is no Include Amount opt-in anymore. A
response records high, medium or low confidence. Imports never auto-apply a
low-confidence category: the row keeps the kind-specific unclassified
category with merchant and confidence recorded. Analyse previews show the
low-confidence suggestion unselected for review. Transactions exposes a
Needs review filter for low-confidence or fallback rows, and a
classification filter over how each row was classified: manually, by AI,
by a merchant rule, by transfer matching, or not at all. A model-proposed
merchant name is dropped (the row keeps its validated category and tags)
when it is identifier-shaped, longer than 100 characters, or contains
control or format code points such as bidirectional overrides and
zero-width characters, which could visually spoof the review UI; proposed
taxonomy names are rejected under the same hidden-rune rule.
Categories and tags have editable hints. Categories -> Propose taxonomy sends
up to 300 grouped, redacted transaction samples, then shows proposed
categories, tags and merchants with evidence. Every item is approved by hand;
applying a child also approves its proposed parents, mints IDs locally, and
checks the revision. Existing registry entries, journal facts, and unapproved
items remain unchanged.
Provider rate limits
--------------------
Classification and Enable Banking use separate in-memory request gates. AI
imports and previews, including previews choosing another model, share the AI
gate. HTTP 429 retries retain the original request and privacy/authentication
controls. Other HTTP failures are not automatically retried.
Every outbound attempt is paced, including successful calls: AI request starts
are at least 3 seconds apart; bank starts at least 1 second apart. These are
conservative application defaults, not universal provider quota guarantees.
On 429, exponential fallback begins at 15 seconds for AI and 30 seconds for
banking. Consecutive failures escalate across operations, capped at 15 minutes;
success resets failure escalation but retains learned spacing up to 30 seconds.
Retry-After seconds and HTTP dates can extend delays, never shorten them.
Each operation makes at most four attempts and spends at most two minutes in
automatic retry waits. Normal pacing waits and queueing honor cancellation.
Network attempt timeouts remain separate (bank: at most 30 seconds; AI:
45 seconds by default).
Long hints, exhausted attempts, and canceled retries retain the provider's
cooldown. New calls fail without contacting that provider before its retry
time. Local merchant rules still work. Unrepresentably large positive hints
disable automatic retries rather than overflow into an early request.
These gates live in the running client, not the persisted financial journal.
Retries cannot lift provider quotas; wait until the reported time. Existing
failed classifications require another Analyse preview and explicit Apply.
Bank GET requests may retry 429. Authorization and once-only session exchange
POST requests never replay automatically, but still establish a cooldown.
Session status checks use the authorized UID list and expiry, without repeated
account-details requests; full account metadata remains saved from Exchange.
An unavailable session reports its original safe rate error once, without
additional unavailable-account errors or a false reconnection requirement.
Failed-account cursors and the last successful complete sync remain unchanged.
Manual sync, history import and balance HTTP requests forward actual PSU
metadata to account-data GET requests, consistently across retries/pages.
Only IP, User-Agent and available browser Accept headers are forwarded; never
cookies, authorization headers, referrer URLs, or arbitrary supplied Psu-* fields.
Scheduled sync contexts have no PSU metadata. User metadata is request-local,
never saved on a shared bank client or copied into classification requests.
With a public origin configured, a loopback reverse proxy must append its
observed client IP to X-Forwarded-For. Only its rightmost appended address is
trusted; direct/non-loopback callers cannot override their peer IP. Missing or
malformed proxy addresses remain unknown, not invented. Tailscale Serve uses
this deployment arrangement; other proxies must preserve this trust contract.
For a confirmed background ASPSP_RATE_LIMIT_EXCEEDED, defer that account and
endpoint at least six hours (longer provider hints take precedence), rather
than replaying every few seconds. This background-only quota does not suppress
an eligible genuine user request or another account/endpoint. Short generic
platform limits still use the shared bank controller for both request modes.
Only the bounded, exact error code is inspected; private provider messages are
not returned or logged. Quota state is client-local, finite entries expire,
and restart does not lift the upstream bank's actual quota.
Source: https://enablebanking.com/docs/faq/
Enable Banking
--------------
Register your application and public certificate with Enable Banking. For
personal production access follow its linked-own-accounts registration rules:
https://enablebanking.com/docs/api/linked-accounts/
Configure the allowed redirect URL to the exact externally reachable URL:
https://finance.example.internal/api/banking/callback
The browser must be able to reach this callback through your VPN.
Open Settings -> Enable Banking credentials. Enter the application ID, upload
its matching private-key PEM file, and Save configuration. Settings supplies
the exact browser-origin callback URL to register. Keys must be a single
unencrypted RSA PKCS#1 or PKCS#8 PEM, >=2048 bits and <=32 KiB.
No SSH, environment file, Nix rebuild or restart is needed.
Save validates local credentials without contacting Enable Banking. It does
not register/activate the app or authorize an account. Configured means local
setup is present; complete registration above and authorize under Accounts.
Leave the key upload empty when editing the same application to keep its key.
Same-app key/callback rotation preserves sessions and cursors. Changing the
application ID requires a new key upload and reconnecting your banks. Remove
configuration confirms before disabling banking and clearing local connection
state. Canonical accounts/history remain; upstream bank consent is not revoked.
Each successful change invalidates pending authorization flows.
Credentials persist privately (0600) in state/enablebanking.json beneath the
data directory. The API never returns the private key; browser storage never
contains it. Session state is bound to the application configuration generation,
so stale sessions cannot be revived by a restart after app changes/removal.
Back up the entire data directory securely, not isolated credential/state files.
Optional environment startup fallback, used ONLY when no saved config exists:
ENABLEBANKING_APP_ID=<registered application ID>
ENABLEBANKING_KEY_FILE=/run/secrets/enablebanking.key
ENABLEBANKING_REDIRECT_URL=https://finance.example.internal/api/banking/callback
The referenced key must be readable by the service. Environment changes need
a restart. UI configuration or explicit removal overrides all three variables;
invalid saved configuration fails startup rather than using another key.
Accounts shows the configured callback URL. Register it exactly with Enable
Banking. Settings saves the current browser callback URL. The redirect carries a
one-time code and state, not a reusable API key. Go verifies state, exchanges the
code for a session_id, and persists session details locally with mode 0600.
Accounts -> Connect: enter the exact Enable Banking institution name and
country code (DE for Germany), then choose History to import (months).
The default is 12; whole numbers from 1 to 120 are accepted. Authorize through
the bank to save this choice with the consent. A successful callback immediately
wakes the synchronization worker; Sync now is also available.
The dashboard shows, for example, \"ING needs reconnection\" when consent expires
or is revoked. Reconnect ING starts the same approval flow with that bank and
country already selected. The new consent replaces the old account bindings
without duplicating local accounts or their financial history. Transient provider
errors are displayed separately from expired consent.
Only booked transactions are persisted. Each new account initially requests the
selected number of calendar months (12 by default); the bank may provide less.
Automatic sync runs twice a day, every 12 hours from the last successful run,
and deliberately overlaps each account's last successful sync by 14 days.
Per-account cursors prevent another account's recent sync from skipping a new
account's history. Reconnection preserves the saved history choice and existing
cursors. Changing the choice or reconnecting does not backfill already-synced
accounts. Older records can be imported using CSV. Older saved consents without
a history choice use 12 months for accounts that have no successful-sync cursor.
A failed provider call retains local data. Failures whose cause Finance Duck
determines locally are named: a bank rate limit with its retry time, an expired
consent, an HTTP status, an unreachable provider, or a response the journal
cannot use (for example a booked transaction without a booking date). Only an
unrecognized cause falls back to "transaction retrieval failed". Provider
response text is never shown.
While every failing bank has named its own retry time, the account is reported
as waiting, not broken: the dashboard says synchronization retries by itself,
the account card shows a rate-limit badge with that time, and the scheduler
sleeps until the deadline instead of spending hourly session checks on a
refusal it already knows about. Sync now still tries immediately. Any failure
without such a deadline keeps the hourly retry. Retry times are persisted with
the sync state in whole seconds and rendered in the browser's time zone.
Sync now can retry sooner. Balances are fetched
on demand, with exact amount/currency/type values, rather than inferred from an
incomplete historical journal.
Historical imports for connected accounts
-----------------------------------------
Accounts -> account card -> Import older history opens a per-account dialog.
Choose Months back (12 by default, whole numbers 1-120) and confirm. The request
covers the selected past calendar months through today, subject to bank limits.
Only the chosen account is fetched. Existing transactions are deduplicated, and
new records follow normal transfer matching and classification. The dialog shows
the imported count, including zero for a repeated range with no new records.
Normal sync cursors, last-sync time and saved initial-history choices are not
changed. Inactive connected accounts may import history manually. Expired or
revoked bank authorization must be reconnected first. On failure, the UI refreshes
the journal to reveal any records already committed; it does not retry the import.
CSV import and identity
-----------------------
Import is a two-step flow: upload prepares a mapping and a preview, and nothing
reaches the journal until it is confirmed. Cancel, a browser reload, a failed
parse or a journal change between preview and confirmation all import nothing.
A prepared statement is held in memory only, expires after one hour, and at most
five may await confirmation. Uploads are limited to 2 MiB.
Recognized locally, without AI: N26 (English/German, old Date/Datum and new
Booking Date/Buchungsdatum schemas), ING Umsatzanzeige (Buchung,
Wertstellungsdatum, Auftraggeber/Empfaenger, Buchungstext, Verwendungszweck,
Betrag, Waehrung, below its metadata preamble), and Kontist's documented
transaction vocabulary (Payment Date/Buchungsdatum, Name, Amount/Betrag,
Purpose/Verwendungszweck). Accepted everywhere: comma/semicolon/tab separators,
UTF-8 with or without BOM, Windows-1252, CRLF, quoted multiline descriptions,
ISO and German dates, decimal point and comma, and thousands groups of exactly
three digits. Kontist date and decimal conventions are inferred from the file's
own first populated values; slash dates are read as month/day/year unless the
first component exceeds twelve. Foreign original amounts, exchange rates,
balances and categories are never used as account money. Currency comes from a
currency column, else an "Amount (EUR)"-style header, else the account; a
conflict with the account currency fails the whole statement. Use the original
export rather than spreadsheet-reformatted dates and numbers. Currency is
preserved but never converted or summed across currencies.
Any other layout requires a saved OpenRouter key and model: the columns are
mapped by the model, not the records. The request contains the delimiter, the
column names and up to four sample rows in which every letter is replaced by x
and every digit by 0, so descriptions, counterparties, references, IBANs and
amounts are not sent. The proposal is untrusted: each column must name a
supplied header exactly, money must come from one signed column or one
debit/credit pair, the date and decimal conventions must be from the supported
list, and no column may serve two fields. AI-mapped facts are recorded with
source "csv" and carry no transaction reference, because a repeating SEPA
mandate reference must never become a transaction identity. ING facts likewise
carry no reference. Review the previewed dates, amount signs and currency before
confirming; a wrong mapping is visible there, not after import.
Import sources: n26_csv, ing_csv, kontist_csv, scalable_csv, traderepublic_csv,
csv (AI-mapped), enablebanking.
Stable provider entry references are scoped by account, source and debit/credit
direction: a debit and credit can share a reference without being collapsed.
Conflicting booking dates, amounts or currencies within one direction still
fail closed. Existing journal IDs and raw references are retained.
Enable Banking transaction_id is NOT guaranteed stable and is not used as the
primary identity. Fallback fingerprints retain identical-record occurrence
counts: two identical rows remain two transactions, and repeat imports do not
add two more. Without stable
IDs, identical records from separately truncated exports are intrinsically
ambiguous. Import consistent overlapping/full exports. Uncertain cross-source
collisions are rejected rather than silently double counted; retain the error
and reconcile the input locally before retrying. Facts are never silently
replaced when upstream descriptions or amounts change for an existing identity.
Transfers use reciprocal records from different owned accounts, equal/opposite
exact amounts and matching currency, with own-IBAN evidence and booking dates
within three calendar days. Equal competing payments ARE paired, by nearest
booking date and then by transaction ID: every candidate set is a complete
bipartite graph between two fixed accounts at one amount and currency, so every
pairing yields the same accounts, kinds and postings, and iteration order
decides nothing. Leaving them unpaired was the worse option, because both legs
then fell through to the sign-based fallback and appeared as spending and income
that never happened. An existing link is never revisited, and neither is a
record whose classification source is "manual": a hand-made link or unlink
outlives every later import. The linked records remain separate immutable
facts; analytical double-entry postings balance and transfers do not count as
income/spending. Populate local account IBANs to support recognition.
Investment accounts and broker imports
--------------------------------------
An account has a kind, "cash" (the default, and what an absent kind means) or
"investment". An investment account holds a cash balance and positions. It also
carries a settlement IBAN (reference_iban), used when an export names no
counterparty of its own, so deposits and withdrawals pair with the funding
account through ordinary transfer matching. Leave it empty and those rows simply
stay unpaired, which costs accuracy in spending analysis but never invents
income.
Two broker exports are recognized locally, by their complete column set. A
layout is matched whole because a row's meaning depends on a combination of its
classifying columns, so a partial match is a different file wearing the same
names. Everything below about events, instruments, precision and the checks
applies to both; the per-export differences are listed under each.
Scalable Capital exports (scalable_csv) are recognized locally by their full
column set: date, time, status, reference, description, assetType, type, isin,
shares, price, amount, fee, tax, currency. The layout is matched whole, because
a row's meaning depends on the combination of status, assetType and type.
The booking date is the date column exactly as printed. Batch rows are stamped
midnight UTC rendered in local time, so their time column reads 01:00 in winter
and 02:00 in summer; reading date and time together would move half the year's
corporate actions and distributions to the previous day.
Only status "Executed" imports. A cancelled retry is all zeros, so it satisfies
every arithmetic check and would otherwise enter the journal as a phantom trade.
The ten row types, and what each settles:
type assetType cash position
Deposit Cash amount -
Withdrawal Cash amount -
Fee Cash amount -
Interest Cash amount -
Distribution Cash amount -
Buy Security amount - fee - tax +shares
Sell Security amount - fee - tax -shares
Reinvestment_Distribution Security amount - fee - tax +shares
Corporate action Security NONE shares as printed
Security transfer Security NONE shares as printed
A cash row's amount is the money that actually settled and is already net of
the tax the broker withheld or refunded, so its fee and tax columns are recorded
on the fact and never subtracted again. Subtracting them a second time
double-counts by exactly the tax figure. A security row's amount is a gross
pinned to shares times price. A corporate action or depot transfer quotes a
position valuation, not cash: treating it as money conjures or destroys it, and
a depot switch of a whole portfolio does that once per instrument.
The share column is signed only for corporate actions and depot transfers. Buys
and sells are unsigned and take their direction from the type. Both conventions
are resolved at import, once.
Every security row is checked against shares times price, allowing for the
rounding the export's own printed figures propagate. Both ends are rounded and
neither states by how much: one export prints the notional to the cent, so
0,426581 shares at 63,06 settle as 26,90 where the product is 26,90019786;
another prints a price to fewer places than the fill actually had, settling six
NVIDIA shares at 808,5599 against a printed 134,76 whose product is 808,56.
The allowance is half a unit of the gross's stated precision plus one part in a
hundred thousand of the gross. Measured over a complete real export of 88
security rows, exactly one deviates at all, by one part in eight million.
What that still refuses: a price taken from the wrong share class, and the lost
decimal separator the check exists for, four orders of magnitude out. What it
accepts: the broker's own rounding, including a whole cent once a gross stated
to the cent passes about five hundred euro, where a real one-cent error cannot
be told from that rounding.
It cannot catch a separator lost uniformly across a row: 1 x 25,795 and
1 x 25795 both satisfy it. A price cross-check against an outside provider is
the only remedy and is deliberately not implemented. A spreadsheet round-trip
is what strips those separators, so import the broker's original file.
Rejected whole, with the record number: an unknown status, an unknown type, a
classifying column that disagrees with its type, an account type other than the
one the import targets, a currency other than the account's, a security row
without a resolvable identifier, an invalid ISIN, a signed buy or sell where the
export leaves them unsigned, a corporate action or depot transfer carrying a fee
or tax, and any failed arithmetic check. A zero amount is accepted; it corrupts
nothing, and a free share allocation is legitimately priced at zero.
Money holds four decimal places; share counts and unit prices hold eight. An
amount is the row's share count times its price, so it carries as many decimal
places as the two together need: a reinvested distribution in a real export
reaches nine, past both. Amounts are therefore read at arbitrary precision,
rounded to four places half away from zero, and the exact discarded residue is
summed and reported in the import review rather than hidden. Trailing zeros are
padding, not precision: an export that writes a six-place price to ten places is
read at six. A share count or a price beyond eight places is refused instead of
truncated: rounding a share count misstates a holding, and rounding a price
would break the check the amount is verified against.
Fee and tax are always stored as deductions from a gross, so a refunded tax is
a negative deduction, and an export that writes its fee as the negative
adjustment it made to the cash is normalized once, at import. Whether a cash
row's amount is already net of its tax, or a gross the deductions still apply
to, is a fact about the source and is decided there too.
Instruments are registered from the export, keyed by ISIN, with an ID derived
from the ISIN so re-importing never creates a second entry for one security. One
ISIN appears under several names over the years and sometimes under the ISIN
itself; the most recent real name wins, and an import never renames an
instrument that already exists. The name is editable display text; the ISIN is
identity and cannot be changed. Crypto is held under the ISIN-shaped identifier
the broker issues for it, so it needs no separate identity scheme.
Market prices and valuation
---------------------------
An instrument carries an optional market symbol, which is the listing its price
is read from, and the last quote fetched for it with the day that quote closed.
The symbol is set by hand and never derived: one ISIN lists on several exchanges
in different currencies, an ISIN search returns the wrong one often enough to
matter, and a price from the wrong listing misstates wealth without failing any
check. A quote whose currency differs from the instrument's is refused and not
stored.
The quote belongs to the price job. Saving an instrument can neither set it nor
erase it; changing the symbol discards it, because the stored price belongs to
the previous listing. A symbol that cannot be priced keeps its last quote and is
reported as a failure, so the failure mode is a stale figure with a visible
date, never a wrong one. An instrument with no symbol is counted as unpriced,
named in the report, and excluded from every total: cost is not value, and
substituting it would report a number the journal cannot support.
A quote is a rate, not money: money holds four decimal places, while a unit
price can need more. Quotes are therefore stored at the share count's eight-
place precision, and a provider figure is rounded to seven significant digits
before it is stored. Seven is what a 32-bit float carries, and the provider's
closes are 32-bit floats widened to 64: 165.26 arrives as 165.25999450683594,
and rounding at eight would preserve 165.25999 as though it were a price.
The provider is an undocumented, unauthenticated endpoint, and it refuses any
request whose User-Agent names a programming language, so the client sends a
browser agent; without it every fetch answers HTTP 429 on the first call. Runs
are paced, fetches are bounded and never follow redirects, and no response text
reaches an error message. The automatic run starts shortly after launch and
repeats daily. Nothing is committed when no quote changed.
A holding's value is its share count times its quote, rounded half away from
zero to money's four places. Positions is that value summed per account, wealth
is cash plus positions plus hand-valued assets, and result is value plus
everything the position returned less everything put into it - the outcome to
date, realised and not. A hand-valued asset (a house, a car, a private loan) is
entered on the Wealth page with a stated value, a currency and the day the
estimate was made; a negative value records a liability. None of these figures
are read from the DuckDB index: the report is recomputed from the journal so it
can be checked against a broker's own screen.
A broker reuses one reference across every leg of an economic event: the cash
and position sides of a corporate action arrive with the same reference byte for
byte, and the position leg's zero amount does not even differ in direction.
Transaction identity therefore includes the event and its instrument. The
reference itself also embeds an account-level identifier that repeats across
unrelated events, so it is evidence of an event, never of a transaction.
Trade Republic exports and their differences
-------------------------------------------
Trade Republic exports (traderepublic_csv) are recognized by their full column
set: datetime, date, account_type, category, type, asset_class, name, symbol,
shares, price, amount, fee, tax, currency, original_amount, original_currency,
fx_rate, description, transaction_id, counterparty_name, counterparty_iban,
payment_reference, mcc_code.
Nine row types, classified by category and type:
category type cash position
CASH TRANSFER_INBOUND amount -
CASH TRANSFER_INSTANT_INBOUND amount -
CASH TRANSFER_OUTBOUND amount -
CASH TRANSFER_INSTANT_OUTBOUND amount -
CASH INTEREST_PAYMENT amount -
CASH DIVIDEND amount -
CASH TAX_OPTIMIZATION amount -
TRADING BUY amount +shares
TRADING SELL amount -shares
where cash is in every case amount minus the fee and tax deducted from it.
No row type moves a position without moving cash, so the cash-neutral class
that Scalable's corporate actions and depot transfers belong to does not arise.
Three conventions are the opposite of Scalable's, and each one moves money if
read the other way round:
- fee and tax are signed adjustments to cash, not deductions. A one euro
order fee is written -1.00 and withheld tax -4.33, so both are negated at
import and the journal keeps its single convention.
- a cash row's amount is the gross, not the net. Interest of 16.46 with -4.33
of tax credits 12.13.
- a TAX_OPTIMIZATION row carries zero in the amount column and its money in
the tax column, signed both ways. Read as cash, all of them move nothing;
read correctly, they are the loss-offset pot settling, in either direction.
A DIVIDEND row populates the share column with the holding the dividend was
paid on, not with a position change. Adding it would double the holding, so it
is read as the attribution it is and otherwise discarded.
The security identifier is the symbol column when that is an ISIN, and
otherwise the one ISIN the description names: crypto carries a bare ticker in
the column and its identifier only in the text. A row that moves a position
and resolves to neither is refused.
The counterparty of a transfer is the counterparty_iban column when populated,
else the IBAN the description carries in parentheses, else the account's
configured settlement IBAN. Free text contributes only a value shaped like an
IBAN, so a description naming no account contributes nothing.
The booking date is the date column exactly as printed. The datetime column is
UTC while the date column is local, so they disagree for rows booked late in
the evening; deriving the date from the timestamp moves those rows a day back.
Only account_type DEFAULT imports. One export covers one account, and a second
account type in the same file would merge two cash balances into one.
original_amount, original_currency and fx_rate are informational: settlement is
in the currency column, which must match the account's. payment_reference and
mcc_code are unused - no card rows appear in this export type, and if they ever
do they are spending with a merchant, not broker activity.
Broker facts carry enrichment kind "investment". Like a transfer it has no
category and no merchant, it is excluded from spending and income analytics and
from bulk reclassification, and the AI never sees it. Crucially, a broker fact
never reaches the sign-based fallback, so an unmatched deposit is not income and
a broker fee is not household spending. Analytical postings route it to
clearing:investments, where the residue left behind is exactly the cash an
investment account has returned: distributions and interest received, less fees.
Wealth and reconciliation
-------------------------
The Wealth page reports, per account, the cash balance as every recorded
movement summed, the positions as every signed share count summed, and named
checks. It is computed from the journal, not from the DuckDB index, because it
exists to be compared with the figures a bank or broker shows on its own screen.
A cash balance equals the real balance only when the journal holds that
account's complete history. A broker export does; a date-windowed bank statement
does not.
Checks that fail mean the journal disagrees with itself: row arithmetic, cash
never negative, holdings never negative. A negative holding means a position was
closed that was never opened in the imported data, so the export is partial or a
sign is wrong. Checks that only note: fee and tax recorded but not applied,
deposits or withdrawals with no counterpart in another account, and holdings
left out of the wealth figure for want of a quote.
Out of scope, deliberately: intraday prices, net worth over time, FIFO lot
accounting, realised gains, Vorabpauschale, and currency conversion. A position's
"invested" figure is cash in less cash out, not a cost basis: a depot transfer
moves a position with no cash at all, and a sale returns cash without
identifying which lot it closed.
Canonical files and recovery
----------------------------
finance/
config.toml model/amount/import opt-in only, no API keys
accounts.finance
categories.finance
tags.finance
merchants.finance
instruments.finance
assets.finance
journal/YYYY/YYYY-MM.finance
state/sync-state.json sensitive local consent/session metadata
state/openrouter.json sensitive UI-managed OpenRouter key or explicit disable
state/enablebanking.json sensitive UI-managed banking key/configuration or disable
cache/finance.duckdb disposable analytical projection
The custom grammar is deliberately small:
category {
id: "cat_example"
name: "Groceries"
parent_id: "cat_expenses"
kind: "expense"
}
A transaction block has facts: {...} and enrichment: {...} JSON-valued fields.
A broker fact additionally carries an investment: {...} object holding the
event, instrument, signed quantity, price, gross, fee and tax; absent fields are
omitted, and its presence is what marks a fact as a broker fact.
Financial amounts are quoted decimal strings, never binary floating point.
Up to four fractional digits are supported; arithmetic uses exact ten-thousandths
with explicit overflow checks. DuckDB stores DECIMAL(24,4).
Share quantities are quoted decimal strings with up to eight fractional digits,
arithmetic uses exact hundred-millionths, and a quantity times a price is
multiplied at 128-bit width before rounding back to four places.
Each block starts with account/category/tag/merchant/instrument/transaction and
'{' on its own line; fields use name: JSON. Strings use JSON escaping (including
\n for multiline descriptions). JSON values may span lines. Blank lines and
full-line # or // comments are accepted between fields/blocks. Unknown fields,
duplicate keys, malformed records, invalid references and taxonomy cycles are
rejected.
The grammar is version-one strict: extension/split fields are not accepted yet.
Future format extensions require an explicit parser migration.
Stable category IDs survive renaming and moving; assigned categories must remain
leaves. Registry display names (category, tag, merchant, instrument) are
capped at 200 characters server-side, matching every UI form. Built-in roots
and fallback leaves are protected. Move assigned records
to another leaf before adding children to their former category. Category
merges migrate referenced transactions/defaults; tag merges deduplicate links;
tag deletion removes all affected links after UI confirmation. Merchant merging
migrates transactions and retains source names/aliases on the target merchant.
Unchanged blocks and comments retain their text. UI enrichment edits do not
rewrite imported facts. File hashes detect external edits; invalid files stop
loading/indexing with a file/line error, not a partially refreshed dashboard.
A process lock prevents multiple app writers; use one instance per finance dir.
Revision conflicts require refreshing/re-previewing, not blind overwriting.
Multi-file writes are recoverable and logically atomic within the application.
Do not run external writers during a commit; external tools do not participate
in the app's process lock. Stop the service for manual bulk edits or backups.
If a write is interrupted, keep all state files and restart for recovery before
editing the journal manually. Keep backups of the entire canonical directory,
including hidden/state recovery files, plus the separately stored secrets.
The cache can be excluded. Avoid exposing any financial directory through a
static file server, Git public remote, or unencrypted shared backup.
Rebuild from text:
./bin/finance-duck -data ./finance -rebuild
Stop the running app before using that command (single writer lock), or use
Settings -> Rebuild index while it runs. A broken/deleted DuckDB file can be
removed while stopped and regenerated; it never contains the only copy of
financial records. When an index rebuild fails, UI mutations still preserve
canonical data and the index error is surfaced rather than serving stale totals.
Reclassification
----------------
AI / Classification: choose dates, model and independent Merchant/Category/Tags
fields. Analyse starts a background run and reports live progress: analysed
count, proposed changes, and per-transaction errors as they happen. Analyse
classifies up to 10 transactions of one kind per provider request; the
registry and history are sent once per batch, and a request rejected outright
for schema complexity halves until the provider accepts it, remembering the
working size for the rest of the run. Requests stay
paced seconds apart, so a large range takes minutes; the page may be left
and revisited, and Stop abandons the run without writing anything. A run that
has produced no successful proposal and fails three times in a row with the
same error stops early and reports that error instead of repeating it across
the whole range. Only one run exists at a time.
Starting analysis reads the latest journal, independent of the page's revision.
The page refreshes registry labels before starting; analysis itself writes nothing.
History precedent sent with each request marks the user's own decisions
(manual edits and merchant rules) as source user, ranks them ahead of the
model's earlier answers, and reserves window slots for them, so one manual
correction outweighs repeated uncorrected AI output for the same payee.
Manually linking a merchant also records the counterparty as an alias, so
recurring payees classify locally without any provider request.
The finished run is a read-only preview. Apply all/selected writes all
selected changes in one canonical commit; financial facts never change. Apply
checks selected transactions against the preview snapshot; unrelated journal
commits do not require another analysis.
Previews are kept in memory for up to 24 hours from the start of analysis and
disappear on restart. Cancel writes nothing. Transfers and broker facts are
skipped, and unselected fields are preserved.
When a selected transaction is linked to a merchant, applying the preview and
manual transaction edits may add its normalized counterparty as an alias if
that alias is unambiguous and the merchant has fewer than 32 aliases. A new
merchant proposal starts with the current counterparty as its first alias.
Failed rows remain unchanged and are listed separately from proposed changes.
Boundaries and verification
---------------------------
There are no splits, budgets, tax/invoice/receipt processing, login/multi-user
support, arbitrary SQL or natural-language query execution. Investment support
covers positions, cash and a daily closing price per instrument: no intraday
prices, net worth over time, FIFO lots, realised gains, Vorabpauschale or
currency conversion.
Natural-language query DSL and Sankey exploration remain explicitly later work.
There is no browser-to-bank credential handling or payment initiation.
Automated tests exercise deterministic financial invariants and mock remote
provider HTTP behavior. They do not replace testing real consent renewals and
real booked transaction samples for your banks. No real user credentials or
financial history are included in the repository. Test fixtures are synthetic.
API documentation sources:
https://enablebanking.com/docs/api/reference/
https://openrouter.ai/docs/guides/features/structured-outputs
https://openrouter.ai/docs/guides/features/zdr
https://openrouter.ai/docs/guides/routing/provider-selection
Reference in New Issue View Git Blame Copy Permalink