Lars Nolden
922ae507bd
An account now has a kind, and an investment account holds positions as well as
cash. A broker row is not a new entity: it is a bank fact with an optional
position leg, so deduplication, the journal, fact immutability, the DuckDB
projection and the transactions view carry it unchanged. Facts.Amount stays the
cash leg and is zero on the rows that move only a position.
Scalable Capital exports are recognized locally as a fourth format, read by
their own parser because a column mapping cannot describe them: the amount
column is settled cash on a cash row, a gross to be netted on a trade, and a
position valuation that must never touch cash on a corporate action or a depot
transfer. A cash amount is already net of the tax the broker withheld or
refunded, so that tax is recorded on the fact and never subtracted a second
time; treating a corporate action's valuation as money conjures cash, and a
depot switch would do it once per instrument. The share column is signed only
for those two types, so buys and sells take their direction from the type. Every
security row is checked against shares times price at 128-bit width, because a
lost decimal separator survives every other check. An unknown status, type or
assetType, a foreign currency, a missing ISIN, or one failed check rejects the
whole file with the record number.
Instruments live in instruments.finance, keyed by ISIN with an ID derived from
it, so re-importing never registers a security twice. One ISIN appears under
several broker descriptions over the years and sometimes under the ISIN itself:
the most recent real description names it, and an import never renames one that
already exists. A broker also reuses a single reference across every leg of one
event, so transaction identity includes the event and its instrument.
domain.Fallback returns kind "investment" for any fact carrying a position leg,
so no broker row reaches the sign-based branch. That single rule is what stops
an unmatched deposit from counting as income and a broker fee from counting as
household spending; the monthly PRIME fee and its matching credit now cancel in
clearing:investments with no configuration at all. Investment rows are excluded
from spending analytics, from bulk reclassification and from the model, exactly
as transfers are.
Equal competing transfers are paired instead of skipped. Every connected
component of the candidate graph is a complete bipartite graph between two fixed
accounts at one amount and currency, so every pairing produces the same
accounts, kinds and postings and only the displayed counterpart differs.
Refusing to choose was the expensive option: both legs fell through to the
sign-based fallback and appeared as spending and income that never happened.
Pairing follows the nearest booking date, then the transaction ID, so iteration
order decides nothing. POST /api/transactions/{id}/transfer rewrites the old and
the new pair in one commit, because reciprocity is validated and a half-applied
link is an invalid dataset, and the matcher now skips any record classified
manually so a hand-made link or unlink outlives the next import.
Wealth reports each account's cash and positions from the journal rather than
the index, with named checks - row arithmetic, cash never negative, holdings
never negative - because it exists to be compared against the figures a broker
shows on its own screen. A negative holding means the imported history is
partial. Share counts are exact to eight places; a reinvested distribution
quoted to six is rounded to money's four and the residue is reported rather than
hidden. Market prices, market value, net worth over time, FIFO lot accounting,
realised gains and currency conversion are deliberately absent.
601 lines
35 KiB
Plaintext
601 lines
35 KiB
Plaintext
FINANCE DUCK — GO + REACT
|
|
|
|
Local build on NixOS
|
|
-------------------
|
|
nix-shell
|
|
npm --prefix web ci
|
|
npm --prefix web run build
|
|
go build -o bin/finance-duck ./cmd/finance-duck
|
|
./bin/finance-duck -data ./finance
|
|
|
|
Open http://localhost:8080. Create an account, then import its CSV statement
|
|
from Accounts and confirm the reviewed mapping. The application starts empty
|
|
except for expense/income fallback categories. Create your category tree, tags,
|
|
and merchants in the UI. Enable a merchant's default rule explicitly only when
|
|
its category/tags are reliable; leave it disabled for ambiguous merchants such
|
|
as Amazon.
|
|
|
|
Tests: go test ./...
|
|
The Go build embeds web/dist, so build React first. CGO and a C++ linker are
|
|
required by the native DuckDB driver. shell.nix supplies the toolchain. React
|
|
and CSS are served locally; there are no CDN requests or tracking scripts.
|
|
|
|
Private deployment
|
|
------------------
|
|
There is NO application login. Default bind: 127.0.0.1:8080.
|
|
For a VPN/reverse-proxy hostname, use an exact browser origin:
|
|
./bin/finance-duck -data /srv/finance -listen 127.0.0.1:8080 \
|
|
-public-url https://finance.example.internal
|
|
Preserve the public Host header at the reverse proxy. The application rejects
|
|
other Hosts and cross-origin writes. Configure a long proxy request timeout
|
|
for bulk AI previews. Do not expose the reverse proxy to the public Internet.
|
|
A VPN-address bind can be used instead, with the corresponding -public-url.
|
|
|
|
Docker: docker compose up --build -d
|
|
Compose publishes only 127.0.0.1:8080. Set FINANCE_PUBLIC_URL when proxying it.
|
|
The container is unprivileged with a read-only root and a persistent named
|
|
volume at /data. A host bind mount must be writable by UID/GID 10001.
|
|
Never change the port mapping to public 0.0.0.0 without VPN/firewall isolation.
|
|
|
|
Native NixOS: the flake exports nixosModules.default. Import it into your host,
|
|
then set services.finance-duck.enable=true and publicURL to the exact browser
|
|
origin. Deploy with your normal nixos-rebuild switch. The module runs ONE
|
|
systemd service; React is embedded in the Go binary, not a separate service.
|
|
Defaults: listen 127.0.0.1:8080, service user finance-duck, persistent data at
|
|
/var/lib/finance-duck (0700). It does not configure networking or Tailscale.
|
|
Optional services.finance-duck.environmentFile names an existing runtime file
|
|
read by systemd for optional provider startup fallbacks. Both Enable Banking
|
|
and OpenRouter can be configured directly in Settings without that file.
|
|
Keep private keys outside the Nix store and readable only by the service.
|
|
See README's native NixOS section for optional environment-file permissions.
|
|
|
|
Deployed host: bender@100.87.224.3
|
|
--------------------------------
|
|
URL: https://nixos.taile9e6d9.ts.net:8444 (Tailscale only, no application login).
|
|
The backend binds 127.0.0.1:8080. Existing Funnel on 443 and OpenClaw on 8443
|
|
are separate. Never enable Funnel for Finance Duck.
|
|
|
|
Update from your workstation:
|
|
ssh -t bender@100.87.224.3 'sudo finance-duck-update'
|
|
Or on the host:
|
|
sudo finance-duck-update
|
|
sudo finance-duck-update --no-pull # deploy existing checkout edits
|
|
|
|
The updater serializes updates, pulls ~/projects/finance-duck as bender with
|
|
git pull --ff-only, creates a Git-filtered source snapshot, updates ONLY the
|
|
finance-duck input in /etc/nixos/flake.lock, runs nixos-rebuild switch, then
|
|
checks /api/health with the correct Host. The app has its own pinned Nixpkgs;
|
|
the host's Nixpkgs and NixVirt pins stay unchanged. A build failure leaves the
|
|
running service untouched; an activation/health failure exits nonzero.
|
|
|
|
Application and deployment source are tracked in the project repository.
|
|
Commit and push changes before running the normal updater. New local source
|
|
files must be tracked with git add before a --no-pull deployment. Pull refuses
|
|
to overwrite conflicting local edits; resolve them before retrying. No
|
|
automatic stashing, resetting, committing or pushing occurs. Keep credentials
|
|
and live finance data out of Git.
|
|
|
|
Host files:
|
|
/etc/nixos/flake.nix application input and module import
|
|
/etc/nixos/modules/finance-duck.nix service, tailnet proxy, update command
|
|
/var/lib/finance-duck-source replaceable Git-filtered source snapshot
|
|
/var/lib/finance-duck persistent data, finance-duck:finance-duck 0700
|
|
The source snapshot excludes .git, ignored files and untracked files. It is
|
|
not a data backup. The initial deployment starts empty without provider
|
|
credentials; it does not copy the workstation's finance/ directory.
|
|
|
|
Inspect on the host:
|
|
systemctl status finance-duck finance-duck-tailscale-serve
|
|
sudo journalctl -u finance-duck -f
|
|
tailscale serve status
|
|
curl -f https://nixos.taile9e6d9.ts.net:8444/api/health
|
|
|
|
Before significant upgrades, stop the service and back up the entire data
|
|
directory to a protected location (including hidden recovery files). Restart
|
|
after the backup, and protect any separate credential files as well:
|
|
sudo systemctl stop finance-duck
|
|
sudo install -d -m 0700 /var/backups/finance-duck
|
|
sudo sh -c 'umask 077; tar -C /var/lib -czf /var/backups/finance-duck/data-$(date +%Y%m%dT%H%M%S).tar.gz finance-duck'
|
|
sudo systemctl start finance-duck
|
|
Copy backups to secure off-host storage; a backup on this disk cannot protect
|
|
against disk loss.
|
|
|
|
Binary/system rollback:
|
|
sudo nixos-rebuild switch --rollback
|
|
This switches the previous NixOS generation, NOT financial data or config
|
|
source. The generation before the first deployment has no Finance Duck service.
|
|
For a lasting rollback, restore the intended source/config/lock before the
|
|
next update. Do not remove or roll back live data blindly.
|
|
|
|
OpenRouter
|
|
----------
|
|
Open Settings -> OpenRouter credentials, paste the API key, and Save key.
|
|
Choose the exact provider/model identifier in Classification preferences and
|
|
save preferences. Replace key rotates it; Remove key disables AI. No SSH,
|
|
Nix rebuild or restart is needed. Changes affect future classification
|
|
requests; in-flight requests retain their original key.
|
|
|
|
"Classify newly imported transactions with AI" in Classification preferences
|
|
controls whether imports contact the provider at all. It applies to CSV imports
|
|
and bank synchronization, defaults to on, and is stored as classify_on_import
|
|
in config.toml. With it off, no import makes a provider request: enabled
|
|
merchant-default rules still classify, an alias match still attaches its
|
|
merchant, and everything else arrives on the editable fallback with no
|
|
provenance error. AI classification -> Analyse is unaffected by this preference.
|
|
|
|
The key is stored as private 0600 plaintext in state/openrouter.json under
|
|
the data directory, never in config.toml or browser storage. API responses
|
|
never return the saved key. Backups of the data directory contain this secret.
|
|
Only allow trusted users to reach the application; there is no separate
|
|
administrator login for changing credentials.
|
|
|
|
OPENROUTER_API_KEY remains a startup fallback only when the saved credential
|
|
file is absent. A saved key overrides it. Removing the key in Settings saves
|
|
an explicit disable, which also overrides the environment after restart.
|
|
Malformed saved credentials fail startup rather than reverting to another key.
|
|
|
|
Configured means present, not verified with OpenRouter. A successful
|
|
AI classification -> Analyse request verifies access with the selected model.
|
|
The model and endpoint must support strict structured outputs and the
|
|
configured privacy routing. Every classification request sets:
|
|
provider.data_collection = deny
|
|
provider.zdr = true
|
|
provider.require_parameters = true
|
|
No retry relaxes these requirements. OpenRouter must also have prompt logging
|
|
disabled in your account settings. The underlying provider processes prompts;
|
|
this is not local AI and cannot promise that a remote provider honors policy.
|
|
|
|
Amounts and currency are omitted by default. Include Amount in Settings is
|
|
explicit opt-in. Local account/provider IDs, known counterparty names, banking
|
|
identifiers and recognizable references are stripped; candidate identifiers
|
|
are per-request opaque tokens. Categories/tags and candidate merchant names
|
|
are deliberately sent as classification context. Free-form text can contain
|
|
unknown personal names, so automatic sanitization is not an anonymity guarantee.
|
|
Conservative redaction can reduce recognition quality. Inspect your descriptions
|
|
and do not configure an API key if no financial text may leave the server.
|
|
|
|
Classification failures do not discard imports: facts are committed first and
|
|
failed enrichment stays unclassified with an error visible in Transactions.
|
|
Classification requests use one transaction at a time, not batches. Known
|
|
merchant defaults can classify without any configured AI key.
|
|
|
|
Provider rate limits
|
|
--------------------
|
|
Classification and Enable Banking use separate in-memory request gates. AI
|
|
imports and previews, including previews choosing another model, share the AI
|
|
gate. HTTP 429 retries retain the original request and privacy/authentication
|
|
controls. Other HTTP failures are not automatically retried.
|
|
|
|
Every outbound attempt is paced, including successful calls: AI request starts
|
|
are at least 3 seconds apart; bank starts at least 1 second apart. These are
|
|
conservative application defaults, not universal provider quota guarantees.
|
|
On 429, exponential fallback begins at 15 seconds for AI and 30 seconds for
|
|
banking. Consecutive failures escalate across operations, capped at 15 minutes;
|
|
success resets failure escalation but retains learned spacing up to 30 seconds.
|
|
Retry-After seconds and HTTP dates can extend delays, never shorten them.
|
|
Each operation makes at most four attempts and spends at most two minutes in
|
|
automatic retry waits. Normal pacing waits and queueing honor cancellation.
|
|
Network attempt timeouts remain separate (bank: at most 30 seconds; AI:
|
|
45 seconds by default).
|
|
|
|
Long hints, exhausted attempts, and canceled retries retain the provider's
|
|
cooldown. New calls fail without contacting that provider before its retry
|
|
time. Local merchant rules still work. Unrepresentably large positive hints
|
|
disable automatic retries rather than overflow into an early request.
|
|
These gates live in the running client, not the persisted financial journal.
|
|
Retries cannot lift provider quotas; wait until the reported time. Existing
|
|
failed classifications require another Analyse preview and explicit Apply.
|
|
|
|
Bank GET requests may retry 429. Authorization and once-only session exchange
|
|
POST requests never replay automatically, but still establish a cooldown.
|
|
Session status checks use the authorized UID list and expiry, without repeated
|
|
account-details requests; full account metadata remains saved from Exchange.
|
|
An unavailable session reports its original safe rate error once, without
|
|
additional unavailable-account errors or a false reconnection requirement.
|
|
Failed-account cursors and the last successful complete sync remain unchanged.
|
|
|
|
Manual sync, history import and balance HTTP requests forward actual PSU
|
|
metadata to account-data GET requests, consistently across retries/pages.
|
|
Only IP, User-Agent and available browser Accept headers are forwarded; never
|
|
cookies, authorization headers, referrer URLs, or arbitrary supplied Psu-* fields.
|
|
Scheduled sync contexts have no PSU metadata. User metadata is request-local,
|
|
never saved on a shared bank client or copied into classification requests.
|
|
|
|
With a public origin configured, a loopback reverse proxy must append its
|
|
observed client IP to X-Forwarded-For. Only its rightmost appended address is
|
|
trusted; direct/non-loopback callers cannot override their peer IP. Missing or
|
|
malformed proxy addresses remain unknown, not invented. Tailscale Serve uses
|
|
this deployment arrangement; other proxies must preserve this trust contract.
|
|
|
|
For a confirmed background ASPSP_RATE_LIMIT_EXCEEDED, defer that account and
|
|
endpoint at least six hours (longer provider hints take precedence), rather
|
|
than replaying every few seconds. This background-only quota does not suppress
|
|
an eligible genuine user request or another account/endpoint. Short generic
|
|
platform limits still use the shared bank controller for both request modes.
|
|
Only the bounded, exact error code is inspected; private provider messages are
|
|
not returned or logged. Quota state is client-local, finite entries expire,
|
|
and restart does not lift the upstream bank's actual quota.
|
|
Source: https://enablebanking.com/docs/faq/
|
|
|
|
Enable Banking
|
|
--------------
|
|
Register your application and public certificate with Enable Banking. For
|
|
personal production access follow its linked-own-accounts registration rules:
|
|
https://enablebanking.com/docs/api/linked-accounts/
|
|
Configure the allowed redirect URL to the exact externally reachable URL:
|
|
https://finance.example.internal/api/banking/callback
|
|
The browser must be able to reach this callback through your VPN.
|
|
|
|
Open Settings -> Enable Banking credentials. Enter the application ID, upload
|
|
its matching private-key PEM file, and Save configuration. Settings supplies
|
|
the exact browser-origin callback URL to register. Keys must be a single
|
|
unencrypted RSA PKCS#1 or PKCS#8 PEM, >=2048 bits and <=32 KiB.
|
|
No SSH, environment file, Nix rebuild or restart is needed.
|
|
|
|
Save validates local credentials without contacting Enable Banking. It does
|
|
not register/activate the app or authorize an account. Configured means local
|
|
setup is present; complete registration above and authorize under Accounts.
|
|
|
|
Leave the key upload empty when editing the same application to keep its key.
|
|
Same-app key/callback rotation preserves sessions and cursors. Changing the
|
|
application ID requires a new key upload and reconnecting your banks. Remove
|
|
configuration confirms before disabling banking and clearing local connection
|
|
state. Canonical accounts/history remain; upstream bank consent is not revoked.
|
|
Each successful change invalidates pending authorization flows.
|
|
|
|
Credentials persist privately (0600) in state/enablebanking.json beneath the
|
|
data directory. The API never returns the private key; browser storage never
|
|
contains it. Session state is bound to the application configuration generation,
|
|
so stale sessions cannot be revived by a restart after app changes/removal.
|
|
Back up the entire data directory securely, not isolated credential/state files.
|
|
|
|
Optional environment startup fallback, used ONLY when no saved config exists:
|
|
ENABLEBANKING_APP_ID=<registered application ID>
|
|
ENABLEBANKING_KEY_FILE=/run/secrets/enablebanking.key
|
|
ENABLEBANKING_REDIRECT_URL=https://finance.example.internal/api/banking/callback
|
|
The referenced key must be readable by the service. Environment changes need
|
|
a restart. UI configuration or explicit removal overrides all three variables;
|
|
invalid saved configuration fails startup rather than using another key.
|
|
|
|
Accounts shows the configured callback URL. Register it exactly with Enable
|
|
Banking. Settings saves the current browser callback URL. The redirect carries a
|
|
one-time code and state, not a reusable API key. Go verifies state, exchanges the
|
|
code for a session_id, and persists session details locally with mode 0600.
|
|
|
|
Accounts -> Connect: enter the exact Enable Banking institution name and
|
|
country code (DE for Germany), then choose History to import (months).
|
|
The default is 12; whole numbers from 1 to 120 are accepted. Authorize through
|
|
the bank to save this choice with the consent. A successful callback immediately
|
|
wakes the synchronization worker; Sync now is also available.
|
|
The dashboard shows, for example, \"ING needs reconnection\" when consent expires
|
|
or is revoked. Reconnect ING starts the same approval flow with that bank and
|
|
country already selected. The new consent replaces the old account bindings
|
|
without duplicating local accounts or their financial history. Transient provider
|
|
errors are displayed separately from expired consent.
|
|
|
|
Only booked transactions are persisted. Each new account initially requests the
|
|
selected number of calendar months (12 by default); the bank may provide less.
|
|
Automatic sync runs twice a day, every 12 hours from the last successful run,
|
|
and deliberately overlaps each account's last successful sync by 14 days.
|
|
Per-account cursors prevent another account's recent sync from skipping a new
|
|
account's history. Reconnection preserves the saved history choice and existing
|
|
cursors. Changing the choice or reconnecting does not backfill already-synced
|
|
accounts. Older records can be imported using CSV. Older saved consents without
|
|
a history choice use 12 months for accounts that have no successful-sync cursor.
|
|
A failed provider call retains local data. Failures whose cause Finance Duck
|
|
determines locally are named: a bank rate limit with its retry time, an expired
|
|
consent, an HTTP status, an unreachable provider, or a response the journal
|
|
cannot use (for example a booked transaction without a booking date). Only an
|
|
unrecognized cause falls back to "transaction retrieval failed". Provider
|
|
response text is never shown.
|
|
|
|
While every failing bank has named its own retry time, the account is reported
|
|
as waiting, not broken: the dashboard says synchronization retries by itself,
|
|
the account card shows a rate-limit badge with that time, and the scheduler
|
|
sleeps until the deadline instead of spending hourly session checks on a
|
|
refusal it already knows about. Sync now still tries immediately. Any failure
|
|
without such a deadline keeps the hourly retry. Retry times are persisted with
|
|
the sync state in whole seconds and rendered in the browser's time zone.
|
|
Sync now can retry sooner. Balances are fetched
|
|
on demand, with exact amount/currency/type values, rather than inferred from an
|
|
incomplete historical journal.
|
|
|
|
Historical imports for connected accounts
|
|
-----------------------------------------
|
|
Accounts -> account card -> Import older history opens a per-account dialog.
|
|
Choose Months back (12 by default, whole numbers 1-120) and confirm. The request
|
|
covers the selected past calendar months through today, subject to bank limits.
|
|
Only the chosen account is fetched. Existing transactions are deduplicated, and
|
|
new records follow normal transfer matching and classification. The dialog shows
|
|
the imported count, including zero for a repeated range with no new records.
|
|
Normal sync cursors, last-sync time and saved initial-history choices are not
|
|
changed. Inactive connected accounts may import history manually. Expired or
|
|
revoked bank authorization must be reconnected first. On failure, the UI refreshes
|
|
the journal to reveal any records already committed; it does not retry the import.
|
|
|
|
CSV import and identity
|
|
-----------------------
|
|
Import is a two-step flow: upload prepares a mapping and a preview, and nothing
|
|
reaches the journal until it is confirmed. Cancel, a browser reload, a failed
|
|
parse or a journal change between preview and confirmation all import nothing.
|
|
A prepared statement is held in memory only, expires after one hour, and at most
|
|
five may await confirmation. Uploads are limited to 2 MiB.
|
|
|
|
Recognized locally, without AI: N26 (English/German, old Date/Datum and new
|
|
Booking Date/Buchungsdatum schemas), ING Umsatzanzeige (Buchung,
|
|
Wertstellungsdatum, Auftraggeber/Empfaenger, Buchungstext, Verwendungszweck,
|
|
Betrag, Waehrung, below its metadata preamble), and Kontist's documented
|
|
transaction vocabulary (Payment Date/Buchungsdatum, Name, Amount/Betrag,
|
|
Purpose/Verwendungszweck). Accepted everywhere: comma/semicolon/tab separators,
|
|
UTF-8 with or without BOM, Windows-1252, CRLF, quoted multiline descriptions,
|
|
ISO and German dates, decimal point and comma, and thousands groups of exactly
|
|
three digits. Kontist date and decimal conventions are inferred from the file's
|
|
own first populated values; slash dates are read as month/day/year unless the
|
|
first component exceeds twelve. Foreign original amounts, exchange rates,
|
|
balances and categories are never used as account money. Currency comes from a
|
|
currency column, else an "Amount (EUR)"-style header, else the account; a
|
|
conflict with the account currency fails the whole statement. Use the original
|
|
export rather than spreadsheet-reformatted dates and numbers. Currency is
|
|
preserved but never converted or summed across currencies.
|
|
|
|
Any other layout requires a saved OpenRouter key and model: the columns are
|
|
mapped by the model, not the records. The request contains the delimiter, the
|
|
column names and up to four sample rows in which every letter is replaced by x
|
|
and every digit by 0, so descriptions, counterparties, references, IBANs and
|
|
amounts are not sent. The proposal is untrusted: each column must name a
|
|
supplied header exactly, money must come from one signed column or one
|
|
debit/credit pair, the date and decimal conventions must be from the supported
|
|
list, and no column may serve two fields. AI-mapped facts are recorded with
|
|
source "csv" and carry no transaction reference, because a repeating SEPA
|
|
mandate reference must never become a transaction identity. ING facts likewise
|
|
carry no reference. Review the previewed dates, amount signs and currency before
|
|
confirming; a wrong mapping is visible there, not after import.
|
|
|
|
Import sources: n26_csv, ing_csv, kontist_csv, scalable_csv, csv (AI-mapped),
|
|
enablebanking.
|
|
|
|
Stable provider entry references are scoped by account, source and debit/credit
|
|
direction: a debit and credit can share a reference without being collapsed.
|
|
Conflicting booking dates, amounts or currencies within one direction still
|
|
fail closed. Existing journal IDs and raw references are retained.
|
|
Enable Banking transaction_id is NOT guaranteed stable and is not used as the
|
|
primary identity. Fallback fingerprints retain identical-record occurrence
|
|
counts: two identical rows remain two transactions, and repeat imports do not
|
|
add two more. Without stable
|
|
IDs, identical records from separately truncated exports are intrinsically
|
|
ambiguous. Import consistent overlapping/full exports. Uncertain cross-source
|
|
collisions are rejected rather than silently double counted; retain the error
|
|
and reconcile the input locally before retrying. Facts are never silently
|
|
replaced when upstream descriptions or amounts change for an existing identity.
|
|
|
|
Transfers use reciprocal records from different owned accounts, equal/opposite
|
|
exact amounts and matching currency, with own-IBAN evidence and booking dates
|
|
within three calendar days. Equal competing payments ARE paired, by nearest
|
|
booking date and then by transaction ID: every candidate set is a complete
|
|
bipartite graph between two fixed accounts at one amount and currency, so every
|
|
pairing yields the same accounts, kinds and postings, and iteration order
|
|
decides nothing. Leaving them unpaired was the worse option, because both legs
|
|
then fell through to the sign-based fallback and appeared as spending and income
|
|
that never happened. An existing link is never revisited, and neither is a
|
|
record whose classification source is "manual": a hand-made link or unlink
|
|
outlives every later import. The linked records remain separate immutable
|
|
facts; analytical double-entry postings balance and transfers do not count as
|
|
income/spending. Populate local account IBANs to support recognition.
|
|
|
|
Investment accounts and broker imports
|
|
--------------------------------------
|
|
An account has a kind, "cash" (the default, and what an absent kind means) or
|
|
"investment". An investment account holds a cash balance and positions. It also
|
|
carries a settlement IBAN (reference_iban): a broker export has no counterparty
|
|
column, so deposits and withdrawals are stamped with that IBAN and pair with the
|
|
funding account through ordinary transfer matching. Leave it empty and those
|
|
rows simply stay unpaired, which costs accuracy in spending analysis but never
|
|
invents income.
|
|
|
|
Scalable Capital exports (scalable_csv) are recognized locally by their full
|
|
column set: date, time, status, reference, description, assetType, type, isin,
|
|
shares, price, amount, fee, tax, currency. The layout is matched whole, because
|
|
a row's meaning depends on the combination of status, assetType and type.
|
|
|
|
The booking date is the date column exactly as printed. Batch rows are stamped
|
|
midnight UTC rendered in local time, so their time column reads 01:00 in winter
|
|
and 02:00 in summer; reading date and time together would move half the year's
|
|
corporate actions and distributions to the previous day.
|
|
|
|
Only status "Executed" imports. A cancelled retry is all zeros, so it satisfies
|
|
every arithmetic check and would otherwise enter the journal as a phantom trade.
|
|
|
|
The ten row types, and what each settles:
|
|
|
|
type assetType cash position
|
|
Deposit Cash amount -
|
|
Withdrawal Cash amount -
|
|
Fee Cash amount -
|
|
Interest Cash amount -
|
|
Distribution Cash amount -
|
|
Buy Security amount - fee - tax +shares
|
|
Sell Security amount - fee - tax -shares
|
|
Reinvestment_Distribution Security amount - fee - tax +shares
|
|
Corporate action Security NONE shares as printed
|
|
Security transfer Security NONE shares as printed
|
|
|
|
A cash row's amount is the money that actually settled and is already net of
|
|
the tax the broker withheld or refunded, so its fee and tax columns are recorded
|
|
on the fact and never subtracted again. Subtracting them a second time
|
|
double-counts by exactly the tax figure. A security row's amount is a gross
|
|
pinned to shares times price. A corporate action or depot transfer quotes a
|
|
position valuation, not cash: treating it as money conjures or destroys it, and
|
|
a depot switch of a whole portfolio does that once per instrument.
|
|
|
|
The share column is signed only for corporate actions and depot transfers. Buys
|
|
and sells are unsigned and take their direction from the type. Both conventions
|
|
are resolved at import, once.
|
|
|
|
Every security row is checked against shares times price at full precision.
|
|
This is the only check that catches a lost decimal separator, and it cannot
|
|
catch one that was lost uniformly across a row: 1 x 25,795 and 1 x 25795 both
|
|
satisfy it. A price cross-check against an outside provider is the only remedy
|
|
and is deliberately not implemented.
|
|
|
|
Rejected whole, with the record number: an unknown status, an unknown type, an
|
|
assetType that disagrees with its type, a currency other than the account's, a
|
|
security row without an ISIN, an invalid ISIN, a signed buy or sell, a corporate
|
|
action or depot transfer carrying a fee or tax, and any failed arithmetic check.
|
|
A zero amount is accepted; it corrupts nothing, and a free share allocation is
|
|
legitimately priced at zero.
|
|
|
|
Money holds four decimal places and share counts hold eight. A reinvested
|
|
distribution is quoted to six, so its amount is rounded half away from zero and
|
|
the exact residue is reported in the import review and never hidden. A share
|
|
count beyond eight places is refused rather than truncated, because a holding is
|
|
verified against the broker's own figure.
|
|
|
|
Instruments are registered from the export, keyed by ISIN, with an ID derived
|
|
from the ISIN so re-importing never creates a second entry for one security. One
|
|
ISIN appears under several descriptions over the years and sometimes under the
|
|
ISIN itself; the most recent real description names it, and an import never
|
|
renames an instrument that already exists. The name is editable display text;
|
|
the ISIN is identity and cannot be changed.
|
|
|
|
A broker reuses one reference across every leg of an economic event: the cash
|
|
and position sides of a corporate action arrive with the same reference byte for
|
|
byte, and the position leg's zero amount does not even differ in direction.
|
|
Transaction identity therefore includes the event and its instrument. The
|
|
reference itself also embeds an account-level identifier that repeats across
|
|
unrelated events, so it is evidence of an event, never of a transaction.
|
|
|
|
Broker facts carry enrichment kind "investment". Like a transfer it has no
|
|
category and no merchant, it is excluded from spending and income analytics and
|
|
from bulk reclassification, and the AI never sees it. Crucially, a broker fact
|
|
never reaches the sign-based fallback, so an unmatched deposit is not income and
|
|
a broker fee is not household spending. Analytical postings route it to
|
|
clearing:investments, where the residue left behind is exactly the cash an
|
|
investment account has returned: distributions and interest received, less fees.
|
|
|
|
Wealth and reconciliation
|
|
-------------------------
|
|
The Wealth page reports, per account, the cash balance as every recorded
|
|
movement summed, the positions as every signed share count summed, and named
|
|
checks. It is computed from the journal, not from the DuckDB index, because it
|
|
exists to be compared with the figures a bank or broker shows on its own screen.
|
|
|
|
A cash balance equals the real balance only when the journal holds that
|
|
account's complete history. A broker export does; a date-windowed bank statement
|
|
does not.
|
|
|
|
Checks that fail mean the journal disagrees with itself: row arithmetic, cash
|
|
never negative, holdings never negative. A negative holding means a position was
|
|
closed that was never opened in the imported data, so the export is partial or a
|
|
sign is wrong. Checks that only note: fee and tax recorded but not applied, and
|
|
deposits or withdrawals with no counterpart in another account.
|
|
|
|
Out of scope, deliberately: market prices, market value, net worth over time,
|
|
FIFO lot accounting, realised gains, Vorabpauschale, and currency conversion. A
|
|
position's "invested" figure is cash in less cash out, not a cost basis: a depot
|
|
transfer moves a position with no cash at all, and a sale returns cash without
|
|
identifying which lot it closed.
|
|
|
|
Canonical files and recovery
|
|
----------------------------
|
|
finance/
|
|
config.toml model/amount/import opt-in only, no API keys
|
|
accounts.finance
|
|
categories.finance
|
|
tags.finance
|
|
merchants.finance
|
|
instruments.finance
|
|
journal/YYYY/YYYY-MM.finance
|
|
state/sync-state.json sensitive local consent/session metadata
|
|
state/openrouter.json sensitive UI-managed OpenRouter key or explicit disable
|
|
state/enablebanking.json sensitive UI-managed banking key/configuration or disable
|
|
cache/finance.duckdb disposable analytical projection
|
|
|
|
The custom grammar is deliberately small:
|
|
category {
|
|
id: "cat_example"
|
|
name: "Groceries"
|
|
parent_id: "cat_expenses"
|
|
kind: "expense"
|
|
}
|
|
A transaction block has facts: {...} and enrichment: {...} JSON-valued fields.
|
|
A broker fact additionally carries an investment: {...} object holding the
|
|
event, instrument, signed quantity, price, gross, fee and tax; absent fields are
|
|
omitted, and its presence is what marks a fact as a broker fact.
|
|
Financial amounts are quoted decimal strings, never binary floating point.
|
|
Up to four fractional digits are supported; arithmetic uses exact ten-thousandths
|
|
with explicit overflow checks. DuckDB stores DECIMAL(24,4).
|
|
Share quantities are quoted decimal strings with up to eight fractional digits,
|
|
arithmetic uses exact hundred-millionths, and a quantity times a price is
|
|
multiplied at 128-bit width before rounding back to four places.
|
|
|
|
Each block starts with account/category/tag/merchant/instrument/transaction and
|
|
'{' on its own line; fields use name: JSON. Strings use JSON escaping (including
|
|
\n for multiline descriptions). JSON values may span lines. Blank lines and
|
|
full-line # or // comments are accepted between fields/blocks. Unknown fields,
|
|
duplicate keys, malformed records, invalid references and taxonomy cycles are
|
|
rejected.
|
|
The grammar is version-one strict: extension/split fields are not accepted yet.
|
|
Future format extensions require an explicit parser migration.
|
|
|
|
Stable category IDs survive renaming and moving; assigned categories must remain
|
|
leaves. Built-in roots and fallback leaves are protected. Move assigned records
|
|
to another leaf before adding children to their former category. Category
|
|
merges migrate referenced transactions/defaults; tag merges deduplicate links;
|
|
tag deletion removes all affected links after UI confirmation. Merchant merging
|
|
migrates transactions and retains source names/aliases on the target merchant.
|
|
|
|
Unchanged blocks and comments retain their text. UI enrichment edits do not
|
|
rewrite imported facts. File hashes detect external edits; invalid files stop
|
|
loading/indexing with a file/line error, not a partially refreshed dashboard.
|
|
A process lock prevents multiple app writers; use one instance per finance dir.
|
|
Revision conflicts require refreshing/re-previewing, not blind overwriting.
|
|
|
|
Multi-file writes are recoverable and logically atomic within the application.
|
|
Do not run external writers during a commit; external tools do not participate
|
|
in the app's process lock. Stop the service for manual bulk edits or backups.
|
|
If a write is interrupted, keep all state files and restart for recovery before
|
|
editing the journal manually. Keep backups of the entire canonical directory,
|
|
including hidden/state recovery files, plus the separately stored secrets.
|
|
The cache can be excluded. Avoid exposing any financial directory through a
|
|
static file server, Git public remote, or unencrypted shared backup.
|
|
|
|
Rebuild from text:
|
|
./bin/finance-duck -data ./finance -rebuild
|
|
Stop the running app before using that command (single writer lock), or use
|
|
Settings -> Rebuild index while it runs. A broken/deleted DuckDB file can be
|
|
removed while stopped and regenerated; it never contains the only copy of
|
|
financial records. When an index rebuild fails, UI mutations still preserve
|
|
canonical data and the index error is surfaced rather than serving stale totals.
|
|
|
|
Reclassification
|
|
----------------
|
|
AI / Classification: choose dates, model and independent Merchant/Category/Tags
|
|
fields. Analyse produces a read-only preview. Apply all/selected writes all
|
|
selected changes in one canonical commit; financial facts never change. A
|
|
manual edit, external journal change or taxonomy change invalidates old previews.
|
|
Previews are kept in memory for up to one hour and disappear on restart. Cancel
|
|
writes nothing. Transfers and broker facts are skipped, and unselected fields
|
|
are preserved.
|
|
Failed rows remain unchanged and are listed separately from proposed changes.
|
|
|
|
Boundaries and verification
|
|
---------------------------
|
|
There are no splits, budgets, tax/invoice/receipt processing, login/multi-user
|
|
support, arbitrary SQL or natural-language query execution. Investment support
|
|
covers positions and cash, not valuation: no market prices, market value,
|
|
net worth over time, FIFO lots, realised gains, Vorabpauschale or currency
|
|
conversion.
|
|
Natural-language query DSL and Sankey exploration remain explicitly later work.
|
|
There is no browser-to-bank credential handling or payment initiation.
|
|
|
|
Automated tests exercise deterministic financial invariants and mock remote
|
|
provider HTTP behavior. They do not replace testing real consent renewals and
|
|
real booked transaction samples for your banks. No real user credentials or
|
|
financial history are included in the repository. Test fixtures are synthetic.
|
|
|
|
API documentation sources:
|
|
https://enablebanking.com/docs/api/reference/
|
|
https://openrouter.ai/docs/guides/features/structured-outputs
|
|
https://openrouter.ai/docs/guides/features/zdr
|
|
https://openrouter.ai/docs/guides/routing/provider-selection
|