lars/finance-duck
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Track investments as broker facts with a position leg

Browse Source 
An account now has a kind, and an investment account holds positions as well as
cash. A broker row is not a new entity: it is a bank fact with an optional
position leg, so deduplication, the journal, fact immutability, the DuckDB
projection and the transactions view carry it unchanged. Facts.Amount stays the
cash leg and is zero on the rows that move only a position.

Scalable Capital exports are recognized locally as a fourth format, read by
their own parser because a column mapping cannot describe them: the amount
column is settled cash on a cash row, a gross to be netted on a trade, and a
position valuation that must never touch cash on a corporate action or a depot
transfer. A cash amount is already net of the tax the broker withheld or
refunded, so that tax is recorded on the fact and never subtracted a second
time; treating a corporate action's valuation as money conjures cash, and a
depot switch would do it once per instrument. The share column is signed only
for those two types, so buys and sells take their direction from the type. Every
security row is checked against shares times price at 128-bit width, because a
lost decimal separator survives every other check. An unknown status, type or
assetType, a foreign currency, a missing ISIN, or one failed check rejects the
whole file with the record number.

Instruments live in instruments.finance, keyed by ISIN with an ID derived from
it, so re-importing never registers a security twice. One ISIN appears under
several broker descriptions over the years and sometimes under the ISIN itself:
the most recent real description names it, and an import never renames one that
already exists. A broker also reuses a single reference across every leg of one
event, so transaction identity includes the event and its instrument.

domain.Fallback returns kind "investment" for any fact carrying a position leg,
so no broker row reaches the sign-based branch. That single rule is what stops
an unmatched deposit from counting as income and a broker fee from counting as
household spending; the monthly PRIME fee and its matching credit now cancel in
clearing:investments with no configuration at all. Investment rows are excluded
from spending analytics, from bulk reclassification and from the model, exactly
as transfers are.

Equal competing transfers are paired instead of skipped. Every connected
component of the candidate graph is a complete bipartite graph between two fixed
accounts at one amount and currency, so every pairing produces the same
accounts, kinds and postings and only the displayed counterpart differs.
Refusing to choose was the expensive option: both legs fell through to the
sign-based fallback and appeared as spending and income that never happened.
Pairing follows the nearest booking date, then the transaction ID, so iteration
order decides nothing. POST /api/transactions/{id}/transfer rewrites the old and
the new pair in one commit, because reciprocity is validated and a half-applied
link is an invalid dataset, and the matcher now skips any record classified
manually so a hand-made link or unlink outlives the next import.

Wealth reports each account's cash and positions from the journal rather than
the index, with named checks - row arithmetic, cash never negative, holdings
never negative - because it exists to be compared against the figures a broker
shows on its own screen. A negative holding means the imported history is
partial. Share counts are exact to eight places; a reinvested distribution
quoted to six is rounded to money's four and the residue is reported rather than
hidden. Market prices, market value, net worth over time, FIFO lot accounting,
realised gains and currency conversion are deliberately absent.
This commit is contained in:
Lars Nolden
2026-09-11 21:58:47 +02:00
parent 673cbf917b
commit 922ae507bd
27 changed files with 3071 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files
OPERATIONS.txt
+147 -13
View File
@@ -351,7 +351,8 @@ mandate reference must never become a transaction identity. ING facts likewise
carry no reference. Review the previewed dates, amount signs and currency before
confirming; a wrong mapping is visible there, not after import.
Import sources: n26_csv, ing_csv, kontist_csv, csv (AI-mapped), enablebanking.
Import sources: n26_csv, ing_csv, kontist_csv, scalable_csv, csv (AI-mapped),
enablebanking.
Stable provider entry references are scoped by account, source and debit/credit
direction: a debit and credit can share a reference without being collapsed.
@@ -368,10 +369,131 @@ and reconcile the input locally before retrying. Facts are never silently
replaced when upstream descriptions or amounts change for an existing identity.
Transfers use reciprocal records from different owned accounts, equal/opposite
exact amounts and matching currency, with own-IBAN evidence and unambiguous
matching. Ambiguous pairs are not guessed. The linked records remain separate
immutable facts; analytical double-entry postings balance and transfers do not
count as income/spending. Populate local account IBANs to support recognition.
exact amounts and matching currency, with own-IBAN evidence and booking dates
within three calendar days. Equal competing payments ARE paired, by nearest
booking date and then by transaction ID: every candidate set is a complete
bipartite graph between two fixed accounts at one amount and currency, so every
pairing yields the same accounts, kinds and postings, and iteration order
decides nothing. Leaving them unpaired was the worse option, because both legs
then fell through to the sign-based fallback and appeared as spending and income
that never happened. An existing link is never revisited, and neither is a
record whose classification source is "manual": a hand-made link or unlink
outlives every later import. The linked records remain separate immutable
facts; analytical double-entry postings balance and transfers do not count as
income/spending. Populate local account IBANs to support recognition.
Investment accounts and broker imports
--------------------------------------
An account has a kind, "cash" (the default, and what an absent kind means) or
"investment". An investment account holds a cash balance and positions. It also
carries a settlement IBAN (reference_iban): a broker export has no counterparty
column, so deposits and withdrawals are stamped with that IBAN and pair with the
funding account through ordinary transfer matching. Leave it empty and those
rows simply stay unpaired, which costs accuracy in spending analysis but never
invents income.
Scalable Capital exports (scalable_csv) are recognized locally by their full
column set: date, time, status, reference, description, assetType, type, isin,
shares, price, amount, fee, tax, currency. The layout is matched whole, because
a row's meaning depends on the combination of status, assetType and type.
The booking date is the date column exactly as printed. Batch rows are stamped
midnight UTC rendered in local time, so their time column reads 01:00 in winter
and 02:00 in summer; reading date and time together would move half the year's
corporate actions and distributions to the previous day.
Only status "Executed" imports. A cancelled retry is all zeros, so it satisfies
every arithmetic check and would otherwise enter the journal as a phantom trade.
The ten row types, and what each settles:
type assetType cash position
Deposit Cash amount -
Withdrawal Cash amount -
Fee Cash amount -
Interest Cash amount -
Distribution Cash amount -
Buy Security amount - fee - tax +shares
Sell Security amount - fee - tax -shares
Reinvestment_Distribution Security amount - fee - tax +shares
Corporate action Security NONE shares as printed
Security transfer Security NONE shares as printed
A cash row's amount is the money that actually settled and is already net of
the tax the broker withheld or refunded, so its fee and tax columns are recorded
on the fact and never subtracted again. Subtracting them a second time
double-counts by exactly the tax figure. A security row's amount is a gross
pinned to shares times price. A corporate action or depot transfer quotes a
position valuation, not cash: treating it as money conjures or destroys it, and
a depot switch of a whole portfolio does that once per instrument.
The share column is signed only for corporate actions and depot transfers. Buys
and sells are unsigned and take their direction from the type. Both conventions
are resolved at import, once.
Every security row is checked against shares times price at full precision.
This is the only check that catches a lost decimal separator, and it cannot
catch one that was lost uniformly across a row: 1 x 25,795 and 1 x 25795 both
satisfy it. A price cross-check against an outside provider is the only remedy
and is deliberately not implemented.
Rejected whole, with the record number: an unknown status, an unknown type, an
assetType that disagrees with its type, a currency other than the account's, a
security row without an ISIN, an invalid ISIN, a signed buy or sell, a corporate
action or depot transfer carrying a fee or tax, and any failed arithmetic check.
A zero amount is accepted; it corrupts nothing, and a free share allocation is
legitimately priced at zero.
Money holds four decimal places and share counts hold eight. A reinvested
distribution is quoted to six, so its amount is rounded half away from zero and
the exact residue is reported in the import review and never hidden. A share
count beyond eight places is refused rather than truncated, because a holding is
verified against the broker's own figure.
Instruments are registered from the export, keyed by ISIN, with an ID derived
from the ISIN so re-importing never creates a second entry for one security. One
ISIN appears under several descriptions over the years and sometimes under the
ISIN itself; the most recent real description names it, and an import never
renames an instrument that already exists. The name is editable display text;
the ISIN is identity and cannot be changed.
A broker reuses one reference across every leg of an economic event: the cash
and position sides of a corporate action arrive with the same reference byte for
byte, and the position leg's zero amount does not even differ in direction.
Transaction identity therefore includes the event and its instrument. The
reference itself also embeds an account-level identifier that repeats across
unrelated events, so it is evidence of an event, never of a transaction.
Broker facts carry enrichment kind "investment". Like a transfer it has no
category and no merchant, it is excluded from spending and income analytics and
from bulk reclassification, and the AI never sees it. Crucially, a broker fact
never reaches the sign-based fallback, so an unmatched deposit is not income and
a broker fee is not household spending. Analytical postings route it to
clearing:investments, where the residue left behind is exactly the cash an
investment account has returned: distributions and interest received, less fees.
Wealth and reconciliation
-------------------------
The Wealth page reports, per account, the cash balance as every recorded
movement summed, the positions as every signed share count summed, and named
checks. It is computed from the journal, not from the DuckDB index, because it
exists to be compared with the figures a bank or broker shows on its own screen.
A cash balance equals the real balance only when the journal holds that
account's complete history. A broker export does; a date-windowed bank statement
does not.
Checks that fail mean the journal disagrees with itself: row arithmetic, cash
never negative, holdings never negative. A negative holding means a position was
closed that was never opened in the imported data, so the export is partial or a
sign is wrong. Checks that only note: fee and tax recorded but not applied, and
deposits or withdrawals with no counterpart in another account.
Out of scope, deliberately: market prices, market value, net worth over time,
FIFO lot accounting, realised gains, Vorabpauschale, and currency conversion. A
position's "invested" figure is cash in less cash out, not a cost basis: a depot
transfer moves a position with no cash at all, and a sale returns cash without
identifying which lot it closed.
Canonical files and recovery
----------------------------
@@ -381,6 +503,7 @@ finance/
categories.finance
tags.finance
merchants.finance
instruments.finance
journal/YYYY/YYYY-MM.finance
state/sync-state.json sensitive local consent/session metadata
state/openrouter.json sensitive UI-managed OpenRouter key or explicit disable
@@ -395,15 +518,22 @@ The custom grammar is deliberately small:
kind: "expense"
}
A transaction block has facts: {...} and enrichment: {...} JSON-valued fields.
A broker fact additionally carries an investment: {...} object holding the
event, instrument, signed quantity, price, gross, fee and tax; absent fields are
omitted, and its presence is what marks a fact as a broker fact.
Financial amounts are quoted decimal strings, never binary floating point.
Up to four fractional digits are supported; arithmetic uses exact ten-thousandths
with explicit overflow checks. DuckDB stores DECIMAL(24,4).
Share quantities are quoted decimal strings with up to eight fractional digits,
arithmetic uses exact hundred-millionths, and a quantity times a price is
multiplied at 128-bit width before rounding back to four places.
Each block starts with account/category/tag/merchant/transaction and '{' on its
own line; fields use name: JSON. Strings use JSON escaping (including \n for
multiline descriptions). JSON values may span lines. Blank lines and full-line
# or // comments are accepted between fields/blocks. Unknown fields, duplicate
keys, malformed records, invalid references and taxonomy cycles are rejected.
Each block starts with account/category/tag/merchant/instrument/transaction and
'{' on its own line; fields use name: JSON. Strings use JSON escaping (including
\n for multiline descriptions). JSON values may span lines. Blank lines and
full-line # or // comments are accepted between fields/blocks. Unknown fields,
duplicate keys, malformed records, invalid references and taxonomy cycles are
rejected.
The grammar is version-one strict: extension/split fields are not accepted yet.
Future format extensions require an explicit parser migration.
@@ -444,13 +574,17 @@ fields. Analyse produces a read-only preview. Apply all/selected writes all
selected changes in one canonical commit; financial facts never change. A
manual edit, external journal change or taxonomy change invalidates old previews.
Previews are kept in memory for up to one hour and disappear on restart. Cancel
writes nothing. Transfers are skipped, and unselected fields are preserved.
writes nothing. Transfers and broker facts are skipped, and unselected fields
are preserved.
Failed rows remain unchanged and are listed separately from proposed changes.
Boundaries and verification
---------------------------
There are no splits, budgets, investments, tax/invoice/receipt processing,
login/multi-user support, arbitrary SQL or natural-language query execution.
There are no splits, budgets, tax/invoice/receipt processing, login/multi-user
support, arbitrary SQL or natural-language query execution. Investment support
covers positions and cash, not valuation: no market prices, market value,
net worth over time, FIFO lots, realised gains, Vorabpauschale or currency
conversion.
Natural-language query DSL and Sankey exploration remain explicitly later work.
There is no browser-to-bank credential handling or payment initiation.