lars/finance-duck
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Track investments as broker facts with a position leg

Browse Source 
An account now has a kind, and an investment account holds positions as well as
cash. A broker row is not a new entity: it is a bank fact with an optional
position leg, so deduplication, the journal, fact immutability, the DuckDB
projection and the transactions view carry it unchanged. Facts.Amount stays the
cash leg and is zero on the rows that move only a position.

Scalable Capital exports are recognized locally as a fourth format, read by
their own parser because a column mapping cannot describe them: the amount
column is settled cash on a cash row, a gross to be netted on a trade, and a
position valuation that must never touch cash on a corporate action or a depot
transfer. A cash amount is already net of the tax the broker withheld or
refunded, so that tax is recorded on the fact and never subtracted a second
time; treating a corporate action's valuation as money conjures cash, and a
depot switch would do it once per instrument. The share column is signed only
for those two types, so buys and sells take their direction from the type. Every
security row is checked against shares times price at 128-bit width, because a
lost decimal separator survives every other check. An unknown status, type or
assetType, a foreign currency, a missing ISIN, or one failed check rejects the
whole file with the record number.

Instruments live in instruments.finance, keyed by ISIN with an ID derived from
it, so re-importing never registers a security twice. One ISIN appears under
several broker descriptions over the years and sometimes under the ISIN itself:
the most recent real description names it, and an import never renames one that
already exists. A broker also reuses a single reference across every leg of one
event, so transaction identity includes the event and its instrument.

domain.Fallback returns kind "investment" for any fact carrying a position leg,
so no broker row reaches the sign-based branch. That single rule is what stops
an unmatched deposit from counting as income and a broker fee from counting as
household spending; the monthly PRIME fee and its matching credit now cancel in
clearing:investments with no configuration at all. Investment rows are excluded
from spending analytics, from bulk reclassification and from the model, exactly
as transfers are.

Equal competing transfers are paired instead of skipped. Every connected
component of the candidate graph is a complete bipartite graph between two fixed
accounts at one amount and currency, so every pairing produces the same
accounts, kinds and postings and only the displayed counterpart differs.
Refusing to choose was the expensive option: both legs fell through to the
sign-based fallback and appeared as spending and income that never happened.
Pairing follows the nearest booking date, then the transaction ID, so iteration
order decides nothing. POST /api/transactions/{id}/transfer rewrites the old and
the new pair in one commit, because reciprocity is validated and a half-applied
link is an invalid dataset, and the matcher now skips any record classified
manually so a hand-made link or unlink outlives the next import.

Wealth reports each account's cash and positions from the journal rather than
the index, with named checks - row arithmetic, cash never negative, holdings
never negative - because it exists to be compared against the figures a broker
shows on its own screen. A negative holding means the imported history is
partial. Share counts are exact to eight places; a reinvested distribution
quoted to six is rounded to money's four and the residue is reported rather than
hidden. Market prices, market value, net worth over time, FIFO lot accounting,
realised gains and currency conversion are deliberately absent.
This commit is contained in:
Lars Nolden
2026-09-11 21:58:47 +02:00
parent 673cbf917b
commit 922ae507bd
27 changed files with 3071 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/app/import.go
+72 -8
View File
@@ -35,12 +35,26 @@ func addProposal(d *domain.Dataset, p classification.Proposal) error {
}
return nil
}
func (a *App) importFacts(ctx context.Context, s State, facts []domain.Facts) (ImportResult, error) {
func (a *App) importFacts(ctx context.Context, s State, facts []domain.Facts, instruments []domain.Instrument) (ImportResult, error) {
// Instruments first: a broker fact references one, and the canonical
// dataset is validated as a whole, so a trade cannot be committed before
// the security it trades exists.
known := make(map[string]bool, len(s.Data.Instruments))
for _, v := range s.Data.Instruments {
known[v.ID] = true
}
registered := false
for _, v := range instruments {
if !known[v.ID] {
known[v.ID], registered = true, true
s.Data.Instruments = append(s.Data.Instruments, v)
}
}
added, err := banking.NormalizeAndDedupe(s.Data, facts)
if err != nil {
return ImportResult{}, err
}
if len(added) == 0 {
if len(added) == 0 && !registered {
return ImportResult{State: s}, nil
}
s.Data.Transactions = append(s.Data.Transactions, added...)
@@ -55,7 +69,7 @@ func (a *App) importFacts(ctx context.Context, s State, facts []domain.Facts) (I
ids[t.Facts.ID] = true
}
for i, t := range s.Data.Transactions {
if !ids[t.Facts.ID] || t.Enrichment.Kind == "transfer" {
if !ids[t.Facts.ID] || t.Enrichment.Kind == "transfer" || t.Enrichment.Kind == domain.KindInvestment {
continue
}
// With AI classification off for imports, no provider is contacted at
@@ -105,9 +119,16 @@ type CSVImport struct {
New int `json:"new"`
Duplicates int `json:"duplicates"`
Samples []domain.Facts `json:"samples"`
// Broker is present when the statement is a broker export. Its rows carry
// positions as well as cash, so they are read by a dedicated parser rather
// than by a column mapping, and the review needs to show what that parser
// decided: which securities it would register, which rows it skipped, and
// which figures it deliberately did not apply.
Broker *banking.ScalableImport `json:"broker,omitempty"`
facts []domain.Facts
created time.Time
facts []domain.Facts
instruments []domain.Instrument
created time.Time
}
const csvImportLifetime = time.Hour
@@ -141,6 +162,28 @@ func (a *App) PrepareCSVImport(ctx context.Context, rev, accountID string, r io.
return CSVImport{}, err
}
prepared := CSVImport{ID: domain.NewID("csvimport"), Revision: s.Revision, AccountID: account.ID, MappedBy: "preset", created: time.Now()}
// A broker export is recognized before any column mapping is attempted. Its
// rows are not interchangeable statement lines: the same amount column is
// cash on one row, a gross to be netted on another, and a position
// valuation that must not touch cash on a third, so a column mapping cannot
// describe it.
if header, broker := banking.DetectScalableCSV(file); broker {
read, e := banking.ParseScalableCSV(file, account, s.Data.Instruments)
if e != nil {
return CSVImport{}, e
}
added, e := banking.NormalizeAndDedupe(s.Data, read.Facts)
if e != nil {
return CSVImport{}, e
}
prepared.Source, prepared.SourceLabel = banking.SourceScalable, "Scalable Capital"
prepared.Mapping = banking.CSVMapping{HeaderRow: header, DateFormat: "yyyy-mm-dd", DecimalFormat: "comma"}
prepared.Columns = brokerColumns()
prepared.Records, prepared.New, prepared.Duplicates = len(read.Facts), len(added), len(read.Facts)-len(added)
prepared.Samples, prepared.facts, prepared.instruments = csvSamples(read.Facts), read.Facts, read.Instruments
prepared.Broker = &read
return a.retain(prepared)
}
mapping, source, label, recognized := banking.DetectCSVMapping(file)
if !recognized {
sample, e := file.Sample()
@@ -188,6 +231,12 @@ func (a *App) PrepareCSVImport(ctx context.Context, rev, accountID string, r io.
prepared.Columns = csvColumns(mapping, account)
prepared.Records, prepared.New, prepared.Duplicates = len(facts), len(added), len(facts)-len(added)
prepared.Samples, prepared.facts = csvSamples(facts), facts
return a.retain(prepared)
}
// retain holds a reviewed statement until it is confirmed or expires. Nothing
// is written to the journal here.
func (a *App) retain(prepared CSVImport) (CSVImport, error) {
a.mu.Lock()
defer a.mu.Unlock()
for id, old := range a.csvImports {
@@ -202,6 +251,21 @@ func (a *App) PrepareCSVImport(ctx context.Context, rev, accountID string, r io.
return prepared, nil
}
// brokerColumns describes what the broker parser decided, in the same
// reviewable shape as a column mapping. The dispatch is the part that can be
// wrong in a way that moves money, so it is the part shown.
func brokerColumns() []CSVColumn {
return []CSVColumn{
{Field: "Booking date", Column: "date, exactly as printed; the time column is local and crosses midnight, so it is ignored"},
{Field: "Imported rows", Column: `status "Executed" only; cancelled retries are all zeros and would import as phantom trades`},
{Field: "Cash movement", Column: "cash rows: amount, already net of tax; trades: amount − fee − tax; corporate actions and depot transfers: none"},
{Field: "Position change", Column: "shares, signed by type for buys and sells and exactly as printed for corporate actions and depot transfers"},
{Field: "Instrument", Column: "isin; the description only names it"},
{Field: "Reference", Column: "reference, which the broker reuses across every leg of one event"},
{Field: "Decimals", Column: "German: comma decimal, and a dot only groups thousands in exact three-digit runs"},
}
}
// ConfirmCSVImport imports exactly the facts that were previewed, provided the
// journal has not changed since.
func (a *App) ConfirmCSVImport(ctx context.Context, id, rev string) (ImportResult, error) {
@@ -221,7 +285,7 @@ func (a *App) ConfirmCSVImport(ctx context.Context, id, rev string) (ImportResul
if s.Revision != prepared.Revision {
return ImportResult{}, errors.New("revision conflict: data changed after the preview; upload the statement again")
}
result, err := a.importFacts(ctx, s, prepared.facts)
result, err := a.importFacts(ctx, s, prepared.facts, prepared.instruments)
if err != nil {
return ImportResult{}, err
}
@@ -414,7 +478,7 @@ func (a *App) Backfill(ctx context.Context, rev, accountID string, historyMonths
}
// Use normal import processing without changing sync cursors or saved consent
// settings, including when the requested range adds no transactions.
result, err := a.importFacts(ctx, s, facts)
result, err := a.importFacts(ctx, s, facts, nil)
if err != nil {
return ImportResult{}, err
}
@@ -779,7 +843,7 @@ func (a *App) Sync(ctx context.Context) (State, error) {
failures = append(failures, account.DisplayName+": "+meta.Error)
continue
}
result, e := a.importFacts(ctx, s, facts)
result, e := a.importFacts(ctx, s, facts, nil)
if e != nil {
failures = append(failures, account.DisplayName+": "+e.Error())
waiting = false