lars/finance-duck
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Import Trade Republic exports, whose conventions invert Scalable's

Browse Source 
A second broker export is recognized locally, by its full column set, and read
through the same pipeline: detection and parsing now dispatch on the format, so
the upload path, the review dialog, deduplication, the journal and the Wealth
report are unchanged. Its nine row types cover cash transfers, interest,
dividends, tax settlements and trades in funds, shares and crypto; none of them
moves a position without moving cash, so the cash-neutral class that Scalable's
corporate actions belong to does not arise here.

Three of its conventions are the opposite of the export already supported, and
reading any of them the other way round moves money. Fee and tax are the signed
adjustments it made to the cash rather than deductions from a gross, so a one
euro order fee arrives as -1.00 and is negated at import; the journal keeps one
convention and the domain never learns that two exist. A cash row's amount is
the gross, not the net, so interest of 16.46 with -4.33 of tax credits 12.13 -
where the other export states its cash already net and its tax is recorded and
never applied. Whether a cash row carries a gross now decides which of those it
was, which also makes the first kind's settlement checkable and stops the Wealth
report from claiming a figure was left unapplied when it was not. And a
TAX_OPTIMIZATION row puts zero in the amount column and its money in the tax
column, signed both ways: read as cash, all six in a real export move nothing.

Two more rows lie about their own columns. A dividend fills the share column
with the holding the dividend was paid on, not with a position change, so adding
it would double the holding. Crypto carries a bare ticker in the symbol column
and its ISIN-shaped identifier only in the description, so the identifier is
taken from the symbol when that is an ISIN and otherwise from the one the
description names; a position row resolving to neither is refused rather than
attached to a guess.

The shares-times-price check now holds a gross to the precision the export
stated it at rather than to four places. This export prints the notional rounded
to cents, and 29 of 59 real trades do not land on a whole cent: demanding
exactness rejected half a portfolio. One unit of the stated precision is still
four orders of magnitude tighter than the misplaced separator the check exists
to catch, and where an export prints the full product the check stays exact.

A unit price moves from money to the eight-place quantity type, because a crypto
price is quoted to six and rounding it would break the check the amount is
verified against. Trailing zeros are dropped before any precision test: this
export pads a six-place price to ten, and the padding would otherwise exhaust
the precision the value needs.

A transfer's counterparty comes from the export's own IBAN column when it has
one, from the IBAN the description names in parentheses when it does not, and
from the account's configured settlement IBAN when neither names anything. Free
text contributes only a value shaped like an IBAN. Without this, 108 transfers
stay unpaired and their bank-side counterparts read as spending and income.

Verified end to end against a real export: 26 rows import to a cash balance of
32187.02 matching the figure computed by hand from the source rows, all four
positions close at exactly zero, and every trade satisfies its own arithmetic.
This commit is contained in:
Lars Nolden
2026-09-11 23:04:22 +02:00
parent 87f052a3ea
commit 762ad3fae5
12 changed files with 1059 additions and 288 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/banking/scalable.go
+15 -216
View File
@@ -39,64 +39,9 @@ var scalableEvents = map[string]string{
"security transfer": domain.EventPositionTransfer,
}
// ScalableNote records a figure the export carried that the import deliberately
// did not apply, so it can be reviewed before confirming and recognized later
// if a balance disagrees.
type ScalableNote struct {
Record int `json:"record"`
Date string `json:"date"`
Description string `json:"description"`
Fee domain.Money `json:"fee,omitempty"`
Tax domain.Money `json:"tax,omitempty"`
}
// ScalableImport is a read broker export awaiting review.
type ScalableImport struct {
Facts []domain.Facts `json:"-"`
// Instruments are securities the export named that the registry does not
// hold yet. An import never renames an existing instrument: the name is
// editable display text, and the export's own description for one ISIN
// changes over time.
Instruments []domain.Instrument `json:"instruments"`
// Cancelled counts rows the broker did not execute. Their money and share
// columns are all zeros, so they satisfy every arithmetic check and would
// otherwise import as phantom trades.
Cancelled int `json:"cancelled"`
// Rounded counts rows whose money carried more than four decimal places,
// and Rounding is the exact total adjustment that rounding applied, at
// whatever precision the export used.
Rounded int `json:"rounded"`
Rounding string `json:"rounding"`
// Unapplied lists cash rows carrying a fee or tax. A broker cash amount is
// already net of them, so subtracting them again would double-count; they
// are recorded on the fact and reported here.
Unapplied []ScalableNote `json:"unapplied"`
}
// DetectScalableCSV reports whether a document is a Scalable Capital export and
// which 1-based record holds its header.
func DetectScalableCSV(f CSVFile) (header int, ok bool) {
for i, row := range f.rows {
if i >= maxCSVPreambleRows {
break
}
columns, usable := csvColumnIndex(row)
if !usable || len(columns) != len(scalableColumns) {
continue
}
matched := true
for _, name := range scalableColumns {
if _, exists := columns[name]; !exists {
matched = false
break
}
}
if matched {
return i + 1, true
}
}
return 0, false
}
// DetectScalableCSV reports whether a document is a Scalable Capital export
// and which 1-based record holds its header.
func DetectScalableCSV(f CSVFile) (header int, ok bool) { return matchColumns(f, scalableColumns) }
// ParseScalableCSV converts a broker export into bank facts carrying position
// legs.
@@ -125,26 +70,16 @@ func DetectScalableCSV(f CSVFile) (header int, ok bool) {
// check, rejects the whole file. Every one of those cases can move money, and a
// partially imported broker history cannot be told from a truncated export
// afterwards.
func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instrument) (ScalableImport, error) {
// Empty rather than nil: these are arrays in the reviewed JSON, and a null
// where a caller expects a list is a bug waiting on a different machine.
result := ScalableImport{Instruments: []domain.Instrument{}, Unapplied: []ScalableNote{}}
if account.ID == "" {
return result, errors.New("broker import requires a selected account")
}
if !account.Investing() {
return result, fmt.Errorf("account %q must be an investment account to hold a broker export", account.DisplayName)
func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instrument) (BrokerImport, error) {
result := newBrokerImport()
if err := investmentTarget(account); err != nil {
return result, err
}
header, ok := DetectScalableCSV(f)
if !ok {
return result, errors.New("not a Scalable Capital export")
}
headers := f.rows[header-1]
index := map[string]int{}
for i, raw := range headers {
index[headerName(raw)] = i
}
cell := func(row []string, name string) string { return strings.TrimSpace(row[index[name]]) }
headers, cell := brokerColumnIndex(f, header)
instruments := map[string]domain.Instrument{}
byISIN := map[string]domain.Instrument{}
@@ -216,15 +151,15 @@ func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instr
result.Instruments[slot].Name = description
}
}
amount, amountDrift, err := scalableMoney(cell(row, "amount"))
amount, amountDrift, err := brokerMoney(cell(row, "amount"), decimalGerman)
if err != nil {
return result, fmt.Errorf("broker record %d has an invalid amount %q: %w", record, cell(row, "amount"), err)
}
fee, feeDrift, err := scalableMoney(cell(row, "fee"))
fee, feeDrift, err := brokerMoney(cell(row, "fee"), decimalGerman)
if err != nil {
return result, fmt.Errorf("broker record %d has an invalid fee %q: %w", record, cell(row, "fee"), err)
}
tax, taxDrift, err := scalableMoney(cell(row, "tax"))
tax, taxDrift, err := brokerMoney(cell(row, "tax"), decimalGerman)
if err != nil {
return result, fmt.Errorf("broker record %d has an invalid tax %q: %w", record, cell(row, "tax"), err)
}
@@ -235,24 +170,21 @@ func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instr
cash := amount
if investment.CashOnly() {
if nonzeroMoney(fee) || nonzeroMoney(tax) {
result.Unapplied = append(result.Unapplied, ScalableNote{Record: record, Date: booking, Description: description, Fee: fee, Tax: tax})
result.Unapplied = append(result.Unapplied, BrokerNote{Record: record, Date: booking, Description: description, Fee: fee, Tax: tax})
}
investment.Fee, investment.Tax = fee, tax
} else {
if isin == "" {
return result, fmt.Errorf("broker record %d moves a position without an ISIN", record)
}
shares, err := scalableQuantity(cell(row, "shares"))
shares, err := brokerQuantity(cell(row, "shares"), decimalGerman)
if err != nil {
return result, fmt.Errorf("broker record %d has an invalid share count %q: %w", record, cell(row, "shares"), err)
}
price, priceDrift, err := scalableMoney(cell(row, "price"))
price, err := brokerQuantity(cell(row, "price"), decimalGerman)
if err != nil {
return result, fmt.Errorf("broker record %d has an invalid price %q: %w", record, cell(row, "price"), err)
}
if priceDrift.Sign() != 0 {
return result, fmt.Errorf("broker record %d has a price %q beyond four decimal places", record, cell(row, "price"))
}
signed, err := scalableSignedShares(event, shares)
if err != nil {
return result, fmt.Errorf("broker record %d: %w", record, err)
@@ -265,7 +197,7 @@ func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instr
cash = "0.00"
} else {
investment.Fee, investment.Tax = fee, tax
if cash, err = scalableSettlement(amount, fee, tax); err != nil {
if cash, err = brokerSettlement(amount, fee, tax); err != nil {
return result, fmt.Errorf("broker record %d: %w", record, err)
}
}
@@ -294,118 +226,6 @@ func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instr
return result, nil
}
// nonzeroMoney reports a figure that could change a balance. The export leaves
// a column blank where it does not apply and writes an explicit zero where it
// applies but is nil; only the second kind is worth putting in front of
// someone before they confirm an import.
func nonzeroMoney(m domain.Money) bool {
minor, err := m.Minor()
return err == nil && minor != 0
}
// residueScale is the precision the discarded remainder is accumulated at.
// A broker amount is its share count times its price, so it carries as many
// decimal places as the two together need: a real export reinvests to nine.
// Eighteen is far past anything a settlement can produce and still exact.
const residueScale = 18
// scalableMoney reads one German-formatted money cell, rounds it to money's
// four decimal places half away from zero, and returns the exact remainder
// that rounding discarded, in units of 1e-18. The remainder is reported rather
// than hidden, and never guessed at: it is the only honest account of why a
// computed balance can differ from the broker's by a fraction of a cent.
//
// An empty cell is empty money, not zero: blank marks a column that does not
// apply to the row.
func scalableMoney(value string) (domain.Money, *big.Int, error) {
plain, ok, err := scalablePlain(value)
if !ok || err != nil {
return "", new(big.Int), err
}
magnitude, negative, err := scalableDigits(plain)
if err != nil {
return "", new(big.Int), err
}
// One money place is 1e14 residue units. Rounding compares twice the
// remainder against that, so a tie rounds away from zero.
place := new(big.Int).Exp(big.NewInt(10), big.NewInt(residueScale-4), nil)
rounded, remainder := new(big.Int).QuoRem(magnitude, place, new(big.Int))
if new(big.Int).Lsh(remainder, 1).Cmp(place) >= 0 {
rounded.Add(rounded, big.NewInt(1))
}
if !rounded.IsInt64() {
return "", new(big.Int), fmt.Errorf("value is out of range for money")
}
residue := new(big.Int).Sub(magnitude, new(big.Int).Mul(rounded, place))
minor := rounded.Int64()
if negative {
minor, residue = -minor, residue.Neg(residue)
}
return domain.FormatMoney(minor), residue, nil
}
// scalableDigits splits a plain decimal string into its exact magnitude in
// residue units and its sign.
func scalableDigits(plain string) (magnitude *big.Int, negative bool, err error) {
digits := plain
if rest, cut := strings.CutPrefix(digits, "-"); cut {
negative, digits = true, rest
}
whole, decimals, _ := strings.Cut(digits, ".")
if whole == "" {
return nil, false, fmt.Errorf("decimal needs a digit before the separator")
}
if len(decimals) > residueScale {
return nil, false, fmt.Errorf("more than %d fractional digits", residueScale)
}
scaled, ok := new(big.Int).SetString(whole+decimals+strings.Repeat("0", residueScale-len(decimals)), 10)
if !ok {
return nil, false, fmt.Errorf("not a decimal number")
}
return scaled, negative, nil
}
// decimalString renders exact units at a scale without trailing zeros, so an
// adjustment of 1e-9 is reported as such rather than padded to eighteen places.
func decimalString(units *big.Int, scale int) string {
sign := ""
magnitude := new(big.Int).Abs(units)
if units.Sign() < 0 {
sign = "-"
}
digits := magnitude.String()
if len(digits) <= scale {
digits = strings.Repeat("0", scale+1-len(digits)) + digits
}
whole, fraction := digits[:len(digits)-scale], strings.TrimRight(digits[len(digits)-scale:], "0")
if fraction == "" {
return sign + whole
}
return sign + whole + "." + fraction
}
// scalableQuantity reads one German-formatted share count. Nothing is rounded:
// a holding is verified against the broker's own figure, so a count beyond
// eight decimal places is refused instead of silently truncated.
func scalableQuantity(value string) (domain.Quantity, error) {
plain, ok, err := scalablePlain(value)
if !ok || err != nil {
return "", err
}
return domain.ParseQuantity(plain)
}
// scalablePlain normalizes one numeric cell to a plain decimal string, or
// reports that the cell was blank.
func scalablePlain(value string) (string, bool, error) {
value = strings.NewReplacer("\u00a0", "", "\u202f", "", "'", "").Replace(strings.TrimSpace(value))
if value == "" {
return "", false, nil
}
plain, err := germanDecimal(value)
return plain, err == nil, err
}
// scalableSignedShares resolves the export's two sign conventions. A buy, sell
// or reinvestment carries an unsigned count and takes its direction from the
// type; a corporate action or depot transfer is already signed.
@@ -428,24 +248,3 @@ func scalableSignedShares(event string, shares domain.Quantity) (domain.Quantity
}
return domain.FormatQuantity(units), nil
}
// scalableSettlement is gross minus fee minus tax: the cash a trade moved. The
// broker states fee and tax as positive deductions whichever way the trade
// went, so both are subtracted from a signed gross.
func scalableSettlement(gross, fee, tax domain.Money) (domain.Money, error) {
total := int64(0)
for _, deduction := range []struct {
sign int64
money domain.Money
}{{1, gross}, {-1, fee}, {-1, tax}} {
if deduction.money == "" {
continue
}
minor, err := deduction.money.Minor()
if err != nil {
return "", err
}
total += deduction.sign * minor
}
return domain.FormatMoney(total), nil
}