Track investments as broker facts with a position leg
An account now has a kind, and an investment account holds positions as well as
cash. A broker row is not a new entity: it is a bank fact with an optional
position leg, so deduplication, the journal, fact immutability, the DuckDB
projection and the transactions view carry it unchanged. Facts.Amount stays the
cash leg and is zero on the rows that move only a position.
Scalable Capital exports are recognized locally as a fourth format, read by
their own parser because a column mapping cannot describe them: the amount
column is settled cash on a cash row, a gross to be netted on a trade, and a
position valuation that must never touch cash on a corporate action or a depot
transfer. A cash amount is already net of the tax the broker withheld or
refunded, so that tax is recorded on the fact and never subtracted a second
time; treating a corporate action's valuation as money conjures cash, and a
depot switch would do it once per instrument. The share column is signed only
for those two types, so buys and sells take their direction from the type. Every
security row is checked against shares times price at 128-bit width, because a
lost decimal separator survives every other check. An unknown status, type or
assetType, a foreign currency, a missing ISIN, or one failed check rejects the
whole file with the record number.
Instruments live in instruments.finance, keyed by ISIN with an ID derived from
it, so re-importing never registers a security twice. One ISIN appears under
several broker descriptions over the years and sometimes under the ISIN itself:
the most recent real description names it, and an import never renames one that
already exists. A broker also reuses a single reference across every leg of one
event, so transaction identity includes the event and its instrument.
domain.Fallback returns kind "investment" for any fact carrying a position leg,
so no broker row reaches the sign-based branch. That single rule is what stops
an unmatched deposit from counting as income and a broker fee from counting as
household spending; the monthly PRIME fee and its matching credit now cancel in
clearing:investments with no configuration at all. Investment rows are excluded
from spending analytics, from bulk reclassification and from the model, exactly
as transfers are.
Equal competing transfers are paired instead of skipped. Every connected
component of the candidate graph is a complete bipartite graph between two fixed
accounts at one amount and currency, so every pairing produces the same
accounts, kinds and postings and only the displayed counterpart differs.
Refusing to choose was the expensive option: both legs fell through to the
sign-based fallback and appeared as spending and income that never happened.
Pairing follows the nearest booking date, then the transaction ID, so iteration
order decides nothing. POST /api/transactions/{id}/transfer rewrites the old and
the new pair in one commit, because reciprocity is validated and a half-applied
link is an invalid dataset, and the matcher now skips any record classified
manually so a hand-made link or unlink outlives the next import.
Wealth reports each account's cash and positions from the journal rather than
the index, with named checks - row arithmetic, cash never negative, holdings
never negative - because it exists to be compared against the figures a broker
shows on its own screen. A negative holding means the imported history is
partial. Share counts are exact to eight places; a reinvested distribution
quoted to six is rounded to money's four and the residue is reported rather than
hidden. Market prices, market value, net worth over time, FIFO lot accounting,
realised gains and currency conversion are deliberately absent.
This commit is contained in:
+32
-6
@@ -671,13 +671,11 @@ func parseMappedCSVDecimal(value, format string) (domain.Money, error) {
|
||||
case "dot-or-comma":
|
||||
return parseCSVAmount(value)
|
||||
case "comma":
|
||||
// A dot can only be grouping here, and only in exact thousands groups.
|
||||
if !strings.Contains(value, ",") && strings.Contains(value, ".") {
|
||||
if digits, ok := ungroup(value, "."); ok {
|
||||
value = digits
|
||||
}
|
||||
plain, err := germanDecimal(value)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return parseCSVAmount(value)
|
||||
return domain.ParseMoney(plain)
|
||||
case "dot":
|
||||
value = strings.TrimPrefix(value, "+")
|
||||
if strings.Contains(value, ",") {
|
||||
@@ -693,6 +691,34 @@ func parseMappedCSVDecimal(value, format string) (domain.Money, error) {
|
||||
}
|
||||
}
|
||||
|
||||
// germanDecimal rewrites a German-formatted number as a plain decimal string
|
||||
// without parsing it, so a caller can choose its own precision. A dot is only
|
||||
// grouping when every group is exactly three digits: "1.014" is 1014 while
|
||||
// "1.14" stays 1.14. Broker exports carry both shapes in one share column.
|
||||
func germanDecimal(value string) (string, error) {
|
||||
value = strings.TrimPrefix(strings.TrimSpace(value), "+")
|
||||
if strings.Contains(value, ",") {
|
||||
if strings.Count(value, ",") != 1 {
|
||||
return "", errors.New("invalid decimal separator")
|
||||
}
|
||||
whole, fraction, _ := strings.Cut(value, ",")
|
||||
if strings.Contains(whole, ".") {
|
||||
digits, ok := ungroup(whole, ".")
|
||||
if !ok {
|
||||
return "", errors.New("invalid grouping")
|
||||
}
|
||||
whole = digits
|
||||
}
|
||||
return whole + "." + fraction, nil
|
||||
}
|
||||
if strings.Contains(value, ".") {
|
||||
if digits, ok := ungroup(value, "."); ok {
|
||||
return digits, nil
|
||||
}
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
// ungroup removes thousands separators, and only when every group is exactly
|
||||
// three digits: "1.234" is 1234, while "1.23" stays a decimal value.
|
||||
func ungroup(value, separator string) (string, bool) {
|
||||
|
||||
+82
-23
@@ -25,13 +25,31 @@ func identity(f domain.Facts) string {
|
||||
if strings.HasPrefix(string(f.Amount), "-") {
|
||||
direction = "debit"
|
||||
}
|
||||
return digest(f.AccountID, f.Source, f.ExternalID, direction)
|
||||
return digest(f.AccountID, f.Source, f.ExternalID, direction, leg(f))
|
||||
}
|
||||
|
||||
// leg distinguishes the records of one broker event. A broker reuses a single
|
||||
// reference across every leg: the cash side of a corporate action and its
|
||||
// position side arrive with the same reference byte for byte, and a position
|
||||
// leg's zero amount does not even differ in direction. The event and its
|
||||
// instrument separate them without making money part of an identity, so a
|
||||
// corrected upstream figure is still reported rather than imported twice.
|
||||
func leg(f domain.Facts) string {
|
||||
if f.Investment == nil {
|
||||
return ""
|
||||
}
|
||||
return f.Investment.Event + "\x00" + f.Investment.InstrumentID
|
||||
}
|
||||
func fingerprint(f domain.Facts) string {
|
||||
return digest(f.AccountID, f.BookingDate, f.ValueDate, f.Amount.String(), f.Currency, strings.Join(strings.Fields(f.RawDescription), " "), strings.ToLower(strings.Join(strings.Fields(f.Counterparty), " ")), f.CounterpartyIBAN)
|
||||
inv := domain.Investment{}
|
||||
if f.Investment != nil {
|
||||
inv = *f.Investment
|
||||
}
|
||||
return digest(f.AccountID, f.BookingDate, f.ValueDate, f.Amount.String(), f.Currency, strings.Join(strings.Fields(f.RawDescription), " "), strings.ToLower(strings.Join(strings.Fields(f.Counterparty), " ")), f.CounterpartyIBAN,
|
||||
inv.Event, inv.InstrumentID, string(inv.Quantity), string(inv.Price), string(inv.Gross), string(inv.Fee), string(inv.Tax))
|
||||
}
|
||||
func looseFingerprint(f domain.Facts) string {
|
||||
return digest(f.AccountID, f.BookingDate, f.Amount.String(), f.Currency)
|
||||
return digest(f.AccountID, f.BookingDate, f.Amount.String(), f.Currency, leg(f))
|
||||
}
|
||||
func sameBookedMoney(a, b domain.Facts) bool {
|
||||
return a.AccountID == b.AccountID && a.BookingDate == b.BookingDate && a.Amount == b.Amount && a.Currency == b.Currency
|
||||
@@ -46,6 +64,8 @@ func sourceLabel(source string) string {
|
||||
return "ING CSV"
|
||||
case "kontist_csv":
|
||||
return "Kontist CSV"
|
||||
case SourceScalable:
|
||||
return "Scalable CSV"
|
||||
case "csv":
|
||||
return "mapped CSV"
|
||||
default:
|
||||
@@ -310,10 +330,21 @@ func normalizeFacts(f domain.Facts, accounts map[string]bool) (domain.Facts, err
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// MatchTransfers links only mutually unique candidates, with reciprocal own
|
||||
// IBANs, inverse exact money in one currency, and booking dates within 3 calendar
|
||||
// days. Existing manual links are retained. Ambiguous equal payments stay ordinary
|
||||
// transactions: iteration order must never decide which transfer gets linked.
|
||||
// MatchTransfers links own-account pairs with reciprocal own IBANs, inverse
|
||||
// exact money in one currency, and booking dates within 3 calendar days.
|
||||
//
|
||||
// Equal competing payments are paired by nearest booking date rather than left
|
||||
// alone. Every connected component of the candidate graph is a complete
|
||||
// bipartite graph between two fixed accounts at one amount and one currency:
|
||||
// an edge needs exactly inverse money, and a record's own counterparty IBAN
|
||||
// names exactly one other account. So every perfect matching produces the same
|
||||
// accounts, amounts, kinds and postings, and the only thing a choice decides is
|
||||
// which row displays as which one's counterpart. Refusing to choose is the
|
||||
// expensive option: both legs then fall through to the sign-based fallback and
|
||||
// show up as spending and income that never happened.
|
||||
//
|
||||
// Ordering is by date gap, then by transaction ID, so iteration order cannot
|
||||
// decide anything. Existing links and hand-made decisions are never revisited.
|
||||
func MatchTransfers(data *domain.Dataset) {
|
||||
if data == nil {
|
||||
return
|
||||
@@ -336,10 +367,28 @@ func MatchTransfers(data *domain.Dataset) {
|
||||
byAccount[id] = iban
|
||||
}
|
||||
}
|
||||
candidates := make([][]int, len(data.Transactions))
|
||||
matchable := func(t domain.Transaction) bool {
|
||||
if t.Enrichment.Kind == "transfer" || t.Enrichment.TransferPeerID != "" {
|
||||
return false
|
||||
}
|
||||
// A hand-made decision outlives the next import. Without this, an
|
||||
// operator who unlinks a pair that is not really a transfer watches the
|
||||
// matcher relink it on the following import, forever.
|
||||
if t.Enrichment.Classification.Source == "manual" {
|
||||
return false
|
||||
}
|
||||
// Only a broker cash movement can be a transfer leg; a trade's cash
|
||||
// side settles against a position, not against another account.
|
||||
return t.Facts.Investment == nil || t.Facts.Investment.CashOnly()
|
||||
}
|
||||
type candidate struct {
|
||||
i, j int
|
||||
gap time.Duration
|
||||
}
|
||||
candidates := []candidate{}
|
||||
for i := range data.Transactions {
|
||||
a := data.Transactions[i]
|
||||
if a.Enrichment.Kind == "transfer" || a.Enrichment.TransferPeerID != "" {
|
||||
if !matchable(a) {
|
||||
continue
|
||||
}
|
||||
ai := byAccount[a.Facts.AccountID]
|
||||
@@ -357,7 +406,7 @@ func MatchTransfers(data *domain.Dataset) {
|
||||
}
|
||||
for j := i + 1; j < len(data.Transactions); j++ {
|
||||
b := data.Transactions[j]
|
||||
if b.Enrichment.Kind == "transfer" || b.Enrichment.TransferPeerID != "" || b.Facts.AccountID != own[target] || normalizeIBAN(b.Facts.CounterpartyIBAN) != ai || a.Facts.Currency != b.Facts.Currency {
|
||||
if !matchable(b) || b.Facts.AccountID != own[target] || normalizeIBAN(b.Facts.CounterpartyIBAN) != ai || a.Facts.Currency != b.Facts.Currency {
|
||||
continue
|
||||
}
|
||||
bm, err := b.Facts.Amount.Minor()
|
||||
@@ -368,29 +417,39 @@ func MatchTransfers(data *domain.Dataset) {
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
delta := ad.Sub(bd)
|
||||
if delta < -72*time.Hour || delta > 72*time.Hour {
|
||||
gap := ad.Sub(bd)
|
||||
if gap < 0 {
|
||||
gap = -gap
|
||||
}
|
||||
if gap > 72*time.Hour {
|
||||
continue
|
||||
}
|
||||
candidates[i] = append(candidates[i], j)
|
||||
candidates[j] = append(candidates[j], i)
|
||||
candidates = append(candidates, candidate{i: i, j: j, gap: gap})
|
||||
}
|
||||
}
|
||||
for i, matches := range candidates {
|
||||
if len(matches) != 1 {
|
||||
sort.Slice(candidates, func(x, y int) bool {
|
||||
if candidates[x].gap != candidates[y].gap {
|
||||
return candidates[x].gap < candidates[y].gap
|
||||
}
|
||||
left, right := data.Transactions[candidates[x].i].Facts.ID, data.Transactions[candidates[y].i].Facts.ID
|
||||
if left != right {
|
||||
return left < right
|
||||
}
|
||||
return data.Transactions[candidates[x].j].Facts.ID < data.Transactions[candidates[y].j].Facts.ID
|
||||
})
|
||||
linked := make([]bool, len(data.Transactions))
|
||||
for _, c := range candidates {
|
||||
if linked[c.i] || linked[c.j] {
|
||||
continue
|
||||
}
|
||||
j := matches[0]
|
||||
if j <= i || len(candidates[j]) != 1 {
|
||||
continue
|
||||
}
|
||||
for _, pair := range [][2]int{{i, j}, {j, i}} {
|
||||
t := &data.Transactions[pair[0]]
|
||||
linked[c.i], linked[c.j] = true, true
|
||||
for _, ends := range [][2]int{{c.i, c.j}, {c.j, c.i}} {
|
||||
t := &data.Transactions[ends[0]]
|
||||
tags := t.Enrichment.TagIDs
|
||||
if tags == nil {
|
||||
tags = []string{}
|
||||
}
|
||||
t.Enrichment = domain.Enrichment{Kind: "transfer", TagIDs: tags, TransferPeerID: data.Transactions[pair[1]].Facts.ID, Classification: domain.Provenance{Source: "transfer_match"}}
|
||||
t.Enrichment = domain.Enrichment{Kind: "transfer", TagIDs: tags, TransferPeerID: data.Transactions[ends[1]].Facts.ID, Classification: domain.Provenance{Source: "transfer_match"}}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -249,11 +249,6 @@ func TestTransfersRequireUniqueReciprocalOwnBankEvidence(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, change := range []func(*domain.Dataset){
|
||||
func(d *domain.Dataset) {
|
||||
copy := d.Transactions[1]
|
||||
copy.Facts.ID = "tx_c"
|
||||
d.Transactions = append(d.Transactions, copy)
|
||||
},
|
||||
func(d *domain.Dataset) { d.Transactions[1].Facts.CounterpartyIBAN = "" },
|
||||
func(d *domain.Dataset) { d.Transactions[1].Facts.Currency = "USD" },
|
||||
func(d *domain.Dataset) { d.Transactions[1].Facts.Amount = "9.99" },
|
||||
@@ -267,11 +262,64 @@ func TestTransfersRequireUniqueReciprocalOwnBankEvidence(t *testing.T) {
|
||||
before := domain.Clone(d)
|
||||
MatchTransfers(&d)
|
||||
if !reflect.DeepEqual(d, before) {
|
||||
t.Fatalf("ambiguous or unsupported transfer evidence linked: %+v", d.Transactions)
|
||||
t.Fatalf("unsupported transfer evidence linked: %+v", d.Transactions)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Two equal top-ups in one week give every leg two candidates. Refusing to
|
||||
// pair them is what turned both legs into spending and income that never
|
||||
// happened, so the pairing must happen, must follow the nearest booking date,
|
||||
// and must not depend on the order the records arrive in.
|
||||
func TestEqualCompetingTransfersPairByNearestDate(t *testing.T) {
|
||||
build := func(reverse bool) domain.Dataset {
|
||||
d := fixtureDataset()
|
||||
leg := func(id, account, amount, date, peerIBAN string) domain.Transaction {
|
||||
f := fixtureFacts()
|
||||
f.ID, f.AccountID, f.Amount, f.BookingDate, f.CounterpartyIBAN = id, account, domain.Money(amount), date, peerIBAN
|
||||
return domain.Transaction{Facts: f, Enrichment: domain.Fallback(f)}
|
||||
}
|
||||
a, b := d.Accounts[0].IBAN, d.Accounts[1].IBAN
|
||||
d.Transactions = []domain.Transaction{
|
||||
leg("tx_out_mon", "account_a", "-800.00", "2026-09-05", b),
|
||||
leg("tx_out_wed", "account_a", "-800.00", "2026-09-07", b),
|
||||
leg("tx_in_tue", "account_b", "800.00", "2026-09-06", a),
|
||||
leg("tx_in_thu", "account_b", "800.00", "2026-09-08", a),
|
||||
}
|
||||
if reverse {
|
||||
for i, j := 0, len(d.Transactions)-1; i < j; i, j = i+1, j-1 {
|
||||
d.Transactions[i], d.Transactions[j] = d.Transactions[j], d.Transactions[i]
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
want := map[string]string{"tx_out_mon": "tx_in_tue", "tx_in_tue": "tx_out_mon", "tx_out_wed": "tx_in_thu", "tx_in_thu": "tx_out_wed"}
|
||||
for _, reverse := range []bool{false, true} {
|
||||
d := build(reverse)
|
||||
MatchTransfers(&d)
|
||||
for _, tx := range d.Transactions {
|
||||
if tx.Enrichment.Kind != "transfer" || tx.Enrichment.TransferPeerID != want[tx.Facts.ID] {
|
||||
t.Fatalf("reverse=%v: %s linked to %q as %q, want %q as transfer", reverse, tx.Facts.ID, tx.Enrichment.TransferPeerID, tx.Enrichment.Kind, want[tx.Facts.ID])
|
||||
}
|
||||
}
|
||||
if err := domain.Validate(d); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A hand-made decision must outlive the next import, or unlinking a pair that
|
||||
// is not really a transfer is undone the moment anything is imported again.
|
||||
func TestManualClassificationSurvivesMatching(t *testing.T) {
|
||||
d := transferDataset()
|
||||
d.Transactions[0].Enrichment.Classification.Source = "manual"
|
||||
before := domain.Clone(d)
|
||||
MatchTransfers(&d)
|
||||
if !reflect.DeepEqual(d, before) {
|
||||
t.Fatalf("matcher overrode a manual decision: %+v", d.Transactions)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMixedReferencedAndAnonymousMultiplicity(t *testing.T) {
|
||||
d := fixtureDataset()
|
||||
anonymous := fixtureFacts()
|
||||
|
||||
@@ -0,0 +1,396 @@
|
||||
package banking
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"finance-duck/internal/domain"
|
||||
)
|
||||
|
||||
// SourceScalable identifies facts imported from a Scalable Capital broker
|
||||
// export.
|
||||
const SourceScalable = "scalable_csv"
|
||||
|
||||
// scalableColumns are the exact normalized headers of a Scalable Capital
|
||||
// transaction export. The layout is matched in full rather than column by
|
||||
// column: a row's meaning depends on the combination of status, assetType and
|
||||
// type, so a partial match would be a different file wearing the same names.
|
||||
var scalableColumns = []string{
|
||||
"date", "time", "status", "reference", "description",
|
||||
"assettype", "type", "isin", "shares", "price", "amount", "fee", "tax", "currency",
|
||||
}
|
||||
|
||||
// scalableEvents maps the export's complete type vocabulary to journal events.
|
||||
// The set is closed on purpose: two of the ten types move a position without
|
||||
// moving money, so an unrecognized type cannot be defaulted either way without
|
||||
// risking a silent balance error. Keys are lowercased with collapsed spaces.
|
||||
var scalableEvents = map[string]string{
|
||||
"deposit": domain.EventDeposit,
|
||||
"withdrawal": domain.EventWithdrawal,
|
||||
"fee": domain.EventFee,
|
||||
"interest": domain.EventInterest,
|
||||
"distribution": domain.EventDistribution,
|
||||
"buy": domain.EventBuy,
|
||||
"sell": domain.EventSell,
|
||||
"reinvestment_distribution": domain.EventReinvest,
|
||||
"corporate action": domain.EventCorporateAction,
|
||||
"security transfer": domain.EventPositionTransfer,
|
||||
}
|
||||
|
||||
// ScalableNote records a figure the export carried that the import deliberately
|
||||
// did not apply, so it can be reviewed before confirming and recognized later
|
||||
// if a balance disagrees.
|
||||
type ScalableNote struct {
|
||||
Record int `json:"record"`
|
||||
Date string `json:"date"`
|
||||
Description string `json:"description"`
|
||||
Fee domain.Money `json:"fee,omitempty"`
|
||||
Tax domain.Money `json:"tax,omitempty"`
|
||||
}
|
||||
|
||||
// ScalableImport is a read broker export awaiting review.
|
||||
type ScalableImport struct {
|
||||
Facts []domain.Facts `json:"-"`
|
||||
// Instruments are securities the export named that the registry does not
|
||||
// hold yet. An import never renames an existing instrument: the name is
|
||||
// editable display text, and the export's own description for one ISIN
|
||||
// changes over time.
|
||||
Instruments []domain.Instrument `json:"instruments"`
|
||||
// Cancelled counts rows the broker did not execute. Their money and share
|
||||
// columns are all zeros, so they satisfy every arithmetic check and would
|
||||
// otherwise import as phantom trades.
|
||||
Cancelled int `json:"cancelled"`
|
||||
// Rounded counts rows whose money carried more than four decimal places,
|
||||
// and Rounding is the exact total adjustment that rounding applied.
|
||||
Rounded int `json:"rounded"`
|
||||
Rounding domain.Quantity `json:"rounding"`
|
||||
// Unapplied lists cash rows carrying a fee or tax. A broker cash amount is
|
||||
// already net of them, so subtracting them again would double-count; they
|
||||
// are recorded on the fact and reported here.
|
||||
Unapplied []ScalableNote `json:"unapplied"`
|
||||
}
|
||||
|
||||
// DetectScalableCSV reports whether a document is a Scalable Capital export and
|
||||
// which 1-based record holds its header.
|
||||
func DetectScalableCSV(f CSVFile) (header int, ok bool) {
|
||||
for i, row := range f.rows {
|
||||
if i >= maxCSVPreambleRows {
|
||||
break
|
||||
}
|
||||
columns, usable := csvColumnIndex(row)
|
||||
if !usable || len(columns) != len(scalableColumns) {
|
||||
continue
|
||||
}
|
||||
matched := true
|
||||
for _, name := range scalableColumns {
|
||||
if _, exists := columns[name]; !exists {
|
||||
matched = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if matched {
|
||||
return i + 1, true
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// ParseScalableCSV converts a broker export into bank facts carrying position
|
||||
// legs.
|
||||
//
|
||||
// The amount column means a different thing per row class, and reading it
|
||||
// wrongly moves money that never moved:
|
||||
//
|
||||
// - a cash row's amount is the money that actually settled, already net of
|
||||
// the tax the broker withheld or refunded, so its tax is recorded and not
|
||||
// applied;
|
||||
// - a buy, sell or reinvestment quotes gross shares times price and settles
|
||||
// gross minus fee minus tax;
|
||||
// - a corporate action or depot transfer quotes a position valuation and
|
||||
// settles no cash at all.
|
||||
//
|
||||
// The share column is signed only for those last two types; buys and sells are
|
||||
// unsigned and take their direction from the type. Both conventions are
|
||||
// resolved here, once.
|
||||
//
|
||||
// The booking date is the date column exactly as printed. Batch rows are
|
||||
// stamped midnight UTC rendered in local time, so the time column crosses
|
||||
// midnight for part of the year and reading date and time together would move
|
||||
// those rows to the previous day.
|
||||
//
|
||||
// A single unrecognized status, type or assetType, or one failed arithmetic
|
||||
// check, rejects the whole file. Every one of those cases can move money, and a
|
||||
// partially imported broker history cannot be told from a truncated export
|
||||
// afterwards.
|
||||
func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instrument) (ScalableImport, error) {
|
||||
// Empty rather than nil: these are arrays in the reviewed JSON, and a null
|
||||
// where a caller expects a list is a bug waiting on a different machine.
|
||||
result := ScalableImport{Instruments: []domain.Instrument{}, Unapplied: []ScalableNote{}}
|
||||
if account.ID == "" {
|
||||
return result, errors.New("broker import requires a selected account")
|
||||
}
|
||||
if !account.Investing() {
|
||||
return result, fmt.Errorf("account %q must be an investment account to hold a broker export", account.DisplayName)
|
||||
}
|
||||
header, ok := DetectScalableCSV(f)
|
||||
if !ok {
|
||||
return result, errors.New("not a Scalable Capital export")
|
||||
}
|
||||
headers := f.rows[header-1]
|
||||
index := map[string]int{}
|
||||
for i, raw := range headers {
|
||||
index[headerName(raw)] = i
|
||||
}
|
||||
cell := func(row []string, name string) string { return strings.TrimSpace(row[index[name]]) }
|
||||
|
||||
instruments := map[string]domain.Instrument{}
|
||||
byISIN := map[string]domain.Instrument{}
|
||||
for _, v := range registry {
|
||||
instruments[v.ID] = v
|
||||
byISIN[v.ISIN] = v
|
||||
}
|
||||
created := map[string]int{}
|
||||
named := map[string]string{}
|
||||
drift := int64(0)
|
||||
for offset, row := range f.rows[header:] {
|
||||
record := header + offset + 1
|
||||
if blankCSVRow(row) {
|
||||
continue
|
||||
}
|
||||
if len(row) != len(headers) {
|
||||
return result, fmt.Errorf("broker record %d has %d columns, expected %d", record, len(row), len(headers))
|
||||
}
|
||||
switch status := cell(row, "status"); {
|
||||
case strings.EqualFold(status, "executed"):
|
||||
case strings.EqualFold(status, "cancelled"), strings.EqualFold(status, "canceled"):
|
||||
result.Cancelled++
|
||||
continue
|
||||
default:
|
||||
return result, fmt.Errorf("broker record %d has unknown status %q: only executed and cancelled rows are understood", record, status)
|
||||
}
|
||||
rawType := cell(row, "type")
|
||||
event, known := scalableEvents[strings.ToLower(strings.Join(strings.Fields(rawType), " "))]
|
||||
if !known {
|
||||
return result, fmt.Errorf("broker record %d has unknown type %q: it may or may not move cash, so nothing was imported", record, rawType)
|
||||
}
|
||||
investment := domain.Investment{Event: event}
|
||||
asset, wanted := cell(row, "assettype"), "Security"
|
||||
if investment.CashOnly() {
|
||||
wanted = "Cash"
|
||||
}
|
||||
if !strings.EqualFold(asset, wanted) {
|
||||
return result, fmt.Errorf("broker record %d pairs type %q with assetType %q, expected %q", record, rawType, asset, wanted)
|
||||
}
|
||||
currency := strings.ToUpper(cell(row, "currency"))
|
||||
if currency != strings.ToUpper(account.Currency) {
|
||||
return result, fmt.Errorf("broker record %d settles in %q but account %q holds %s: currency conversion is not supported", record, currency, account.DisplayName, account.Currency)
|
||||
}
|
||||
booking, err := parseMappedCSVDate(cell(row, "date"), "yyyy-mm-dd")
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid date %q", record, cell(row, "date"))
|
||||
}
|
||||
description := cell(row, "description")
|
||||
isin := strings.ToUpper(strings.Join(strings.Fields(cell(row, "isin")), ""))
|
||||
if isin != "" && !domain.ValidISIN(isin) {
|
||||
return result, fmt.Errorf("broker record %d has an invalid ISIN %q", record, isin)
|
||||
}
|
||||
if isin != "" {
|
||||
held, exists := byISIN[isin]
|
||||
if !exists {
|
||||
held = domain.Instrument{ID: domain.InstrumentID(isin), ISIN: isin, Name: isin, Currency: currency}
|
||||
byISIN[isin] = held
|
||||
instruments[held.ID] = held
|
||||
created[isin] = len(result.Instruments)
|
||||
result.Instruments = append(result.Instruments, held)
|
||||
}
|
||||
investment.InstrumentID = held.ID
|
||||
// One ISIN appears under several descriptions over the years, and
|
||||
// once under the ISIN itself. The most recent real description
|
||||
// names it, and only when this import is the one creating it.
|
||||
slot, mine := created[isin]
|
||||
if mine && description != "" && description != isin && booking >= named[isin] {
|
||||
named[isin] = booking
|
||||
result.Instruments[slot].Name = description
|
||||
}
|
||||
}
|
||||
amount, amountDrift, err := scalableMoney(cell(row, "amount"))
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid amount %q: %w", record, cell(row, "amount"), err)
|
||||
}
|
||||
fee, feeDrift, err := scalableMoney(cell(row, "fee"))
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid fee %q: %w", record, cell(row, "fee"), err)
|
||||
}
|
||||
tax, taxDrift, err := scalableMoney(cell(row, "tax"))
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid tax %q: %w", record, cell(row, "tax"), err)
|
||||
}
|
||||
if amountDrift|feeDrift|taxDrift != 0 {
|
||||
result.Rounded++
|
||||
drift += amountDrift + feeDrift + taxDrift
|
||||
}
|
||||
cash := amount
|
||||
if investment.CashOnly() {
|
||||
if nonzeroMoney(fee) || nonzeroMoney(tax) {
|
||||
result.Unapplied = append(result.Unapplied, ScalableNote{Record: record, Date: booking, Description: description, Fee: fee, Tax: tax})
|
||||
}
|
||||
investment.Fee, investment.Tax = fee, tax
|
||||
} else {
|
||||
if isin == "" {
|
||||
return result, fmt.Errorf("broker record %d moves a position without an ISIN", record)
|
||||
}
|
||||
shares, err := scalableQuantity(cell(row, "shares"))
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid share count %q: %w", record, cell(row, "shares"), err)
|
||||
}
|
||||
price, priceDrift, err := scalableMoney(cell(row, "price"))
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid price %q: %w", record, cell(row, "price"), err)
|
||||
}
|
||||
if priceDrift != 0 {
|
||||
return result, fmt.Errorf("broker record %d has a price %q beyond four decimal places", record, cell(row, "price"))
|
||||
}
|
||||
signed, err := scalableSignedShares(event, shares)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
investment.Quantity, investment.Price, investment.Gross = signed, price, amount
|
||||
if investment.PositionOnly() {
|
||||
if fee != "" || tax != "" {
|
||||
return result, fmt.Errorf("broker record %d is a %s carrying fee %q and tax %q, which have no settled cash to apply to", record, rawType, fee, tax)
|
||||
}
|
||||
cash = "0.00"
|
||||
} else {
|
||||
investment.Fee, investment.Tax = fee, tax
|
||||
if cash, err = scalableSettlement(amount, fee, tax); err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
facts := domain.Facts{
|
||||
Source: SourceScalable, AccountID: account.ID, BookingDate: booking,
|
||||
Amount: cash, Currency: currency, RawDescription: description,
|
||||
ExternalID: cell(row, "reference"), Investment: &investment,
|
||||
}
|
||||
// A broker export has no counterparty column, so a deposit or
|
||||
// withdrawal takes the account's configured settlement IBAN. That is
|
||||
// what lets the ordinary transfer matcher pair it with the funding
|
||||
// account instead of leaving it to look like income.
|
||||
if investment.Event == domain.EventDeposit || investment.Event == domain.EventWithdrawal {
|
||||
facts.CounterpartyIBAN = normalizeIBAN(account.ReferenceIBAN)
|
||||
}
|
||||
if err := domain.ValidateInvestment(facts, account, instruments); err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
result.Facts = append(result.Facts, facts)
|
||||
}
|
||||
if len(result.Facts) == 0 {
|
||||
return result, errors.New("broker export contains no executed records")
|
||||
}
|
||||
result.Rounding = domain.FormatQuantity(drift)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// nonzeroMoney reports a figure that could change a balance. The export leaves
|
||||
// a column blank where it does not apply and writes an explicit zero where it
|
||||
// applies but is nil; only the second kind is worth putting in front of
|
||||
// someone before they confirm an import.
|
||||
func nonzeroMoney(m domain.Money) bool {
|
||||
minor, err := m.Minor()
|
||||
return err == nil && minor != 0
|
||||
}
|
||||
|
||||
// scalableMoney reads one German-formatted money cell and returns the exact
|
||||
// remainder that rounding discarded, in hundred-millionths. The export quotes a
|
||||
// reinvested distribution to six decimal places, which money's four cannot
|
||||
// hold; the residue is reported rather than hidden. An empty cell is empty
|
||||
// money, not zero: blank marks a column that does not apply to the row.
|
||||
func scalableMoney(value string) (domain.Money, int64, error) {
|
||||
plain, ok, err := scalablePlain(value)
|
||||
if !ok || err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
exact, err := domain.ParseQuantity(plain)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
units, err := exact.Units()
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
rounded := units / 10000
|
||||
switch remainder := units % 10000; {
|
||||
case remainder >= 5000:
|
||||
rounded++
|
||||
case remainder <= -5000:
|
||||
rounded--
|
||||
}
|
||||
return domain.FormatMoney(rounded), units - rounded*10000, nil
|
||||
}
|
||||
|
||||
// scalableQuantity reads one German-formatted share count. Nothing is rounded:
|
||||
// a holding is verified against the broker's own figure, so a count beyond
|
||||
// eight decimal places is refused instead of silently truncated.
|
||||
func scalableQuantity(value string) (domain.Quantity, error) {
|
||||
plain, ok, err := scalablePlain(value)
|
||||
if !ok || err != nil {
|
||||
return "", err
|
||||
}
|
||||
return domain.ParseQuantity(plain)
|
||||
}
|
||||
|
||||
// scalablePlain normalizes one numeric cell to a plain decimal string, or
|
||||
// reports that the cell was blank.
|
||||
func scalablePlain(value string) (string, bool, error) {
|
||||
value = strings.NewReplacer("\u00a0", "", "\u202f", "", "'", "").Replace(strings.TrimSpace(value))
|
||||
if value == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
plain, err := germanDecimal(value)
|
||||
return plain, err == nil, err
|
||||
}
|
||||
|
||||
// scalableSignedShares resolves the export's two sign conventions. A buy, sell
|
||||
// or reinvestment carries an unsigned count and takes its direction from the
|
||||
// type; a corporate action or depot transfer is already signed.
|
||||
func scalableSignedShares(event string, shares domain.Quantity) (domain.Quantity, error) {
|
||||
units, err := shares.Units()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if units == 0 {
|
||||
return "", fmt.Errorf("%s requires a nonzero share count", event)
|
||||
}
|
||||
switch event {
|
||||
case domain.EventBuy, domain.EventReinvest, domain.EventSell:
|
||||
if units < 0 {
|
||||
return "", fmt.Errorf("%s carries a signed share count %s; only corporate actions and depot transfers are signed", event, shares)
|
||||
}
|
||||
if event == domain.EventSell {
|
||||
units = -units
|
||||
}
|
||||
}
|
||||
return domain.FormatQuantity(units), nil
|
||||
}
|
||||
|
||||
// scalableSettlement is gross minus fee minus tax: the cash a trade moved. The
|
||||
// broker states fee and tax as positive deductions whichever way the trade
|
||||
// went, so both are subtracted from a signed gross.
|
||||
func scalableSettlement(gross, fee, tax domain.Money) (domain.Money, error) {
|
||||
total := int64(0)
|
||||
for _, deduction := range []struct {
|
||||
sign int64
|
||||
money domain.Money
|
||||
}{{1, gross}, {-1, fee}, {-1, tax}} {
|
||||
if deduction.money == "" {
|
||||
continue
|
||||
}
|
||||
minor, err := deduction.money.Minor()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
total += deduction.sign * minor
|
||||
}
|
||||
return domain.FormatMoney(total), nil
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package banking
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"finance-duck/internal/domain"
|
||||
)
|
||||
|
||||
const scalableHeader = "date;time;status;reference;description;assetType;type;isin;shares;price;amount;fee;tax;currency\n"
|
||||
|
||||
// Every row below is a real Scalable Capital export line. Together they cover
|
||||
// all ten row types, both sign conventions, a reference shared by two legs of
|
||||
// one event, a six-decimal reinvestment, a zero-price corporate action, a
|
||||
// depot switch, a cancelled retry, and one ISIN whose description changes over
|
||||
// time and whose latest description is the one that names it.
|
||||
var scalableRows = []string{
|
||||
`2024-11-10;02:00:00;Executed;ABCDEF012345;Scalable Instant Cash Deposit;Cash;Deposit;;;;800,00;;;EUR`,
|
||||
`2026-08-18;02:00:00;Executed;ITLLRRVPRK11ZGNPAD2VNC;Scalable Broker PRIME bis 16.09.2026;Cash;Deposit;;;;4,99;0,00;;EUR`,
|
||||
`2026-08-18;02:00:00;Executed;LZLVVJYLNJY9ARAK;Entgelt PRIME+ Broker;Cash;Fee;;;;-4,99;0,00;;EUR`,
|
||||
`2026-07-16;14:19:06;Executed;O9HNT63GYQVUNPXEXQMSCJ;Scalable Capital Broker Auszahlung;Cash;Withdrawal;;;;-4.458,19;0,00;0,00;EUR`,
|
||||
`2026-01-02;01:00:00;Executed;INTEREST0001;Zinsen;Cash;Interest;;;;12,34;;1,23;EUR`,
|
||||
`2026-01-20;01:00:00;Executed;429776_rrCjP4EcbpefpNiVQeD495;Taiwan Semiconductor Manufact. ADR;Cash;Distribution;US8740391003;;;29,68;0,00;7,43;EUR`,
|
||||
`2026-01-20;01:00:00;Executed;429776_rrCjP4EcbpefpNiVQeD495;Taiwan Semiconductor Manufact. ADR;Security;Reinvestment_Distribution;US8740391003;0,076494;388,00;-29,679672;0,00;0,00;EUR`,
|
||||
`2025-05-07;09:02:29;Executed;SCALTThBbxx6z5Z;Rheinmetall Long 10x Faktor-Zertifikat HVB;Security;Buy;DE000UG4V0Z7;14;26,45;-370,30;0,00;0,00;EUR`,
|
||||
`2025-09-17;15:14:49;Executed;SCALwBaNVPpjf8p;Rheinmetall Long 10x Factor HVB;Security;Buy;DE000UG4V0Z7;203;1,23;-249,69;0,99;0,00;EUR`,
|
||||
`2025-09-18;13:38:08;Executed;SCALSVuyHibZT4w;Rheinmetall Long 10x Factor HVB;Security;Buy;DE000UG4V0Z7;6;1,10;-6,60;0,99;0,00;EUR`,
|
||||
`2025-10-28;01:00:00;Executed;48231_rrCjP4EcbpefpNiVQeD495;Rheinmetall Long 10x Factor HVB;Cash;Distribution;DE000UG4V0Z7;;;32,64;;-1,42;EUR`,
|
||||
`2025-10-28;01:00:00;Executed;48231_rrCjP4EcbpefpNiVQeD495;Rheinmetall Long 10x Factor HVB;Security;Corporate action;DE000UG4V0Z7;-223;0,14;-31,22;;;EUR`,
|
||||
`2025-10-21;02:00:00;Executed;WWUM 00566579567;FR0014012ZX8;Security;Corporate action;FR0014012ZX8;1,14;0,00;0,00;;;EUR`,
|
||||
`2025-12-05;01:00:00;Executed;WWUM 00590038089;Amundi MSCI USA Daily (2x) Leveraged (Acc);Security;Security transfer;FR0010755611;-65;25,235;-1.640,275;;;EUR`,
|
||||
`2025-12-06;01:00:00;Executed;SWITCH-101-rrCjP4EcbpefpNiVQeD495-FR0010755611-WDP;Amundi MSCI USA Daily (2x) Leveraged (Acc);Security;Security transfer;FR0010755611;65;25,59;1.663,35;;;EUR`,
|
||||
`2026-03-17;01:00:00;Executed;SCALmNYgdoA58V;Amundi Core MSCI World (Acc);Security;Sell;IE000BI8OT95;61;158,385;9.661,485;0,00;220,47;EUR`,
|
||||
`2025-01-27;16:26:31;Cancelled;SCALCRFHbTWXN9h;Amundi Leveraged MSCI USA Daily (Acc);Security;Buy;FR0010755611;0;0,00;0,00;0,00;0,00;EUR`,
|
||||
}
|
||||
|
||||
func brokerAccount() domain.Account {
|
||||
return domain.Account{
|
||||
ID: "acct_broker", DisplayName: "Scalable", Institution: "Scalable Capital",
|
||||
Currency: "EUR", Kind: domain.AccountInvestment,
|
||||
IBAN: "DE02120300000000202051", ReferenceIBAN: "DE89370400440532013000", Active: true,
|
||||
}
|
||||
}
|
||||
|
||||
func readBroker(t *testing.T, rows ...string) ScalableImport {
|
||||
t.Helper()
|
||||
file, err := ReadCSV(strings.NewReader(scalableHeader + strings.Join(rows, "\n") + "\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
result, err := ParseScalableCSV(file, brokerAccount(), nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func TestScalableExportSettlesCashAndPositionsSeparately(t *testing.T) {
|
||||
result := readBroker(t, scalableRows...)
|
||||
if result.Cancelled != 1 {
|
||||
t.Fatalf("cancelled rows imported: %d skipped", result.Cancelled)
|
||||
}
|
||||
if len(result.Facts) != len(scalableRows)-1 {
|
||||
t.Fatalf("imported %d of %d executed rows", len(result.Facts), len(scalableRows)-1)
|
||||
}
|
||||
|
||||
// The cash a row settles, per row class. A cash row's amount is already
|
||||
// net; a trade settles gross minus fee minus tax; a corporate action or
|
||||
// depot transfer settles nothing at all.
|
||||
wantCash := map[string]string{
|
||||
"ABCDEF012345": "800.00",
|
||||
"ITLLRRVPRK11ZGNPAD2VNC": "4.99",
|
||||
"LZLVVJYLNJY9ARAK": "-4.99",
|
||||
"O9HNT63GYQVUNPXEXQMSCJ": "-4458.19",
|
||||
"INTEREST0001": "12.34",
|
||||
"SCALTThBbxx6z5Z": "-370.30",
|
||||
"SCALwBaNVPpjf8p": "-250.68",
|
||||
"SCALSVuyHibZT4w": "-7.59",
|
||||
"WWUM 00566579567": "0.00",
|
||||
"WWUM 00590038089": "0.00",
|
||||
"SWITCH-101-rrCjP4EcbpefpNiVQeD495-FR0010755611-WDP": "0.00",
|
||||
"SCALmNYgdoA58V": "9441.015",
|
||||
}
|
||||
total := int64(0)
|
||||
for _, f := range result.Facts {
|
||||
minor, err := f.Amount.Minor()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
total += minor
|
||||
if want, ok := wantCash[f.ExternalID]; ok && string(f.Amount) != want {
|
||||
t.Errorf("%s settled %s, want %s", f.ExternalID, f.Amount, want)
|
||||
}
|
||||
}
|
||||
if got := string(domain.FormatMoney(total)); got != "5199.2353" {
|
||||
t.Errorf("cash balance %s, want 5199.2353", got)
|
||||
}
|
||||
|
||||
// Signs: a buy and a reinvestment add, a sell removes, and a corporate
|
||||
// action or depot transfer keeps the sign the export printed.
|
||||
holdings := map[string]int64{}
|
||||
for _, f := range result.Facts {
|
||||
if f.Investment.InstrumentID == "" || f.Investment.Quantity == "" {
|
||||
continue
|
||||
}
|
||||
units, err := f.Investment.Quantity.Units()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
holdings[f.Investment.InstrumentID] += units
|
||||
}
|
||||
for isin, want := range map[string]int64{
|
||||
"DE000UG4V0Z7": 0, // 14 + 203 + 6 - 223, the knock-out closing the position
|
||||
"FR0010755611": 0, // a depot switch out and back
|
||||
"FR0014012ZX8": 114000000, // 1.14 free units at no price
|
||||
"US8740391003": 7649400, // 0.076494 reinvested
|
||||
"IE000BI8OT95": -6100000000,
|
||||
} {
|
||||
if got := holdings[domain.InstrumentID(isin)]; got != want {
|
||||
t.Errorf("%s holds %d hundred-millionths, want %d", isin, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// One ISIN, several descriptions over the years, and one row that carries
|
||||
// the ISIN as its own description.
|
||||
names := map[string]string{}
|
||||
for _, v := range result.Instruments {
|
||||
names[v.ISIN] = v.Name
|
||||
}
|
||||
for isin, want := range map[string]string{
|
||||
"DE000UG4V0Z7": "Rheinmetall Long 10x Factor HVB",
|
||||
"FR0014012ZX8": "FR0014012ZX8",
|
||||
"US8740391003": "Taiwan Semiconductor Manufact. ADR",
|
||||
} {
|
||||
if names[isin] != want {
|
||||
t.Errorf("%s named %q, want %q", isin, names[isin], want)
|
||||
}
|
||||
}
|
||||
|
||||
// Six decimal places do not fit in money. The residue is reported, not hidden.
|
||||
if result.Rounded != 1 || result.Rounding != "0.000028" {
|
||||
t.Errorf("rounding reported as %d rows and %s, want 1 row and 0.000028", result.Rounded, result.Rounding)
|
||||
}
|
||||
|
||||
// A broker cash amount is already net of tax, so the tax column is
|
||||
// recorded and never subtracted again.
|
||||
if len(result.Unapplied) != 3 {
|
||||
t.Fatalf("unapplied fee/tax notes: %+v", result.Unapplied)
|
||||
}
|
||||
for _, note := range result.Unapplied {
|
||||
if note.Tax == "" {
|
||||
t.Errorf("note without the figure that was not applied: %+v", note)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The broker reuses one reference for every leg of an economic event, so
|
||||
// dedupe on the reference alone silently drops half of each corporate action.
|
||||
func TestSharedBrokerReferenceKeepsEveryLeg(t *testing.T) {
|
||||
result := readBroker(t, scalableRows...)
|
||||
data := domain.NewDataset()
|
||||
data.Accounts = []domain.Account{brokerAccount()}
|
||||
data.Instruments = result.Instruments
|
||||
added, err := NormalizeAndDedupe(data, result.Facts)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(added) != len(result.Facts) {
|
||||
t.Fatalf("dedupe kept %d of %d legs", len(added), len(result.Facts))
|
||||
}
|
||||
data.Transactions = added
|
||||
if err := domain.Validate(data); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again, err := NormalizeAndDedupe(data, result.Facts)
|
||||
if err != nil || len(again) != 0 {
|
||||
t.Fatalf("re-import was not idempotent: %v %+v", err, again)
|
||||
}
|
||||
}
|
||||
|
||||
// Every one of these can move money that never moved, so each rejects the
|
||||
// whole file rather than importing the rest.
|
||||
func TestScalableRejectsRowsItCannotAccountFor(t *testing.T) {
|
||||
for name, row := range map[string]string{
|
||||
"unknown type": `2026-01-05;01:00:00;Executed;R1;Something;Cash;Vorabpauschale;;;;-12,00;;;EUR`,
|
||||
"unknown status": `2026-01-05;01:00:00;Pending;R1;Something;Cash;Deposit;;;;12,00;;;EUR`,
|
||||
"asset type mismatch": `2026-01-05;01:00:00;Executed;R1;Something;Security;Deposit;;;;12,00;;;EUR`,
|
||||
"foreign currency": `2026-01-05;01:00:00;Executed;R1;Something;Cash;Deposit;;;;12,00;;;USD`,
|
||||
"mismatched gross": `2026-01-05;01:00:00;Executed;R1;Something;Security;Buy;DE000UG4V0Z7;10;2,00;-25,00;0,00;0,00;EUR`,
|
||||
"signed buy": `2026-01-05;01:00:00;Executed;R1;Something;Security;Buy;DE000UG4V0Z7;-10;2,00;20,00;0,00;0,00;EUR`,
|
||||
"paid corporate": `2026-01-05;01:00:00;Executed;R1;Something;Security;Corporate action;DE000UG4V0Z7;-5;2,00;-10,00;1,00;0,00;EUR`,
|
||||
"security without ISIN": `2026-01-05;01:00:00;Executed;R1;Something;Security;Buy;;10;2,00;-20,00;0,00;0,00;EUR`,
|
||||
"invalid ISIN": `2026-01-05;01:00:00;Executed;R1;Something;Security;Buy;NOTANISIN;10;2,00;-20,00;0,00;0,00;EUR`,
|
||||
} {
|
||||
file, err := ReadCSV(strings.NewReader(scalableHeader + row + "\n"))
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
if _, err := ParseScalableCSV(file, brokerAccount(), nil); err == nil {
|
||||
t.Errorf("%s: accepted a row that can move money it should not", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An inconsistent row is caught, and a uniformly mangled one is not. Where the
|
||||
// whole row lost its separator together, shares times price still equals the
|
||||
// amount at every scale, so no check inside the file can see it. This is a
|
||||
// known limit, not an oversight: only a price cross-check against an outside
|
||||
// provider distinguishes 1 x 25,795 from 1 x 25795, and that is deliberately
|
||||
// out of scope. The test exists so nobody claims coverage that is not here.
|
||||
func TestSingleShareRowCatchesOnlyInconsistentArithmetic(t *testing.T) {
|
||||
valid := `2024-12-09;10:48:44;Executed;SCALfhSXRbGWKno;Amundi Leveraged MSCI USA Daily (Acc);Security;Buy;FR0010755611;1;25,795;-25,795;0,99;0,00;EUR`
|
||||
result := readBroker(t, valid)
|
||||
if got := result.Facts[0].Amount; got != "-26.785" {
|
||||
t.Fatalf("one-share buy settled %s, want -26.785", got)
|
||||
}
|
||||
inconsistent := strings.Replace(valid, "25,795;-25,795", "25,795;-257,95", 1)
|
||||
file, err := ReadCSV(strings.NewReader(scalableHeader + inconsistent + "\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := ParseScalableCSV(file, brokerAccount(), nil); err == nil {
|
||||
t.Fatal("accepted a one-share row whose amount is off by a factor of ten")
|
||||
}
|
||||
uniform := readBroker(t, strings.Replace(valid, "1;25,795;-25,795", "1;25795;-25795", 1))
|
||||
if got := uniform.Facts[0].Investment.Gross; got != "-25795.00" {
|
||||
t.Fatalf("uniformly mangled row read as %s: the file-internal identity cannot see it, and that must stay visible here", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A thousands dot and a decimal dot are both present in one share column.
|
||||
func TestBrokerShareColumnDistinguishesGroupingFromDecimals(t *testing.T) {
|
||||
result := readBroker(t,
|
||||
`2026-02-24;17:11:29;Executed;G1;iShares Global Clean Energy Transition (Dist);Security;Buy;IE00B1XNHC34;1.014;9,408;-9.539,712;0,00;0,00;EUR`,
|
||||
`2026-02-25;17:11:29;Executed;G2;iShares Global Clean Energy Transition (Dist);Security;Buy;IE00B1XNHC34;1.14;9,408;-10,7251;0,00;0,00;EUR`,
|
||||
)
|
||||
if got := result.Facts[0].Investment.Quantity; got != "1014" {
|
||||
t.Errorf("grouped share count read as %s, want 1014", got)
|
||||
}
|
||||
if got := result.Facts[1].Investment.Quantity; got != "1.14" {
|
||||
t.Errorf("fractional share count read as %s, want 1.14", got)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user