Import Trade Republic exports, whose conventions invert Scalable's
A second broker export is recognized locally, by its full column set, and read through the same pipeline: detection and parsing now dispatch on the format, so the upload path, the review dialog, deduplication, the journal and the Wealth report are unchanged. Its nine row types cover cash transfers, interest, dividends, tax settlements and trades in funds, shares and crypto; none of them moves a position without moving cash, so the cash-neutral class that Scalable's corporate actions belong to does not arise here. Three of its conventions are the opposite of the export already supported, and reading any of them the other way round moves money. Fee and tax are the signed adjustments it made to the cash rather than deductions from a gross, so a one euro order fee arrives as -1.00 and is negated at import; the journal keeps one convention and the domain never learns that two exist. A cash row's amount is the gross, not the net, so interest of 16.46 with -4.33 of tax credits 12.13 - where the other export states its cash already net and its tax is recorded and never applied. Whether a cash row carries a gross now decides which of those it was, which also makes the first kind's settlement checkable and stops the Wealth report from claiming a figure was left unapplied when it was not. And a TAX_OPTIMIZATION row puts zero in the amount column and its money in the tax column, signed both ways: read as cash, all six in a real export move nothing. Two more rows lie about their own columns. A dividend fills the share column with the holding the dividend was paid on, not with a position change, so adding it would double the holding. Crypto carries a bare ticker in the symbol column and its ISIN-shaped identifier only in the description, so the identifier is taken from the symbol when that is an ISIN and otherwise from the one the description names; a position row resolving to neither is refused rather than attached to a guess. The shares-times-price check now holds a gross to the precision the export stated it at rather than to four places. This export prints the notional rounded to cents, and 29 of 59 real trades do not land on a whole cent: demanding exactness rejected half a portfolio. One unit of the stated precision is still four orders of magnitude tighter than the misplaced separator the check exists to catch, and where an export prints the full product the check stays exact. A unit price moves from money to the eight-place quantity type, because a crypto price is quoted to six and rounding it would break the check the amount is verified against. Trailing zeros are dropped before any precision test: this export pads a six-place price to ten, and the padding would otherwise exhaust the precision the value needs. A transfer's counterparty comes from the export's own IBAN column when it has one, from the IBAN the description names in parentheses when it does not, and from the account's configured settlement IBAN when neither names anything. Free text contributes only a value shaped like an IBAN. Without this, 108 transfers stay unpaired and their bank-side counterparts read as spending and income. Verified end to end against a real export: 26 rows import to a cash balance of 32187.02 matching the figure computed by hand from the source rows, all four positions close at exactly zero, and every trade satisfies its own arithmetic.
This commit is contained in:
+18
-7
@@ -132,7 +132,7 @@ type CSVImport struct {
|
||||
// than by a column mapping, and the review needs to show what that parser
|
||||
// decided: which securities it would register, which rows it skipped, and
|
||||
// which figures it deliberately did not apply.
|
||||
Broker *banking.ScalableImport `json:"broker,omitempty"`
|
||||
Broker *banking.BrokerImport `json:"broker,omitempty"`
|
||||
|
||||
facts []domain.Facts
|
||||
instruments []domain.Instrument
|
||||
@@ -175,8 +175,8 @@ func (a *App) PrepareCSVImport(ctx context.Context, rev, accountID string, r io.
|
||||
// cash on one row, a gross to be netted on another, and a position
|
||||
// valuation that must not touch cash on a third, so a column mapping cannot
|
||||
// describe it.
|
||||
if header, broker := banking.DetectScalableCSV(file); broker {
|
||||
read, e := banking.ParseScalableCSV(file, account, s.Data.Instruments)
|
||||
if source, label, header, broker := banking.DetectBrokerCSV(file); broker {
|
||||
read, e := banking.ParseBrokerCSV(file, account, s.Data.Instruments)
|
||||
if e != nil {
|
||||
return CSVImport{}, e
|
||||
}
|
||||
@@ -184,9 +184,9 @@ func (a *App) PrepareCSVImport(ctx context.Context, rev, accountID string, r io.
|
||||
if e != nil {
|
||||
return CSVImport{}, e
|
||||
}
|
||||
prepared.Source, prepared.SourceLabel = banking.SourceScalable, "Scalable Capital"
|
||||
prepared.Mapping = banking.CSVMapping{HeaderRow: header, DateFormat: "yyyy-mm-dd", DecimalFormat: "comma"}
|
||||
prepared.Columns = brokerColumns()
|
||||
prepared.Source, prepared.SourceLabel = source, label
|
||||
prepared.Mapping = banking.CSVMapping{HeaderRow: header}
|
||||
prepared.Columns = brokerColumns(source)
|
||||
prepared.Records, prepared.New, prepared.Duplicates = len(read.Facts), len(added), len(read.Facts)-len(added)
|
||||
prepared.Samples, prepared.facts, prepared.instruments = csvSamples(read.Facts), read.Facts, read.Instruments
|
||||
prepared.Broker = &read
|
||||
@@ -262,7 +262,18 @@ func (a *App) retain(prepared CSVImport) (CSVImport, error) {
|
||||
// brokerColumns describes what the broker parser decided, in the same
|
||||
// reviewable shape as a column mapping. The dispatch is the part that can be
|
||||
// wrong in a way that moves money, so it is the part shown.
|
||||
func brokerColumns() []CSVColumn {
|
||||
func brokerColumns(source string) []CSVColumn {
|
||||
if source == banking.SourceTradeRepublic {
|
||||
return []CSVColumn{
|
||||
{Field: "Booking date", Column: "date, exactly as printed; the datetime column is UTC and disagrees with it late in the evening"},
|
||||
{Field: "Cash movement", Column: "amount − fee − tax, where the export writes fee and tax as the signed adjustments it made and the amount is the gross"},
|
||||
{Field: "Position change", Column: "shares, already signed; a dividend's shares are the holding it was paid on and move nothing"},
|
||||
{Field: "Instrument", Column: "symbol when it is an ISIN, else the one ISIN the description names; crypto carries a ticker in the column"},
|
||||
{Field: "Counterparty", Column: "counterparty_iban, else the IBAN the description names in parentheses, else this account's settlement IBAN"},
|
||||
{Field: "Reference", Column: "transaction_id"},
|
||||
{Field: "Decimals", Column: "plain decimal point; trailing zeros are padding, not precision"},
|
||||
}
|
||||
}
|
||||
return []CSVColumn{
|
||||
{Field: "Booking date", Column: "date, exactly as printed; the time column is local and crosses midnight, so it is ignored"},
|
||||
{Field: "Imported rows", Column: `status "Executed" only; cancelled retries are all zeros and would import as phantom trades`},
|
||||
|
||||
@@ -149,9 +149,13 @@ func WealthOf(data domain.Dataset) Wealth {
|
||||
st.broken = append(st.broken, fmt.Sprintf("%s %s: %v", f.BookingDate, f.ID, err))
|
||||
}
|
||||
if inv.CashOnly() {
|
||||
// A cash row carrying a gross had its fee and tax applied to reach
|
||||
// that amount, and its settlement is already verified above. Only a
|
||||
// row whose amount arrived net has figures that were recorded and
|
||||
// deliberately never subtracted.
|
||||
fee, _ := inv.Fee.Minor()
|
||||
tax, _ := inv.Tax.Minor()
|
||||
if fee != 0 || tax != 0 {
|
||||
if inv.Gross == "" && (fee != 0 || tax != 0) {
|
||||
st.unappliedRows++
|
||||
st.unappliedFee += fee
|
||||
st.unappliedTax += tax
|
||||
|
||||
@@ -0,0 +1,298 @@
|
||||
package banking
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/big"
|
||||
"strings"
|
||||
|
||||
"finance-duck/internal/domain"
|
||||
)
|
||||
|
||||
// BrokerNote records a figure an export carried that the import deliberately
|
||||
// did not apply, so it can be reviewed before confirming and recognized later
|
||||
// if a balance disagrees.
|
||||
type BrokerNote struct {
|
||||
Record int `json:"record"`
|
||||
Date string `json:"date"`
|
||||
Description string `json:"description"`
|
||||
Fee domain.Money `json:"fee,omitempty"`
|
||||
Tax domain.Money `json:"tax,omitempty"`
|
||||
}
|
||||
|
||||
// BrokerImport is a read broker export awaiting review.
|
||||
type BrokerImport struct {
|
||||
Facts []domain.Facts `json:"-"`
|
||||
// Instruments are securities the export named that the registry does not
|
||||
// hold yet. An import never renames an existing instrument: the name is
|
||||
// editable display text, and an export's own description for one ISIN
|
||||
// changes over time.
|
||||
Instruments []domain.Instrument `json:"instruments"`
|
||||
// Cancelled counts rows the broker did not execute. Their money and share
|
||||
// columns are all zeros, so they satisfy every arithmetic check and would
|
||||
// otherwise import as phantom trades.
|
||||
Cancelled int `json:"cancelled"`
|
||||
// Rounded counts rows whose money carried more than four decimal places,
|
||||
// and Rounding is the exact total adjustment that rounding applied, at
|
||||
// whatever precision the export used.
|
||||
Rounded int `json:"rounded"`
|
||||
Rounding string `json:"rounding"`
|
||||
// Unapplied lists cash rows carrying a fee or tax that was recorded but
|
||||
// not subtracted, because the export had already applied it to the amount.
|
||||
Unapplied []BrokerNote `json:"unapplied"`
|
||||
}
|
||||
|
||||
func newBrokerImport() BrokerImport {
|
||||
// Empty rather than nil: these are arrays in the reviewed JSON, and a null
|
||||
// where a caller expects a list is a bug waiting on a different machine.
|
||||
return BrokerImport{Instruments: []domain.Instrument{}, Unapplied: []BrokerNote{}}
|
||||
}
|
||||
|
||||
// DetectBrokerCSV recognizes a broker export by its complete column set and
|
||||
// reports the 1-based record holding its header. A layout is matched in full
|
||||
// rather than column by column: a row's meaning depends on a combination of its
|
||||
// classifying columns, so a partial match is a different file wearing the same
|
||||
// names.
|
||||
func DetectBrokerCSV(f CSVFile) (source, label string, header int, ok bool) {
|
||||
for _, format := range []struct {
|
||||
source, label string
|
||||
columns []string
|
||||
}{
|
||||
{SourceScalable, "Scalable Capital", scalableColumns},
|
||||
{SourceTradeRepublic, "Trade Republic", tradeRepublicColumns},
|
||||
} {
|
||||
if header, found := matchColumns(f, format.columns); found {
|
||||
return format.source, format.label, header, true
|
||||
}
|
||||
}
|
||||
return "", "", 0, false
|
||||
}
|
||||
|
||||
// ParseBrokerCSV reads whichever recognized broker export the document is.
|
||||
func ParseBrokerCSV(f CSVFile, account domain.Account, registry []domain.Instrument) (BrokerImport, error) {
|
||||
source, _, _, ok := DetectBrokerCSV(f)
|
||||
switch {
|
||||
case !ok:
|
||||
return newBrokerImport(), errors.New("not a recognized broker export")
|
||||
case source == SourceScalable:
|
||||
return ParseScalableCSV(f, account, registry)
|
||||
default:
|
||||
return ParseTradeRepublicCSV(f, account, registry)
|
||||
}
|
||||
}
|
||||
|
||||
func matchColumns(f CSVFile, want []string) (header int, ok bool) {
|
||||
for i, row := range f.rows {
|
||||
if i >= maxCSVPreambleRows {
|
||||
break
|
||||
}
|
||||
columns, usable := csvColumnIndex(row)
|
||||
if !usable || len(columns) != len(want) {
|
||||
continue
|
||||
}
|
||||
matched := true
|
||||
for _, name := range want {
|
||||
if _, exists := columns[name]; !exists {
|
||||
matched = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if matched {
|
||||
return i + 1, true
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// investmentTarget checks that an export can be imported into this account at all.
|
||||
func investmentTarget(account domain.Account) error {
|
||||
if account.ID == "" {
|
||||
return errors.New("broker import requires a selected account")
|
||||
}
|
||||
if !account.Investing() {
|
||||
return fmt.Errorf("account %q must be an investment account to hold a broker export", account.DisplayName)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// brokerColumnIndex resolves a matched header row to column positions.
|
||||
func brokerColumnIndex(f CSVFile, header int) (headers []string, at func([]string, string) string) {
|
||||
headers = f.rows[header-1]
|
||||
index := make(map[string]int, len(headers))
|
||||
for i, raw := range headers {
|
||||
index[headerName(raw)] = i
|
||||
}
|
||||
return headers, func(row []string, name string) string { return strings.TrimSpace(row[index[name]]) }
|
||||
}
|
||||
|
||||
// nonzeroMoney reports a figure that could change a balance. An export leaves a
|
||||
// column blank where it does not apply and writes an explicit zero where it
|
||||
// applies but is nil; only the second kind is worth putting in front of someone
|
||||
// before they confirm an import.
|
||||
func nonzeroMoney(m domain.Money) bool {
|
||||
minor, err := m.Minor()
|
||||
return err == nil && minor != 0
|
||||
}
|
||||
|
||||
// negated flips a signed adjustment into a deduction. One broker states a fee
|
||||
// as the negative amount it took off the cash; the journal stores fees and
|
||||
// taxes as deductions from a gross, so that convention is normalized once, at
|
||||
// import, rather than being carried into the domain.
|
||||
func negated(m domain.Money) (domain.Money, error) {
|
||||
if m == "" {
|
||||
return "", nil
|
||||
}
|
||||
minor, err := m.Minor()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return domain.FormatMoney(-minor), nil
|
||||
}
|
||||
|
||||
// residueScale is the precision a discarded remainder is accumulated at. A
|
||||
// broker amount is its share count times its unit price, so it carries as many
|
||||
// decimal places as the two together need: a real export reinvests to nine.
|
||||
// Eighteen is far past anything a settlement can produce and still exact.
|
||||
const residueScale = 18
|
||||
|
||||
// Decimal conventions a broker export can use. German exports write a comma
|
||||
// decimal and group thousands with a dot; the rest write a plain decimal point.
|
||||
const (
|
||||
decimalGerman = true
|
||||
decimalPlain = false
|
||||
)
|
||||
|
||||
// brokerMoney reads one money cell, rounds it to money's four decimal places
|
||||
// half away from zero, and returns the exact remainder that rounding discarded,
|
||||
// in units of 1e-18. The remainder is reported rather than hidden, and never
|
||||
// guessed at: it is the only honest account of why a computed balance can
|
||||
// differ from the broker's by a fraction of a cent.
|
||||
//
|
||||
// An empty cell is empty money, not zero: blank marks a column that does not
|
||||
// apply to the row.
|
||||
func brokerMoney(value string, german bool) (domain.Money, *big.Int, error) {
|
||||
plain, ok, err := brokerPlain(value, german)
|
||||
if !ok || err != nil {
|
||||
return "", new(big.Int), err
|
||||
}
|
||||
magnitude, negative, err := brokerDigits(plain)
|
||||
if err != nil {
|
||||
return "", new(big.Int), err
|
||||
}
|
||||
// One money place is 1e14 residue units. Rounding compares twice the
|
||||
// remainder against that, so a tie rounds away from zero.
|
||||
place := new(big.Int).Exp(big.NewInt(10), big.NewInt(residueScale-4), nil)
|
||||
rounded, remainder := new(big.Int).QuoRem(magnitude, place, new(big.Int))
|
||||
if new(big.Int).Lsh(remainder, 1).Cmp(place) >= 0 {
|
||||
rounded.Add(rounded, big.NewInt(1))
|
||||
}
|
||||
if !rounded.IsInt64() {
|
||||
return "", new(big.Int), errors.New("value is out of range for money")
|
||||
}
|
||||
residue := new(big.Int).Sub(magnitude, new(big.Int).Mul(rounded, place))
|
||||
minor := rounded.Int64()
|
||||
if negative {
|
||||
minor, residue = -minor, residue.Neg(residue)
|
||||
}
|
||||
return domain.FormatMoney(minor), residue, nil
|
||||
}
|
||||
|
||||
// brokerQuantity reads one share count or unit price. Nothing is rounded: a
|
||||
// holding is verified against the broker's own figure, and a rounded price
|
||||
// would break the shares-times-price check the amount is verified against, so
|
||||
// a value beyond eight decimal places is refused instead of truncated.
|
||||
func brokerQuantity(value string, german bool) (domain.Quantity, error) {
|
||||
plain, ok, err := brokerPlain(value, german)
|
||||
if !ok || err != nil {
|
||||
return "", err
|
||||
}
|
||||
return domain.ParseQuantity(plain)
|
||||
}
|
||||
|
||||
// brokerPlain normalizes one numeric cell to a plain decimal string, or reports
|
||||
// that the cell was blank. Insignificant trailing zeros are dropped: exporters
|
||||
// pad a column to a fixed width, so a six-place price arrives written to ten,
|
||||
// and the padding would otherwise exhaust the precision the value needs.
|
||||
func brokerPlain(value string, german bool) (string, bool, error) {
|
||||
value = strings.NewReplacer("\u00a0", "", "\u202f", "", "'", "").Replace(strings.TrimSpace(value))
|
||||
if value == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
plain := strings.TrimPrefix(value, "+")
|
||||
if german {
|
||||
converted, err := germanDecimal(plain)
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
plain = converted
|
||||
}
|
||||
if whole, fraction, found := strings.Cut(plain, "."); found {
|
||||
if trimmed := strings.TrimRight(fraction, "0"); trimmed == "" {
|
||||
plain = whole
|
||||
} else {
|
||||
plain = whole + "." + trimmed
|
||||
}
|
||||
}
|
||||
return plain, true, nil
|
||||
}
|
||||
|
||||
// brokerDigits splits a plain decimal string into its exact magnitude in
|
||||
// residue units and its sign.
|
||||
func brokerDigits(plain string) (magnitude *big.Int, negative bool, err error) {
|
||||
digits := plain
|
||||
if rest, cut := strings.CutPrefix(digits, "-"); cut {
|
||||
negative, digits = true, rest
|
||||
}
|
||||
whole, decimals, _ := strings.Cut(digits, ".")
|
||||
if whole == "" {
|
||||
return nil, false, errors.New("decimal needs a digit before the separator")
|
||||
}
|
||||
if len(decimals) > residueScale {
|
||||
return nil, false, fmt.Errorf("more than %d fractional digits", residueScale)
|
||||
}
|
||||
scaled, ok := new(big.Int).SetString(whole+decimals+strings.Repeat("0", residueScale-len(decimals)), 10)
|
||||
if !ok {
|
||||
return nil, false, errors.New("not a decimal number")
|
||||
}
|
||||
return scaled, negative, nil
|
||||
}
|
||||
|
||||
// decimalString renders exact units at a scale without trailing zeros, so an
|
||||
// adjustment of 1e-9 is reported as such rather than padded to eighteen places.
|
||||
func decimalString(units *big.Int, scale int) string {
|
||||
sign := ""
|
||||
magnitude := new(big.Int).Abs(units)
|
||||
if units.Sign() < 0 {
|
||||
sign = "-"
|
||||
}
|
||||
digits := magnitude.String()
|
||||
if len(digits) <= scale {
|
||||
digits = strings.Repeat("0", scale+1-len(digits)) + digits
|
||||
}
|
||||
whole, fraction := digits[:len(digits)-scale], strings.TrimRight(digits[len(digits)-scale:], "0")
|
||||
if fraction == "" {
|
||||
return sign + whole
|
||||
}
|
||||
return sign + whole + "." + fraction
|
||||
}
|
||||
|
||||
// brokerSettlement is gross minus fee minus tax: the cash a row moved. Fee and
|
||||
// tax are stored as deductions, so a refunded tax is a negative deduction and
|
||||
// adds to the cash.
|
||||
func brokerSettlement(gross, fee, tax domain.Money) (domain.Money, error) {
|
||||
total := int64(0)
|
||||
for _, deduction := range []struct {
|
||||
sign int64
|
||||
money domain.Money
|
||||
}{{1, gross}, {-1, fee}, {-1, tax}} {
|
||||
if deduction.money == "" {
|
||||
continue
|
||||
}
|
||||
minor, err := deduction.money.Minor()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
total += deduction.sign * minor
|
||||
}
|
||||
return domain.FormatMoney(total), nil
|
||||
}
|
||||
+15
-216
@@ -39,64 +39,9 @@ var scalableEvents = map[string]string{
|
||||
"security transfer": domain.EventPositionTransfer,
|
||||
}
|
||||
|
||||
// ScalableNote records a figure the export carried that the import deliberately
|
||||
// did not apply, so it can be reviewed before confirming and recognized later
|
||||
// if a balance disagrees.
|
||||
type ScalableNote struct {
|
||||
Record int `json:"record"`
|
||||
Date string `json:"date"`
|
||||
Description string `json:"description"`
|
||||
Fee domain.Money `json:"fee,omitempty"`
|
||||
Tax domain.Money `json:"tax,omitempty"`
|
||||
}
|
||||
|
||||
// ScalableImport is a read broker export awaiting review.
|
||||
type ScalableImport struct {
|
||||
Facts []domain.Facts `json:"-"`
|
||||
// Instruments are securities the export named that the registry does not
|
||||
// hold yet. An import never renames an existing instrument: the name is
|
||||
// editable display text, and the export's own description for one ISIN
|
||||
// changes over time.
|
||||
Instruments []domain.Instrument `json:"instruments"`
|
||||
// Cancelled counts rows the broker did not execute. Their money and share
|
||||
// columns are all zeros, so they satisfy every arithmetic check and would
|
||||
// otherwise import as phantom trades.
|
||||
Cancelled int `json:"cancelled"`
|
||||
// Rounded counts rows whose money carried more than four decimal places,
|
||||
// and Rounding is the exact total adjustment that rounding applied, at
|
||||
// whatever precision the export used.
|
||||
Rounded int `json:"rounded"`
|
||||
Rounding string `json:"rounding"`
|
||||
// Unapplied lists cash rows carrying a fee or tax. A broker cash amount is
|
||||
// already net of them, so subtracting them again would double-count; they
|
||||
// are recorded on the fact and reported here.
|
||||
Unapplied []ScalableNote `json:"unapplied"`
|
||||
}
|
||||
|
||||
// DetectScalableCSV reports whether a document is a Scalable Capital export and
|
||||
// which 1-based record holds its header.
|
||||
func DetectScalableCSV(f CSVFile) (header int, ok bool) {
|
||||
for i, row := range f.rows {
|
||||
if i >= maxCSVPreambleRows {
|
||||
break
|
||||
}
|
||||
columns, usable := csvColumnIndex(row)
|
||||
if !usable || len(columns) != len(scalableColumns) {
|
||||
continue
|
||||
}
|
||||
matched := true
|
||||
for _, name := range scalableColumns {
|
||||
if _, exists := columns[name]; !exists {
|
||||
matched = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if matched {
|
||||
return i + 1, true
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
// DetectScalableCSV reports whether a document is a Scalable Capital export
|
||||
// and which 1-based record holds its header.
|
||||
func DetectScalableCSV(f CSVFile) (header int, ok bool) { return matchColumns(f, scalableColumns) }
|
||||
|
||||
// ParseScalableCSV converts a broker export into bank facts carrying position
|
||||
// legs.
|
||||
@@ -125,26 +70,16 @@ func DetectScalableCSV(f CSVFile) (header int, ok bool) {
|
||||
// check, rejects the whole file. Every one of those cases can move money, and a
|
||||
// partially imported broker history cannot be told from a truncated export
|
||||
// afterwards.
|
||||
func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instrument) (ScalableImport, error) {
|
||||
// Empty rather than nil: these are arrays in the reviewed JSON, and a null
|
||||
// where a caller expects a list is a bug waiting on a different machine.
|
||||
result := ScalableImport{Instruments: []domain.Instrument{}, Unapplied: []ScalableNote{}}
|
||||
if account.ID == "" {
|
||||
return result, errors.New("broker import requires a selected account")
|
||||
}
|
||||
if !account.Investing() {
|
||||
return result, fmt.Errorf("account %q must be an investment account to hold a broker export", account.DisplayName)
|
||||
func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instrument) (BrokerImport, error) {
|
||||
result := newBrokerImport()
|
||||
if err := investmentTarget(account); err != nil {
|
||||
return result, err
|
||||
}
|
||||
header, ok := DetectScalableCSV(f)
|
||||
if !ok {
|
||||
return result, errors.New("not a Scalable Capital export")
|
||||
}
|
||||
headers := f.rows[header-1]
|
||||
index := map[string]int{}
|
||||
for i, raw := range headers {
|
||||
index[headerName(raw)] = i
|
||||
}
|
||||
cell := func(row []string, name string) string { return strings.TrimSpace(row[index[name]]) }
|
||||
headers, cell := brokerColumnIndex(f, header)
|
||||
|
||||
instruments := map[string]domain.Instrument{}
|
||||
byISIN := map[string]domain.Instrument{}
|
||||
@@ -216,15 +151,15 @@ func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instr
|
||||
result.Instruments[slot].Name = description
|
||||
}
|
||||
}
|
||||
amount, amountDrift, err := scalableMoney(cell(row, "amount"))
|
||||
amount, amountDrift, err := brokerMoney(cell(row, "amount"), decimalGerman)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid amount %q: %w", record, cell(row, "amount"), err)
|
||||
}
|
||||
fee, feeDrift, err := scalableMoney(cell(row, "fee"))
|
||||
fee, feeDrift, err := brokerMoney(cell(row, "fee"), decimalGerman)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid fee %q: %w", record, cell(row, "fee"), err)
|
||||
}
|
||||
tax, taxDrift, err := scalableMoney(cell(row, "tax"))
|
||||
tax, taxDrift, err := brokerMoney(cell(row, "tax"), decimalGerman)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid tax %q: %w", record, cell(row, "tax"), err)
|
||||
}
|
||||
@@ -235,24 +170,21 @@ func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instr
|
||||
cash := amount
|
||||
if investment.CashOnly() {
|
||||
if nonzeroMoney(fee) || nonzeroMoney(tax) {
|
||||
result.Unapplied = append(result.Unapplied, ScalableNote{Record: record, Date: booking, Description: description, Fee: fee, Tax: tax})
|
||||
result.Unapplied = append(result.Unapplied, BrokerNote{Record: record, Date: booking, Description: description, Fee: fee, Tax: tax})
|
||||
}
|
||||
investment.Fee, investment.Tax = fee, tax
|
||||
} else {
|
||||
if isin == "" {
|
||||
return result, fmt.Errorf("broker record %d moves a position without an ISIN", record)
|
||||
}
|
||||
shares, err := scalableQuantity(cell(row, "shares"))
|
||||
shares, err := brokerQuantity(cell(row, "shares"), decimalGerman)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid share count %q: %w", record, cell(row, "shares"), err)
|
||||
}
|
||||
price, priceDrift, err := scalableMoney(cell(row, "price"))
|
||||
price, err := brokerQuantity(cell(row, "price"), decimalGerman)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid price %q: %w", record, cell(row, "price"), err)
|
||||
}
|
||||
if priceDrift.Sign() != 0 {
|
||||
return result, fmt.Errorf("broker record %d has a price %q beyond four decimal places", record, cell(row, "price"))
|
||||
}
|
||||
signed, err := scalableSignedShares(event, shares)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
@@ -265,7 +197,7 @@ func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instr
|
||||
cash = "0.00"
|
||||
} else {
|
||||
investment.Fee, investment.Tax = fee, tax
|
||||
if cash, err = scalableSettlement(amount, fee, tax); err != nil {
|
||||
if cash, err = brokerSettlement(amount, fee, tax); err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
}
|
||||
@@ -294,118 +226,6 @@ func ParseScalableCSV(f CSVFile, account domain.Account, registry []domain.Instr
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// nonzeroMoney reports a figure that could change a balance. The export leaves
|
||||
// a column blank where it does not apply and writes an explicit zero where it
|
||||
// applies but is nil; only the second kind is worth putting in front of
|
||||
// someone before they confirm an import.
|
||||
func nonzeroMoney(m domain.Money) bool {
|
||||
minor, err := m.Minor()
|
||||
return err == nil && minor != 0
|
||||
}
|
||||
|
||||
// residueScale is the precision the discarded remainder is accumulated at.
|
||||
// A broker amount is its share count times its price, so it carries as many
|
||||
// decimal places as the two together need: a real export reinvests to nine.
|
||||
// Eighteen is far past anything a settlement can produce and still exact.
|
||||
const residueScale = 18
|
||||
|
||||
// scalableMoney reads one German-formatted money cell, rounds it to money's
|
||||
// four decimal places half away from zero, and returns the exact remainder
|
||||
// that rounding discarded, in units of 1e-18. The remainder is reported rather
|
||||
// than hidden, and never guessed at: it is the only honest account of why a
|
||||
// computed balance can differ from the broker's by a fraction of a cent.
|
||||
//
|
||||
// An empty cell is empty money, not zero: blank marks a column that does not
|
||||
// apply to the row.
|
||||
func scalableMoney(value string) (domain.Money, *big.Int, error) {
|
||||
plain, ok, err := scalablePlain(value)
|
||||
if !ok || err != nil {
|
||||
return "", new(big.Int), err
|
||||
}
|
||||
magnitude, negative, err := scalableDigits(plain)
|
||||
if err != nil {
|
||||
return "", new(big.Int), err
|
||||
}
|
||||
// One money place is 1e14 residue units. Rounding compares twice the
|
||||
// remainder against that, so a tie rounds away from zero.
|
||||
place := new(big.Int).Exp(big.NewInt(10), big.NewInt(residueScale-4), nil)
|
||||
rounded, remainder := new(big.Int).QuoRem(magnitude, place, new(big.Int))
|
||||
if new(big.Int).Lsh(remainder, 1).Cmp(place) >= 0 {
|
||||
rounded.Add(rounded, big.NewInt(1))
|
||||
}
|
||||
if !rounded.IsInt64() {
|
||||
return "", new(big.Int), fmt.Errorf("value is out of range for money")
|
||||
}
|
||||
residue := new(big.Int).Sub(magnitude, new(big.Int).Mul(rounded, place))
|
||||
minor := rounded.Int64()
|
||||
if negative {
|
||||
minor, residue = -minor, residue.Neg(residue)
|
||||
}
|
||||
return domain.FormatMoney(minor), residue, nil
|
||||
}
|
||||
|
||||
// scalableDigits splits a plain decimal string into its exact magnitude in
|
||||
// residue units and its sign.
|
||||
func scalableDigits(plain string) (magnitude *big.Int, negative bool, err error) {
|
||||
digits := plain
|
||||
if rest, cut := strings.CutPrefix(digits, "-"); cut {
|
||||
negative, digits = true, rest
|
||||
}
|
||||
whole, decimals, _ := strings.Cut(digits, ".")
|
||||
if whole == "" {
|
||||
return nil, false, fmt.Errorf("decimal needs a digit before the separator")
|
||||
}
|
||||
if len(decimals) > residueScale {
|
||||
return nil, false, fmt.Errorf("more than %d fractional digits", residueScale)
|
||||
}
|
||||
scaled, ok := new(big.Int).SetString(whole+decimals+strings.Repeat("0", residueScale-len(decimals)), 10)
|
||||
if !ok {
|
||||
return nil, false, fmt.Errorf("not a decimal number")
|
||||
}
|
||||
return scaled, negative, nil
|
||||
}
|
||||
|
||||
// decimalString renders exact units at a scale without trailing zeros, so an
|
||||
// adjustment of 1e-9 is reported as such rather than padded to eighteen places.
|
||||
func decimalString(units *big.Int, scale int) string {
|
||||
sign := ""
|
||||
magnitude := new(big.Int).Abs(units)
|
||||
if units.Sign() < 0 {
|
||||
sign = "-"
|
||||
}
|
||||
digits := magnitude.String()
|
||||
if len(digits) <= scale {
|
||||
digits = strings.Repeat("0", scale+1-len(digits)) + digits
|
||||
}
|
||||
whole, fraction := digits[:len(digits)-scale], strings.TrimRight(digits[len(digits)-scale:], "0")
|
||||
if fraction == "" {
|
||||
return sign + whole
|
||||
}
|
||||
return sign + whole + "." + fraction
|
||||
}
|
||||
|
||||
// scalableQuantity reads one German-formatted share count. Nothing is rounded:
|
||||
// a holding is verified against the broker's own figure, so a count beyond
|
||||
// eight decimal places is refused instead of silently truncated.
|
||||
func scalableQuantity(value string) (domain.Quantity, error) {
|
||||
plain, ok, err := scalablePlain(value)
|
||||
if !ok || err != nil {
|
||||
return "", err
|
||||
}
|
||||
return domain.ParseQuantity(plain)
|
||||
}
|
||||
|
||||
// scalablePlain normalizes one numeric cell to a plain decimal string, or
|
||||
// reports that the cell was blank.
|
||||
func scalablePlain(value string) (string, bool, error) {
|
||||
value = strings.NewReplacer("\u00a0", "", "\u202f", "", "'", "").Replace(strings.TrimSpace(value))
|
||||
if value == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
plain, err := germanDecimal(value)
|
||||
return plain, err == nil, err
|
||||
}
|
||||
|
||||
// scalableSignedShares resolves the export's two sign conventions. A buy, sell
|
||||
// or reinvestment carries an unsigned count and takes its direction from the
|
||||
// type; a corporate action or depot transfer is already signed.
|
||||
@@ -428,24 +248,3 @@ func scalableSignedShares(event string, shares domain.Quantity) (domain.Quantity
|
||||
}
|
||||
return domain.FormatQuantity(units), nil
|
||||
}
|
||||
|
||||
// scalableSettlement is gross minus fee minus tax: the cash a trade moved. The
|
||||
// broker states fee and tax as positive deductions whichever way the trade
|
||||
// went, so both are subtracted from a signed gross.
|
||||
func scalableSettlement(gross, fee, tax domain.Money) (domain.Money, error) {
|
||||
total := int64(0)
|
||||
for _, deduction := range []struct {
|
||||
sign int64
|
||||
money domain.Money
|
||||
}{{1, gross}, {-1, fee}, {-1, tax}} {
|
||||
if deduction.money == "" {
|
||||
continue
|
||||
}
|
||||
minor, err := deduction.money.Minor()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
total += deduction.sign * minor
|
||||
}
|
||||
return domain.FormatMoney(total), nil
|
||||
}
|
||||
|
||||
@@ -42,7 +42,7 @@ func brokerAccount() domain.Account {
|
||||
}
|
||||
}
|
||||
|
||||
func readBroker(t *testing.T, rows ...string) ScalableImport {
|
||||
func readBroker(t *testing.T, rows ...string) BrokerImport {
|
||||
t.Helper()
|
||||
file, err := ReadCSV(strings.NewReader(scalableHeader + strings.Join(rows, "\n") + "\n"))
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
package banking
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/big"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"finance-duck/internal/domain"
|
||||
)
|
||||
|
||||
// SourceTradeRepublic identifies facts imported from a Trade Republic export.
|
||||
const SourceTradeRepublic = "traderepublic_csv"
|
||||
|
||||
// tradeRepublicColumns are the exact normalized headers of a Trade Republic
|
||||
// transaction export.
|
||||
var tradeRepublicColumns = []string{
|
||||
"datetime", "date", "account_type", "category", "type", "asset_class",
|
||||
"name", "symbol", "shares", "price", "amount", "fee", "tax", "currency",
|
||||
"original_amount", "original_currency", "fx_rate", "description",
|
||||
"transaction_id", "counterparty_name", "counterparty_iban", "payment_reference", "mcc_code",
|
||||
}
|
||||
|
||||
// tradeRepublicEvents maps the export's complete type vocabulary to journal
|
||||
// events. The set is closed on purpose: an unrecognized type could move cash in
|
||||
// either direction, or none, and defaulting it risks a silent balance error.
|
||||
var tradeRepublicEvents = map[string]string{
|
||||
"TRANSFER_INBOUND": domain.EventDeposit,
|
||||
"TRANSFER_INSTANT_INBOUND": domain.EventDeposit,
|
||||
"TRANSFER_OUTBOUND": domain.EventWithdrawal,
|
||||
"TRANSFER_INSTANT_OUTBOUND": domain.EventWithdrawal,
|
||||
"INTEREST_PAYMENT": domain.EventInterest,
|
||||
"DIVIDEND": domain.EventDistribution,
|
||||
"TAX_OPTIMIZATION": domain.EventTaxSettlement,
|
||||
"BUY": domain.EventBuy,
|
||||
"SELL": domain.EventSell,
|
||||
}
|
||||
|
||||
// isinInText finds the security identifier a row names in its free text. Trade
|
||||
// Republic puts an ISIN in the symbol column for funds and shares, but a bare
|
||||
// ticker for crypto, whose ISIN-shaped identifier appears only in the
|
||||
// description: "Sell trade XF000DOGE012 Dogecoin".
|
||||
var isinInText = regexp.MustCompile(`\b[A-Z]{2}[A-Z0-9]{9}[0-9]\b`)
|
||||
|
||||
// ibanInText finds the counterparty a transfer names in its free text. Older
|
||||
// rows leave the counterparty_iban column empty and write the IBAN in
|
||||
// parentheses instead: "Outgoing transfer for LARS NOLDEN (DE04...)".
|
||||
var ibanInText = regexp.MustCompile(`\(([A-Z]{2}[0-9]{2}[A-Z0-9]{10,30})\)`)
|
||||
|
||||
// ParseTradeRepublicCSV converts a Trade Republic export into bank facts
|
||||
// carrying position legs.
|
||||
//
|
||||
// Three conventions differ from every other export this program reads, and each
|
||||
// one moves money if it is read the other way round:
|
||||
//
|
||||
// - fee and tax are signed adjustments to cash, not deductions from a gross.
|
||||
// The export writes a one euro order fee as -1.00 and withheld tax as
|
||||
// -4.33, so both are negated at import and the journal keeps one
|
||||
// convention: cash is gross minus fee minus tax.
|
||||
// - a cash row's amount is the gross, not the net. Interest of 16.46 with
|
||||
// -4.33 of tax credits 12.13. This is the opposite of an export that
|
||||
// states its cash already net, where the tax is recorded and never
|
||||
// applied.
|
||||
// - a TAX_OPTIMIZATION row carries zero in the amount column and the money
|
||||
// in the tax column, signed both ways. Read as cash, all six of them move
|
||||
// nothing; read correctly, they are the loss-offset pot settling.
|
||||
//
|
||||
// A dividend row populates the share column with the holding the dividend was
|
||||
// paid on, not with a position change. Adding it would double the holding, so
|
||||
// it is read as the attribution it is and discarded.
|
||||
//
|
||||
// The amount on a trade is the notional rounded to cents, not the exact
|
||||
// product, so the shares-times-price check is satisfied to the precision the
|
||||
// broker stated rather than exactly. Of 59 trades in a real export, 30 are
|
||||
// exact at four places and all 59 are within a cent.
|
||||
//
|
||||
// The booking date is the date column exactly as printed. The datetime column
|
||||
// is UTC while the date column is local, so they disagree for rows booked late
|
||||
// in the evening and deriving the date from the timestamp would move them to
|
||||
// the previous day.
|
||||
func ParseTradeRepublicCSV(f CSVFile, account domain.Account, registry []domain.Instrument) (BrokerImport, error) {
|
||||
result := newBrokerImport()
|
||||
if err := investmentTarget(account); err != nil {
|
||||
return result, err
|
||||
}
|
||||
header, ok := DetectTradeRepublicCSV(f)
|
||||
if !ok {
|
||||
return result, errors.New("not a Trade Republic export")
|
||||
}
|
||||
headers, cell := brokerColumnIndex(f, header)
|
||||
|
||||
instruments := map[string]domain.Instrument{}
|
||||
byISIN := map[string]domain.Instrument{}
|
||||
for _, v := range registry {
|
||||
instruments[v.ID] = v
|
||||
byISIN[v.ISIN] = v
|
||||
}
|
||||
created := map[string]int{}
|
||||
named := map[string]string{}
|
||||
drift := new(big.Int)
|
||||
for offset, row := range f.rows[header:] {
|
||||
record := header + offset + 1
|
||||
if blankCSVRow(row) {
|
||||
continue
|
||||
}
|
||||
if len(row) != len(headers) {
|
||||
return result, fmt.Errorf("broker record %d has %d columns, expected %d", record, len(row), len(headers))
|
||||
}
|
||||
// One export covers one account. A second account type in the same file
|
||||
// would silently merge two cash balances into one.
|
||||
if kind := cell(row, "account_type"); !strings.EqualFold(kind, "DEFAULT") {
|
||||
return result, fmt.Errorf("broker record %d belongs to account type %q, and only DEFAULT can be imported into one account", record, kind)
|
||||
}
|
||||
rawType, category := cell(row, "type"), cell(row, "category")
|
||||
event, known := tradeRepublicEvents[strings.ToUpper(strings.TrimSpace(rawType))]
|
||||
if !known {
|
||||
return result, fmt.Errorf("broker record %d has unknown type %q: it may or may not move cash, so nothing was imported", record, rawType)
|
||||
}
|
||||
investment := domain.Investment{Event: event}
|
||||
wanted := "TRADING"
|
||||
if investment.CashOnly() {
|
||||
wanted = "CASH"
|
||||
}
|
||||
if !strings.EqualFold(category, wanted) {
|
||||
return result, fmt.Errorf("broker record %d pairs type %q with category %q, expected %q", record, rawType, category, wanted)
|
||||
}
|
||||
currency := strings.ToUpper(cell(row, "currency"))
|
||||
if currency != strings.ToUpper(account.Currency) {
|
||||
return result, fmt.Errorf("broker record %d settles in %q but account %q holds %s: currency conversion is not supported", record, currency, account.DisplayName, account.Currency)
|
||||
}
|
||||
booking, err := parseMappedCSVDate(cell(row, "date"), "yyyy-mm-dd")
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid date %q", record, cell(row, "date"))
|
||||
}
|
||||
description := cell(row, "description")
|
||||
isin, err := tradeRepublicISIN(cell(row, "symbol"), description, !investment.CashOnly())
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
if isin != "" {
|
||||
held, exists := byISIN[isin]
|
||||
if !exists {
|
||||
name := cell(row, "name")
|
||||
if name == "" {
|
||||
name = isin
|
||||
}
|
||||
held = domain.Instrument{ID: domain.InstrumentID(isin), ISIN: isin, Name: name, Currency: currency}
|
||||
byISIN[isin] = held
|
||||
instruments[held.ID] = held
|
||||
created[isin] = len(result.Instruments)
|
||||
result.Instruments = append(result.Instruments, held)
|
||||
}
|
||||
investment.InstrumentID = held.ID
|
||||
slot, mine := created[isin]
|
||||
if name := cell(row, "name"); mine && name != "" && name != isin && booking >= named[isin] {
|
||||
named[isin] = booking
|
||||
result.Instruments[slot].Name = name
|
||||
}
|
||||
}
|
||||
gross, grossDrift, err := brokerMoney(cell(row, "amount"), decimalPlain)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid amount %q: %w", record, cell(row, "amount"), err)
|
||||
}
|
||||
fee, feeDrift, err := brokerMoney(cell(row, "fee"), decimalPlain)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid fee %q: %w", record, cell(row, "fee"), err)
|
||||
}
|
||||
tax, taxDrift, err := brokerMoney(cell(row, "tax"), decimalPlain)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid tax %q: %w", record, cell(row, "tax"), err)
|
||||
}
|
||||
if grossDrift.Sign() != 0 || feeDrift.Sign() != 0 || taxDrift.Sign() != 0 {
|
||||
result.Rounded++
|
||||
drift.Add(drift, grossDrift).Add(drift, feeDrift).Add(drift, taxDrift)
|
||||
}
|
||||
// The export states what it took off the cash; the journal stores what
|
||||
// was deducted from the gross.
|
||||
if fee, err = negated(fee); err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
if tax, err = negated(tax); err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
if investment.CashOnly() {
|
||||
// The share column on a dividend is the holding it was paid on.
|
||||
investment.Gross, investment.Fee, investment.Tax = gross, fee, tax
|
||||
} else {
|
||||
if isin == "" {
|
||||
return result, fmt.Errorf("broker record %d moves a position without a security identifier", record)
|
||||
}
|
||||
shares, err := brokerQuantity(cell(row, "shares"), decimalPlain)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid share count %q: %w", record, cell(row, "shares"), err)
|
||||
}
|
||||
price, err := brokerQuantity(cell(row, "price"), decimalPlain)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid price %q: %w", record, cell(row, "price"), err)
|
||||
}
|
||||
investment.Quantity, investment.Price, investment.Gross = shares, price, gross
|
||||
investment.Fee, investment.Tax = fee, tax
|
||||
}
|
||||
cash, err := brokerSettlement(gross, fee, tax)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
facts := domain.Facts{
|
||||
Source: SourceTradeRepublic, AccountID: account.ID, BookingDate: booking,
|
||||
Amount: cash, Currency: currency, RawDescription: description,
|
||||
ExternalID: cell(row, "transaction_id"), Counterparty: cell(row, "counterparty_name"),
|
||||
Investment: &investment,
|
||||
}
|
||||
if investment.Event == domain.EventDeposit || investment.Event == domain.EventWithdrawal {
|
||||
facts.CounterpartyIBAN = tradeRepublicIBAN(cell(row, "counterparty_iban"), description, account.ReferenceIBAN)
|
||||
}
|
||||
if err := domain.ValidateInvestment(facts, account, instruments); err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
result.Facts = append(result.Facts, facts)
|
||||
}
|
||||
if len(result.Facts) == 0 {
|
||||
return result, errors.New("broker export contains no records")
|
||||
}
|
||||
result.Rounding = decimalString(drift, residueScale)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// DetectTradeRepublicCSV reports whether a document is a Trade Republic export
|
||||
// and which 1-based record holds its header.
|
||||
func DetectTradeRepublicCSV(f CSVFile) (header int, ok bool) {
|
||||
return matchColumns(f, tradeRepublicColumns)
|
||||
}
|
||||
|
||||
// tradeRepublicISIN resolves the security a row names. The symbol column holds
|
||||
// an ISIN for funds and shares and a bare ticker for crypto, whose ISIN-shaped
|
||||
// identifier appears only in the description. Exactly one identifier must be
|
||||
// findable, or the row is refused rather than attached to a guess.
|
||||
func tradeRepublicISIN(symbol, description string, required bool) (string, error) {
|
||||
candidate := strings.ToUpper(strings.Join(strings.Fields(symbol), ""))
|
||||
if domain.ValidISIN(candidate) {
|
||||
return candidate, nil
|
||||
}
|
||||
found := isinInText.FindAllString(description, -1)
|
||||
unique := map[string]bool{}
|
||||
for _, match := range found {
|
||||
if domain.ValidISIN(match) {
|
||||
unique[match] = true
|
||||
}
|
||||
}
|
||||
if len(unique) == 1 {
|
||||
for match := range unique {
|
||||
return match, nil
|
||||
}
|
||||
}
|
||||
if !required {
|
||||
return "", nil
|
||||
}
|
||||
if candidate == "" {
|
||||
return "", errors.New("row moves a position but names no security")
|
||||
}
|
||||
return "", fmt.Errorf("symbol %q is not an ISIN and its description does not name exactly one", symbol)
|
||||
}
|
||||
|
||||
// tradeRepublicIBAN resolves the account a transfer settles against: the
|
||||
// export's own column when it has one, else the IBAN the description carries in
|
||||
// parentheses, else the account's configured settlement IBAN. Free text only
|
||||
// contributes a value that is shaped like an IBAN, so a description that names
|
||||
// no account contributes nothing.
|
||||
func tradeRepublicIBAN(column, description, fallback string) string {
|
||||
if iban := normalizeIBAN(column); iban != "" {
|
||||
return iban
|
||||
}
|
||||
if match := ibanInText.FindStringSubmatch(strings.ToUpper(description)); match != nil {
|
||||
return normalizeIBAN(match[1])
|
||||
}
|
||||
return normalizeIBAN(fallback)
|
||||
}
|
||||
@@ -0,0 +1,239 @@
|
||||
package banking
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"finance-duck/internal/domain"
|
||||
)
|
||||
|
||||
const tradeRepublicHeader = "datetime;date;account_type;category;type;asset_class;name;symbol;shares;price;amount;fee;tax;currency;original_amount;original_currency;fx_rate;description;transaction_id;counterparty_name;counterparty_iban;payment_reference;mcc_code\n"
|
||||
|
||||
// Real Trade Republic export lines. Between them they cover every one of the
|
||||
// nine row types, both sign conventions for a transfer, a trade whose notional
|
||||
// does not land on a whole cent, a crypto trade whose identifier is only in the
|
||||
// description, a tax settlement that carries its money in the tax column, a
|
||||
// dividend whose share column is the holding rather than a position change, and
|
||||
// numbers padded with insignificant zeros.
|
||||
var tradeRepublicRows = []string{
|
||||
`2025-01-10T13:17:25.211420Z;2025-01-10;DEFAULT;CASH;TRANSFER_INBOUND;;;;;;34337.000000;;;EUR;;;;Incoming transfer from LARS NOLDEN;cccf7fb9-f35a-462a-8d2c-162664479274;;;;`,
|
||||
`2025-01-16T13:59:44.872Z;2025-01-16;DEFAULT;TRADING;BUY;FUND;Edge MSCI World Min Volatility USD (Acc);IE00B8FHGS14;0.9493860000;64.410000;-61.15;;;EUR;;;;Buy trade IE00B8FHGS14 iShares VI plc, quantity: 0.949386;ebbc70c1-a260-4e59-b499-14dcec7e6f04;;;;`,
|
||||
`2025-01-16T13:59:45.293Z;2025-01-16;DEFAULT;TRADING;BUY;FUND;Edge MSCI World Min Volatility USD (Acc);IE00B8FHGS14;485.0000000000;64.410000;-31238.85;-1.00;;EUR;;;;Buy trade IE00B8FHGS14 iShares VI plc, quantity: 485;93aaf560-5d26-4fd3-95ab-3cff4e5f1b12;;;;`,
|
||||
`2025-01-18T00:27:04.446Z;2025-01-18;DEFAULT;TRADING;BUY;CRYPTO;Dogecoin;DOGE;865.7000000000;0.415787;-359.95;-1.00;;EUR;;;;Ausfuehrung Kauf/Verkauf XF000DOGE012;f305e14c-b9a5-43eb-adbc-b8b00f579c80;;;;`,
|
||||
`2025-02-01T12:24:38.795049Z;2025-02-01;DEFAULT;CASH;INTEREST_PAYMENT;;;;;;16.460000;;-4.33;EUR;;;;Interest payment Booking;94ad7cae-6b55-4d11-83ef-668c397e9391;;;;`,
|
||||
`2025-02-10T13:29:45.670Z;2025-02-10;DEFAULT;TRADING;SELL;FUND;Edge MSCI World Min Volatility USD (Acc);IE00B8FHGS14;-20.0000000000;67.210000;1344.20;-1.00;-10.14;EUR;;;;Sell trade IE00B8FHGS14 iShares VI plc, quantity: 20;7b647416-c8e8-45bf-beea-2aea65e3950a;;;;`,
|
||||
`2025-03-07T02:29:08.390291Z;2025-03-07;DEFAULT;CASH;TAX_OPTIMIZATION;;;;;;0.000000;;14.95;EUR;;;;Tax Optimisation;b9a02670-b419-42d4-a8d7-d0336d9ae9cb;;;;`,
|
||||
`2025-09-30T12:49:02.644Z;2025-09-30;DEFAULT;TRADING;BUY;STOCK;DroneShield;AU000000DRO2;167.0000000000;2.9800000000;-497.66;-1.00;;EUR;;;;Buy trade AU000000DRO2 DRONESHIELD LTD, quantity: 167.0;9b08e71c-5d85-49c4-bcf3-31bc7671a278;;;;`,
|
||||
`2025-10-06T09:02:07.835Z;2025-10-06;DEFAULT;TRADING;SELL;FUND;Edge MSCI World Min Volatility USD (Acc);IE00B8FHGS14;-0.4265810000;63.0600000000;26.90;-1.00;;EUR;;;;Sell trade IE00B8FHGS14 iShares VI plc, quantity: 0.426581;dcd1df8b-3324-4f71-be38-4a2e6cde326d;;;;`,
|
||||
`2025-12-23T12:44:47.627337Z;2025-12-23;DEFAULT;CASH;TRANSFER_INSTANT_OUTBOUND;;;;;;-5700.000000;;;EUR;;;;Outgoing transfer for Lars Nolden (DE04100110012623927730);019b4b3d-9c8b-7e5a-b17f-9c884edc0ae8;;;;`,
|
||||
`2026-01-27T08:44:12.845140Z;2026-01-27;DEFAULT;CASH;TAX_OPTIMIZATION;;;;;;0.000000;;-30.44;EUR;;;;Tax Optimisation;019bfe9f-eead-7321-ac84-d1aac355b444;;;;`,
|
||||
`2026-04-09T09:08:43.203685Z;2026-04-09;DEFAULT;CASH;DIVIDEND;STOCK;TSMC (ADR);US8740391003;24.9110320000;;15.790000;;-3.17;EUR;18.48;USD;0.854263;Cash Dividend for ISIN US8740391003;019d7180-3e43-7de4-bcd1-7a61a118944a;;;;`,
|
||||
`2026-05-11T16:08:03.023362Z;2026-05-11;DEFAULT;CASH;TRANSFER_INSTANT_OUTBOUND;;;;;;-1481.000000;;;EUR;;;;Outgoing transfer for LARS NOLDEN (DE41110101002098897347);019e17cb-a6cf-70f7-b602-886ffa8fdffe;LARS NOLDEN;DE41110101002098897347;;`,
|
||||
`2026-05-26T13:18:23.430Z;2026-05-26;DEFAULT;TRADING;SELL;CRYPTO;Dogecoin;DOGE;-865.7000000000;0.0879270000;76.12;-1.00;;EUR;;;;Sell trade XF000DOGE012 Dogecoin, quantity: 865.7;9a82e774-08a8-49bc-a706-6a65691b71c7;;;;`,
|
||||
}
|
||||
|
||||
func readTradeRepublic(t *testing.T, rows ...string) BrokerImport {
|
||||
t.Helper()
|
||||
file, err := ReadCSV(strings.NewReader(tradeRepublicHeader + strings.Join(rows, "\n") + "\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
result, err := ParseTradeRepublicCSV(file, brokerAccount(), nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func TestTradeRepublicSettlesGrossLessItsSignedAdjustments(t *testing.T) {
|
||||
result := readTradeRepublic(t, tradeRepublicRows...)
|
||||
if len(result.Facts) != len(tradeRepublicRows) {
|
||||
t.Fatalf("read %d of %d rows", len(result.Facts), len(tradeRepublicRows))
|
||||
}
|
||||
|
||||
// The export writes fee and tax as the signed adjustments it made, and the
|
||||
// amount as the gross. Cash is what is left, and a tax settlement's money
|
||||
// lives entirely in the tax column.
|
||||
wantCash := map[string]string{
|
||||
"cccf7fb9-f35a-462a-8d2c-162664479274": "34337.00",
|
||||
"ebbc70c1-a260-4e59-b499-14dcec7e6f04": "-61.15",
|
||||
"93aaf560-5d26-4fd3-95ab-3cff4e5f1b12": "-31239.85",
|
||||
"f305e14c-b9a5-43eb-adbc-b8b00f579c80": "-360.95",
|
||||
"94ad7cae-6b55-4d11-83ef-668c397e9391": "12.13",
|
||||
"7b647416-c8e8-45bf-beea-2aea65e3950a": "1333.06",
|
||||
"b9a02670-b419-42d4-a8d7-d0336d9ae9cb": "14.95",
|
||||
"9b08e71c-5d85-49c4-bcf3-31bc7671a278": "-498.66",
|
||||
"dcd1df8b-3324-4f71-be38-4a2e6cde326d": "25.90",
|
||||
"019b4b3d-9c8b-7e5a-b17f-9c884edc0ae8": "-5700.00",
|
||||
"019bfe9f-eead-7321-ac84-d1aac355b444": "-30.44",
|
||||
"019d7180-3e43-7de4-bcd1-7a61a118944a": "12.62",
|
||||
"019e17cb-a6cf-70f7-b602-886ffa8fdffe": "-1481.00",
|
||||
"9a82e774-08a8-49bc-a706-6a65691b71c7": "75.12",
|
||||
}
|
||||
total := int64(0)
|
||||
for _, f := range result.Facts {
|
||||
minor, err := f.Amount.Minor()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
total += minor
|
||||
if want, ok := wantCash[f.ExternalID]; !ok {
|
||||
t.Errorf("unexpected record %s", f.ExternalID)
|
||||
} else if string(f.Amount) != want {
|
||||
t.Errorf("%s settled %s, want %s", f.ExternalID, f.Amount, want)
|
||||
}
|
||||
}
|
||||
if got := string(domain.FormatMoney(total)); got != "-3561.27" {
|
||||
t.Errorf("cash balance %s, want -3561.27", got)
|
||||
}
|
||||
|
||||
holdings := map[string]int64{}
|
||||
for _, f := range result.Facts {
|
||||
if f.Investment.Quantity == "" {
|
||||
continue
|
||||
}
|
||||
units, err := f.Investment.Quantity.Units()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
holdings[f.Investment.InstrumentID] += units
|
||||
}
|
||||
for isin, want := range map[string]int64{
|
||||
"IE00B8FHGS14": 46552280500, // 0.949386 + 485 − 20 − 0.426581
|
||||
"XF000DOGE012": 0, // bought and sold whole
|
||||
"AU000000DRO2": 16700000000,
|
||||
"US8740391003": 0, // a dividend attributes to a security without moving it
|
||||
} {
|
||||
if got := holdings[domain.InstrumentID(isin)]; got != want {
|
||||
t.Errorf("%s holds %d hundred-millionths, want %d", isin, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Crypto carries a bare ticker in the symbol column, so its identifier
|
||||
// comes from the description, and the security is registered like any other.
|
||||
names := map[string]string{}
|
||||
for _, v := range result.Instruments {
|
||||
names[v.ISIN] = v.Name
|
||||
}
|
||||
for isin, want := range map[string]string{
|
||||
"XF000DOGE012": "Dogecoin",
|
||||
"IE00B8FHGS14": "Edge MSCI World Min Volatility USD (Acc)",
|
||||
"US8740391003": "TSMC (ADR)",
|
||||
"AU000000DRO2": "DroneShield",
|
||||
} {
|
||||
if names[isin] != want {
|
||||
t.Errorf("%s named %q, want %q", isin, names[isin], want)
|
||||
}
|
||||
}
|
||||
|
||||
// Padding is not precision: a value written to six or ten places with
|
||||
// trailing zeros needs no rounding at all.
|
||||
if result.Rounded != 0 || result.Rounding != "0" {
|
||||
t.Errorf("rounding reported as %d row(s) and %s, want none", result.Rounded, result.Rounding)
|
||||
}
|
||||
// The export applies its own fee and tax, so nothing is recorded unapplied.
|
||||
if len(result.Unapplied) != 0 {
|
||||
t.Errorf("unapplied notes on an export that nets its own cash: %+v", result.Unapplied)
|
||||
}
|
||||
}
|
||||
|
||||
// A dividend populates the share column with the holding the dividend was paid
|
||||
// on. Adding it as a position change would double the holding.
|
||||
func TestTradeRepublicDividendDoesNotMoveThePosition(t *testing.T) {
|
||||
result := readTradeRepublic(t, tradeRepublicRows[11])
|
||||
dividend := result.Facts[0].Investment
|
||||
if dividend.Quantity != "" || dividend.Price != "" {
|
||||
t.Fatalf("dividend moved a position: %+v", dividend)
|
||||
}
|
||||
if dividend.Event != domain.EventDistribution || dividend.InstrumentID == "" {
|
||||
t.Fatalf("dividend lost its attribution: %+v", dividend)
|
||||
}
|
||||
// 18.48 USD at 0.854263 is 15.79 EUR gross, less 3.17 withheld.
|
||||
if result.Facts[0].Amount != "12.62" || dividend.Gross != "15.79" || dividend.Tax != "3.17" {
|
||||
t.Fatalf("dividend settled %s from gross %s less tax %s", result.Facts[0].Amount, dividend.Gross, dividend.Tax)
|
||||
}
|
||||
}
|
||||
|
||||
// The counterparty comes from the column when the export has one, from the
|
||||
// IBAN the description names when it does not, and from the account's
|
||||
// configured settlement IBAN when neither names anything. Without it a broker
|
||||
// transfer cannot pair with the bank debit that funded it.
|
||||
func TestTradeRepublicResolvesTransferCounterparties(t *testing.T) {
|
||||
result := readTradeRepublic(t, tradeRepublicRows[0], tradeRepublicRows[9], tradeRepublicRows[12])
|
||||
want := []string{
|
||||
"DE89370400440532013000", // neither column nor description: the account's own settlement IBAN
|
||||
"DE04100110012623927730", // named in the description only
|
||||
"DE41110101002098897347", // the column
|
||||
}
|
||||
for i, f := range result.Facts {
|
||||
if f.CounterpartyIBAN != want[i] {
|
||||
t.Errorf("record %d settled against %q, want %q", i+1, f.CounterpartyIBAN, want[i])
|
||||
}
|
||||
}
|
||||
if result.Facts[2].Counterparty != "LARS NOLDEN" {
|
||||
t.Errorf("counterparty name lost: %q", result.Facts[2].Counterparty)
|
||||
}
|
||||
}
|
||||
|
||||
// A notional that does not land on a whole cent is the normal case here, not an
|
||||
// error: the export states cash to the cent while the product runs longer.
|
||||
func TestTradeRepublicChecksGrossToTheStatedPrecision(t *testing.T) {
|
||||
result := readTradeRepublic(t, tradeRepublicRows[8])
|
||||
inv := result.Facts[0].Investment
|
||||
// 0.426581 x 63.06 = 26.90019786, stated as 26.90.
|
||||
if inv.Gross != "26.90" || inv.Quantity != "-0.426581" || inv.Price != "63.06" {
|
||||
t.Fatalf("trade read as %+v", inv)
|
||||
}
|
||||
// A factor of ten is still caught: the tolerance is one cent, not one order.
|
||||
broken := strings.Replace(tradeRepublicRows[8], ";26.90;", ";269.00;", 1)
|
||||
file, err := ReadCSV(strings.NewReader(tradeRepublicHeader + broken + "\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := ParseTradeRepublicCSV(file, brokerAccount(), nil); err == nil {
|
||||
t.Fatal("accepted a gross ten times its own shares times price")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTradeRepublicRejectsRowsItCannotAccountFor(t *testing.T) {
|
||||
const base = `2026-01-05T10:00:00Z;2026-01-05;DEFAULT;CASH;TRANSFER_INBOUND;;;;;;12.00;;;EUR;;;;Incoming transfer;R1;;;;`
|
||||
for name, row := range map[string]string{
|
||||
"unknown type": strings.Replace(base, "TRANSFER_INBOUND", "VORABPAUSCHALE", 1),
|
||||
"category mismatch": strings.Replace(base, "CASH;TRANSFER_INBOUND", "TRADING;TRANSFER_INBOUND", 1),
|
||||
"foreign currency": strings.Replace(base, ";EUR;", ";USD;", 1),
|
||||
"other account type": strings.Replace(base, ";DEFAULT;", ";SAVINGS;", 1),
|
||||
"trade without a security": `2026-01-05T10:00:00Z;2026-01-05;DEFAULT;TRADING;BUY;STOCK;Mystery;;1.0;2.00;-2.00;;;EUR;;;;Buy trade of something;R2;;;;`,
|
||||
"unresolvable ticker": `2026-01-05T10:00:00Z;2026-01-05;DEFAULT;TRADING;BUY;CRYPTO;Bitcoin;BTC;1.0;2.00;-2.00;;;EUR;;;;Kauf Bitcoin;R3;;;;`,
|
||||
} {
|
||||
file, err := ReadCSV(strings.NewReader(tradeRepublicHeader + row + "\n"))
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
if _, err := ParseTradeRepublicCSV(file, brokerAccount(), nil); err == nil {
|
||||
t.Errorf("%s: accepted a row that can move money it should not", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Both formats are recognized from the same upload path, and neither is
|
||||
// mistaken for the other.
|
||||
func TestBrokerDetectionDistinguishesTheTwoExports(t *testing.T) {
|
||||
for _, format := range []struct {
|
||||
name, header, row, want string
|
||||
}{
|
||||
{"trade republic", tradeRepublicHeader, tradeRepublicRows[0], SourceTradeRepublic},
|
||||
{"scalable", scalableHeader, scalableRows[0], SourceScalable},
|
||||
} {
|
||||
file, err := ReadCSV(strings.NewReader(format.header + format.row + "\n"))
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", format.name, err)
|
||||
}
|
||||
source, label, header, ok := DetectBrokerCSV(file)
|
||||
if !ok || source != format.want || header != 1 || label == "" {
|
||||
t.Fatalf("%s detected as %q/%q at row %d (ok=%v)", format.name, source, label, header, ok)
|
||||
}
|
||||
if _, err := ParseBrokerCSV(file, brokerAccount(), nil); err != nil {
|
||||
t.Errorf("%s: %v", format.name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
+62
-17
@@ -246,6 +246,9 @@ func validText(values ...string) bool {
|
||||
}
|
||||
return true
|
||||
}
|
||||
func validHint(s string) bool {
|
||||
return utf8.ValidString(s) && utf8.RuneCountInString(s) <= 200
|
||||
}
|
||||
|
||||
// ValidISIN reports a syntactically valid ISIN: two country letters, nine
|
||||
// alphanumerics and a check digit.
|
||||
@@ -289,8 +292,8 @@ func Validate(d Dataset) error {
|
||||
if err := register(c.ID, "category"); err != nil {
|
||||
return err
|
||||
}
|
||||
if !nonblank(c.Name) || (c.Kind != "expense" && c.Kind != "income") {
|
||||
return fmt.Errorf("category %q: invalid name or kind", c.ID)
|
||||
if !nonblank(c.Name) || !validHint(c.Hint) || (c.Kind != "expense" && c.Kind != "income") {
|
||||
return fmt.Errorf("category %q: invalid name, hint or kind", c.ID)
|
||||
}
|
||||
categories[c.ID] = c
|
||||
if c.ParentID != "" {
|
||||
@@ -327,8 +330,8 @@ func Validate(d Dataset) error {
|
||||
if err := register(t.ID, "tag"); err != nil {
|
||||
return err
|
||||
}
|
||||
if !nonblank(t.Name) {
|
||||
return fmt.Errorf("tag %q: name required", t.ID)
|
||||
if !nonblank(t.Name) || !validHint(t.Hint) {
|
||||
return fmt.Errorf("tag %q: name or hint invalid", t.ID)
|
||||
}
|
||||
tags[t.ID] = true
|
||||
}
|
||||
@@ -463,9 +466,12 @@ func (index enrichmentIndex) validate(f Facts, e Enrichment) error {
|
||||
if e.MerchantID != "" && !index.merchants[e.MerchantID] {
|
||||
return fmt.Errorf("unknown merchant %q", e.MerchantID)
|
||||
}
|
||||
if !validText(e.Classification.Source, e.Classification.Model, e.Classification.Error) {
|
||||
if !validText(e.Classification.Source, e.Classification.Model, e.Classification.Confidence, e.Classification.Error) {
|
||||
return fmt.Errorf("classification metadata must be valid UTF-8")
|
||||
}
|
||||
if e.Classification.Confidence != "" && e.Classification.Confidence != "high" && e.Classification.Confidence != "medium" && e.Classification.Confidence != "low" {
|
||||
return fmt.Errorf("invalid classification confidence")
|
||||
}
|
||||
if e.Classification.Timestamp != "" {
|
||||
if _, err := time.Parse(time.RFC3339Nano, e.Classification.Timestamp); err != nil {
|
||||
return fmt.Errorf("invalid classification timestamp")
|
||||
@@ -538,18 +544,18 @@ func optionalQuantity(q Quantity) (int64, error) {
|
||||
return q.Units()
|
||||
}
|
||||
|
||||
// RoundedProduct multiplies an exact share count by an exact price and rounds
|
||||
// to money's four places, half away from zero. Quantity is 1e-8 units and
|
||||
// price is 1e-4 units, so the product is 1e-12 and needs 128-bit width.
|
||||
// RoundedProduct multiplies an exact share count by an exact unit price and
|
||||
// rounds to money's four places, half away from zero. Both operands are 1e-8
|
||||
// units, so the product is 1e-16 and needs 128-bit width.
|
||||
func RoundedProduct(quantity, price int64) (int64, bool) {
|
||||
product := new(big.Int).Mul(big.NewInt(quantity), big.NewInt(price))
|
||||
half := big.NewInt(50_000_000)
|
||||
half := big.NewInt(500_000_000_000)
|
||||
if product.Sign() < 0 {
|
||||
product.Sub(product, half)
|
||||
} else {
|
||||
product.Add(product, half)
|
||||
}
|
||||
rounded := product.Quo(product, big.NewInt(100_000_000))
|
||||
rounded := product.Quo(product, big.NewInt(1_000_000_000_000))
|
||||
if !rounded.IsInt64() {
|
||||
return 0, false
|
||||
}
|
||||
@@ -592,7 +598,7 @@ func validateInvestment(f Facts, a Account, instruments map[string]Instrument) e
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
price, err := optionalMoney(inv.Price)
|
||||
price, err := optionalQuantity(inv.Price)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -613,10 +619,18 @@ func validateInvestment(f Facts, a Account, instruments map[string]Instrument) e
|
||||
return err
|
||||
}
|
||||
if inv.CashOnly() {
|
||||
if quantity != 0 || inv.Price != "" || inv.Gross != "" {
|
||||
return fmt.Errorf("%s moves cash only: it carries no quantity, price or gross", inv.Event)
|
||||
if quantity != 0 || inv.Price != "" {
|
||||
return fmt.Errorf("%s moves cash only: it carries no quantity or price", inv.Event)
|
||||
}
|
||||
return nil
|
||||
// The gross is optional here. One broker states a cash row already net
|
||||
// of the tax it withheld, and then only the net is knowable, so the
|
||||
// tax is recorded and never applied. Another states the gross and the
|
||||
// deductions separately, and then the settlement is checkable like any
|
||||
// trade's. Which one is a fact about the source, decided at import.
|
||||
if inv.Gross == "" {
|
||||
return nil
|
||||
}
|
||||
return settles(inv, gross, fee, tax, amount)
|
||||
}
|
||||
if inv.InstrumentID == "" {
|
||||
return fmt.Errorf("%s requires an instrument", inv.Event)
|
||||
@@ -633,8 +647,19 @@ func validateInvestment(f Facts, a Account, instruments map[string]Instrument) e
|
||||
if inv.Settling() {
|
||||
expected = -expected
|
||||
}
|
||||
if gross != expected {
|
||||
return fmt.Errorf("%s gross %s does not equal quantity %s times price %s", inv.Event, Money(formatScaled(gross, moneyScale, 2)), inv.Quantity.String(), inv.Price.String())
|
||||
// The gross is checked to the precision the broker stated it at, and no
|
||||
// further. One broker prints the exact product to nine places, and the
|
||||
// check is then exact. Another prints the notional rounded to cents, where
|
||||
// demanding exactness rejects every trade whose product does not land on a
|
||||
// whole cent - measured on a real export, 29 of 59 of them. One unit of
|
||||
// the stated precision is still four orders of magnitude tighter than the
|
||||
// misplaced decimal separator this check exists to catch.
|
||||
difference := expected - gross
|
||||
if difference < 0 {
|
||||
difference = -difference
|
||||
}
|
||||
if difference >= statedUnit(inv.Gross) {
|
||||
return fmt.Errorf("%s gross %s does not equal quantity %s times price %s, which is %s", inv.Event, inv.Gross.String(), inv.Quantity.String(), inv.Price.String(), Money(formatScaled(expected, moneyScale, 2)))
|
||||
}
|
||||
if inv.PositionOnly() {
|
||||
if amount != 0 {
|
||||
@@ -648,10 +673,30 @@ func validateInvestment(f Facts, a Account, instruments map[string]Instrument) e
|
||||
if (inv.Event == EventSell) != (quantity < 0) {
|
||||
return fmt.Errorf("%s must %s the position", inv.Event, map[bool]string{true: "reduce", false: "increase"}[inv.Event == EventSell])
|
||||
}
|
||||
return settles(inv, gross, fee, tax, amount)
|
||||
}
|
||||
|
||||
// settles enforces that the cash a fact moved is its gross less the fee and
|
||||
// the tax deducted from it. Fee and tax are stored as deductions whichever sign
|
||||
// the source printed, so a refunded tax is a negative deduction and a broker
|
||||
// that writes its fee as a negative adjustment is normalized at import.
|
||||
func settles(inv *Investment, gross, fee, tax, amount int64) error {
|
||||
settled := new(big.Int).Sub(big.NewInt(gross), big.NewInt(fee))
|
||||
settled.Sub(settled, big.NewInt(tax))
|
||||
if !settled.IsInt64() || settled.Int64() != amount {
|
||||
return fmt.Errorf("%s cash %s does not equal gross %s minus fee %s minus tax %s", inv.Event, f.Amount.String(), inv.Gross.String(), inv.Fee.String(), inv.Tax.String())
|
||||
return fmt.Errorf("%s cash %s does not equal gross %s minus fee %s minus tax %s", inv.Event, Money(formatScaled(amount, moneyScale, 2)), inv.Gross.String(), inv.Fee.String(), inv.Tax.String())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// statedUnit is one unit of the last decimal place a money figure was written
|
||||
// with, in exact ten-thousandths. Money always renders at least two places, so
|
||||
// a whole-euro figure counts as stated to the cent.
|
||||
func statedUnit(m Money) int64 {
|
||||
_, fraction, _ := strings.Cut(string(m), ".")
|
||||
unit := int64(1)
|
||||
for range moneyScale - len(strings.TrimRight(fraction, "0")) {
|
||||
unit *= 10
|
||||
}
|
||||
return unit
|
||||
}
|
||||
|
||||
@@ -3,9 +3,10 @@ package domain
|
||||
// Money is an exact decimal string bounded to signed 64-bit ten-thousandths.
|
||||
type Money string
|
||||
|
||||
// Quantity is an exact decimal string bounded to signed 64-bit hundred-millionths.
|
||||
// Broker share counts are fractional: savings plans and reinvested distributions
|
||||
// settle in eight decimal places, which Money cannot represent.
|
||||
// Quantity is an exact decimal string bounded to signed 64-bit
|
||||
// hundred-millionths. It carries both share counts and unit prices, because
|
||||
// both exceed money's four places: a reinvested distribution settles a fraction
|
||||
// of a share, and a crypto unit price is quoted to six.
|
||||
type Quantity string
|
||||
|
||||
// Account kinds. An empty kind is a cash account: the field was added after the
|
||||
@@ -39,10 +40,14 @@ func (a Account) Investing() bool { return a.Kind == AccountInvestment }
|
||||
// both money and position; corporate actions and position transfers move
|
||||
// position only and must never touch cash.
|
||||
const (
|
||||
EventDeposit = "deposit"
|
||||
EventWithdrawal = "withdrawal"
|
||||
EventFee = "fee"
|
||||
EventInterest = "interest"
|
||||
EventDeposit = "deposit"
|
||||
EventWithdrawal = "withdrawal"
|
||||
EventFee = "fee"
|
||||
EventInterest = "interest"
|
||||
// EventTaxSettlement is a broker settling withheld tax in cash, in either
|
||||
// direction: a loss-offset pot returning tax already paid, or a
|
||||
// recalculation charging more.
|
||||
EventTaxSettlement = "tax_settlement"
|
||||
EventDistribution = "distribution"
|
||||
EventBuy = "buy"
|
||||
EventSell = "sell"
|
||||
@@ -62,7 +67,7 @@ type Investment struct {
|
||||
Event string `json:"event"`
|
||||
InstrumentID string `json:"instrument_id,omitempty"`
|
||||
Quantity Quantity `json:"quantity,omitempty"`
|
||||
Price Money `json:"price,omitempty"`
|
||||
Price Quantity `json:"price,omitempty"`
|
||||
Gross Money `json:"gross,omitempty"`
|
||||
Fee Money `json:"fee,omitempty"`
|
||||
Tax Money `json:"tax,omitempty"`
|
||||
@@ -71,7 +76,7 @@ type Investment struct {
|
||||
// CashOnly reports an event that moves money without moving a position.
|
||||
func (i Investment) CashOnly() bool {
|
||||
switch i.Event {
|
||||
case EventDeposit, EventWithdrawal, EventFee, EventInterest, EventDistribution:
|
||||
case EventDeposit, EventWithdrawal, EventFee, EventInterest, EventTaxSettlement, EventDistribution:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
|
||||
Reference in New Issue
Block a user