Import Trade Republic exports, whose conventions invert Scalable's
A second broker export is recognized locally, by its full column set, and read through the same pipeline: detection and parsing now dispatch on the format, so the upload path, the review dialog, deduplication, the journal and the Wealth report are unchanged. Its nine row types cover cash transfers, interest, dividends, tax settlements and trades in funds, shares and crypto; none of them moves a position without moving cash, so the cash-neutral class that Scalable's corporate actions belong to does not arise here. Three of its conventions are the opposite of the export already supported, and reading any of them the other way round moves money. Fee and tax are the signed adjustments it made to the cash rather than deductions from a gross, so a one euro order fee arrives as -1.00 and is negated at import; the journal keeps one convention and the domain never learns that two exist. A cash row's amount is the gross, not the net, so interest of 16.46 with -4.33 of tax credits 12.13 - where the other export states its cash already net and its tax is recorded and never applied. Whether a cash row carries a gross now decides which of those it was, which also makes the first kind's settlement checkable and stops the Wealth report from claiming a figure was left unapplied when it was not. And a TAX_OPTIMIZATION row puts zero in the amount column and its money in the tax column, signed both ways: read as cash, all six in a real export move nothing. Two more rows lie about their own columns. A dividend fills the share column with the holding the dividend was paid on, not with a position change, so adding it would double the holding. Crypto carries a bare ticker in the symbol column and its ISIN-shaped identifier only in the description, so the identifier is taken from the symbol when that is an ISIN and otherwise from the one the description names; a position row resolving to neither is refused rather than attached to a guess. The shares-times-price check now holds a gross to the precision the export stated it at rather than to four places. This export prints the notional rounded to cents, and 29 of 59 real trades do not land on a whole cent: demanding exactness rejected half a portfolio. One unit of the stated precision is still four orders of magnitude tighter than the misplaced separator the check exists to catch, and where an export prints the full product the check stays exact. A unit price moves from money to the eight-place quantity type, because a crypto price is quoted to six and rounding it would break the check the amount is verified against. Trailing zeros are dropped before any precision test: this export pads a six-place price to ten, and the padding would otherwise exhaust the precision the value needs. A transfer's counterparty comes from the export's own IBAN column when it has one, from the IBAN the description names in parentheses when it does not, and from the account's configured settlement IBAN when neither names anything. Free text contributes only a value shaped like an IBAN. Without this, 108 transfers stay unpaired and their bank-side counterparts read as spending and income. Verified end to end against a real export: 26 rows import to a cash balance of 32187.02 matching the figure computed by hand from the source rows, all four positions close at exactly zero, and every trade satisfies its own arithmetic.
This commit is contained in:
Lars Nolden
@@ -0,0 +1,277 @@
|
||||
package banking
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/big"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"finance-duck/internal/domain"
|
||||
)
|
||||
|
||||
// SourceTradeRepublic identifies facts imported from a Trade Republic export.
|
||||
const SourceTradeRepublic = "traderepublic_csv"
|
||||
|
||||
// tradeRepublicColumns are the exact normalized headers of a Trade Republic
|
||||
// transaction export.
|
||||
var tradeRepublicColumns = []string{
|
||||
"datetime", "date", "account_type", "category", "type", "asset_class",
|
||||
"name", "symbol", "shares", "price", "amount", "fee", "tax", "currency",
|
||||
"original_amount", "original_currency", "fx_rate", "description",
|
||||
"transaction_id", "counterparty_name", "counterparty_iban", "payment_reference", "mcc_code",
|
||||
}
|
||||
|
||||
// tradeRepublicEvents maps the export's complete type vocabulary to journal
|
||||
// events. The set is closed on purpose: an unrecognized type could move cash in
|
||||
// either direction, or none, and defaulting it risks a silent balance error.
|
||||
var tradeRepublicEvents = map[string]string{
|
||||
"TRANSFER_INBOUND": domain.EventDeposit,
|
||||
"TRANSFER_INSTANT_INBOUND": domain.EventDeposit,
|
||||
"TRANSFER_OUTBOUND": domain.EventWithdrawal,
|
||||
"TRANSFER_INSTANT_OUTBOUND": domain.EventWithdrawal,
|
||||
"INTEREST_PAYMENT": domain.EventInterest,
|
||||
"DIVIDEND": domain.EventDistribution,
|
||||
"TAX_OPTIMIZATION": domain.EventTaxSettlement,
|
||||
"BUY": domain.EventBuy,
|
||||
"SELL": domain.EventSell,
|
||||
}
|
||||
|
||||
// isinInText finds the security identifier a row names in its free text. Trade
|
||||
// Republic puts an ISIN in the symbol column for funds and shares, but a bare
|
||||
// ticker for crypto, whose ISIN-shaped identifier appears only in the
|
||||
// description: "Sell trade XF000DOGE012 Dogecoin".
|
||||
var isinInText = regexp.MustCompile(`\b[A-Z]{2}[A-Z0-9]{9}[0-9]\b`)
|
||||
|
||||
// ibanInText finds the counterparty a transfer names in its free text. Older
|
||||
// rows leave the counterparty_iban column empty and write the IBAN in
|
||||
// parentheses instead: "Outgoing transfer for LARS NOLDEN (DE04...)".
|
||||
var ibanInText = regexp.MustCompile(`\(([A-Z]{2}[0-9]{2}[A-Z0-9]{10,30})\)`)
|
||||
|
||||
// ParseTradeRepublicCSV converts a Trade Republic export into bank facts
|
||||
// carrying position legs.
|
||||
//
|
||||
// Three conventions differ from every other export this program reads, and each
|
||||
// one moves money if it is read the other way round:
|
||||
//
|
||||
// - fee and tax are signed adjustments to cash, not deductions from a gross.
|
||||
// The export writes a one euro order fee as -1.00 and withheld tax as
|
||||
// -4.33, so both are negated at import and the journal keeps one
|
||||
// convention: cash is gross minus fee minus tax.
|
||||
// - a cash row's amount is the gross, not the net. Interest of 16.46 with
|
||||
// -4.33 of tax credits 12.13. This is the opposite of an export that
|
||||
// states its cash already net, where the tax is recorded and never
|
||||
// applied.
|
||||
// - a TAX_OPTIMIZATION row carries zero in the amount column and the money
|
||||
// in the tax column, signed both ways. Read as cash, all six of them move
|
||||
// nothing; read correctly, they are the loss-offset pot settling.
|
||||
//
|
||||
// A dividend row populates the share column with the holding the dividend was
|
||||
// paid on, not with a position change. Adding it would double the holding, so
|
||||
// it is read as the attribution it is and discarded.
|
||||
//
|
||||
// The amount on a trade is the notional rounded to cents, not the exact
|
||||
// product, so the shares-times-price check is satisfied to the precision the
|
||||
// broker stated rather than exactly. Of 59 trades in a real export, 30 are
|
||||
// exact at four places and all 59 are within a cent.
|
||||
//
|
||||
// The booking date is the date column exactly as printed. The datetime column
|
||||
// is UTC while the date column is local, so they disagree for rows booked late
|
||||
// in the evening and deriving the date from the timestamp would move them to
|
||||
// the previous day.
|
||||
func ParseTradeRepublicCSV(f CSVFile, account domain.Account, registry []domain.Instrument) (BrokerImport, error) {
|
||||
result := newBrokerImport()
|
||||
if err := investmentTarget(account); err != nil {
|
||||
return result, err
|
||||
}
|
||||
header, ok := DetectTradeRepublicCSV(f)
|
||||
if !ok {
|
||||
return result, errors.New("not a Trade Republic export")
|
||||
}
|
||||
headers, cell := brokerColumnIndex(f, header)
|
||||
|
||||
instruments := map[string]domain.Instrument{}
|
||||
byISIN := map[string]domain.Instrument{}
|
||||
for _, v := range registry {
|
||||
instruments[v.ID] = v
|
||||
byISIN[v.ISIN] = v
|
||||
}
|
||||
created := map[string]int{}
|
||||
named := map[string]string{}
|
||||
drift := new(big.Int)
|
||||
for offset, row := range f.rows[header:] {
|
||||
record := header + offset + 1
|
||||
if blankCSVRow(row) {
|
||||
continue
|
||||
}
|
||||
if len(row) != len(headers) {
|
||||
return result, fmt.Errorf("broker record %d has %d columns, expected %d", record, len(row), len(headers))
|
||||
}
|
||||
// One export covers one account. A second account type in the same file
|
||||
// would silently merge two cash balances into one.
|
||||
if kind := cell(row, "account_type"); !strings.EqualFold(kind, "DEFAULT") {
|
||||
return result, fmt.Errorf("broker record %d belongs to account type %q, and only DEFAULT can be imported into one account", record, kind)
|
||||
}
|
||||
rawType, category := cell(row, "type"), cell(row, "category")
|
||||
event, known := tradeRepublicEvents[strings.ToUpper(strings.TrimSpace(rawType))]
|
||||
if !known {
|
||||
return result, fmt.Errorf("broker record %d has unknown type %q: it may or may not move cash, so nothing was imported", record, rawType)
|
||||
}
|
||||
investment := domain.Investment{Event: event}
|
||||
wanted := "TRADING"
|
||||
if investment.CashOnly() {
|
||||
wanted = "CASH"
|
||||
}
|
||||
if !strings.EqualFold(category, wanted) {
|
||||
return result, fmt.Errorf("broker record %d pairs type %q with category %q, expected %q", record, rawType, category, wanted)
|
||||
}
|
||||
currency := strings.ToUpper(cell(row, "currency"))
|
||||
if currency != strings.ToUpper(account.Currency) {
|
||||
return result, fmt.Errorf("broker record %d settles in %q but account %q holds %s: currency conversion is not supported", record, currency, account.DisplayName, account.Currency)
|
||||
}
|
||||
booking, err := parseMappedCSVDate(cell(row, "date"), "yyyy-mm-dd")
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid date %q", record, cell(row, "date"))
|
||||
}
|
||||
description := cell(row, "description")
|
||||
isin, err := tradeRepublicISIN(cell(row, "symbol"), description, !investment.CashOnly())
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
if isin != "" {
|
||||
held, exists := byISIN[isin]
|
||||
if !exists {
|
||||
name := cell(row, "name")
|
||||
if name == "" {
|
||||
name = isin
|
||||
}
|
||||
held = domain.Instrument{ID: domain.InstrumentID(isin), ISIN: isin, Name: name, Currency: currency}
|
||||
byISIN[isin] = held
|
||||
instruments[held.ID] = held
|
||||
created[isin] = len(result.Instruments)
|
||||
result.Instruments = append(result.Instruments, held)
|
||||
}
|
||||
investment.InstrumentID = held.ID
|
||||
slot, mine := created[isin]
|
||||
if name := cell(row, "name"); mine && name != "" && name != isin && booking >= named[isin] {
|
||||
named[isin] = booking
|
||||
result.Instruments[slot].Name = name
|
||||
}
|
||||
}
|
||||
gross, grossDrift, err := brokerMoney(cell(row, "amount"), decimalPlain)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid amount %q: %w", record, cell(row, "amount"), err)
|
||||
}
|
||||
fee, feeDrift, err := brokerMoney(cell(row, "fee"), decimalPlain)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid fee %q: %w", record, cell(row, "fee"), err)
|
||||
}
|
||||
tax, taxDrift, err := brokerMoney(cell(row, "tax"), decimalPlain)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid tax %q: %w", record, cell(row, "tax"), err)
|
||||
}
|
||||
if grossDrift.Sign() != 0 || feeDrift.Sign() != 0 || taxDrift.Sign() != 0 {
|
||||
result.Rounded++
|
||||
drift.Add(drift, grossDrift).Add(drift, feeDrift).Add(drift, taxDrift)
|
||||
}
|
||||
// The export states what it took off the cash; the journal stores what
|
||||
// was deducted from the gross.
|
||||
if fee, err = negated(fee); err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
if tax, err = negated(tax); err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
if investment.CashOnly() {
|
||||
// The share column on a dividend is the holding it was paid on.
|
||||
investment.Gross, investment.Fee, investment.Tax = gross, fee, tax
|
||||
} else {
|
||||
if isin == "" {
|
||||
return result, fmt.Errorf("broker record %d moves a position without a security identifier", record)
|
||||
}
|
||||
shares, err := brokerQuantity(cell(row, "shares"), decimalPlain)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid share count %q: %w", record, cell(row, "shares"), err)
|
||||
}
|
||||
price, err := brokerQuantity(cell(row, "price"), decimalPlain)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d has an invalid price %q: %w", record, cell(row, "price"), err)
|
||||
}
|
||||
investment.Quantity, investment.Price, investment.Gross = shares, price, gross
|
||||
investment.Fee, investment.Tax = fee, tax
|
||||
}
|
||||
cash, err := brokerSettlement(gross, fee, tax)
|
||||
if err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
facts := domain.Facts{
|
||||
Source: SourceTradeRepublic, AccountID: account.ID, BookingDate: booking,
|
||||
Amount: cash, Currency: currency, RawDescription: description,
|
||||
ExternalID: cell(row, "transaction_id"), Counterparty: cell(row, "counterparty_name"),
|
||||
Investment: &investment,
|
||||
}
|
||||
if investment.Event == domain.EventDeposit || investment.Event == domain.EventWithdrawal {
|
||||
facts.CounterpartyIBAN = tradeRepublicIBAN(cell(row, "counterparty_iban"), description, account.ReferenceIBAN)
|
||||
}
|
||||
if err := domain.ValidateInvestment(facts, account, instruments); err != nil {
|
||||
return result, fmt.Errorf("broker record %d: %w", record, err)
|
||||
}
|
||||
result.Facts = append(result.Facts, facts)
|
||||
}
|
||||
if len(result.Facts) == 0 {
|
||||
return result, errors.New("broker export contains no records")
|
||||
}
|
||||
result.Rounding = decimalString(drift, residueScale)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// DetectTradeRepublicCSV reports whether a document is a Trade Republic export
|
||||
// and which 1-based record holds its header.
|
||||
func DetectTradeRepublicCSV(f CSVFile) (header int, ok bool) {
|
||||
return matchColumns(f, tradeRepublicColumns)
|
||||
}
|
||||
|
||||
// tradeRepublicISIN resolves the security a row names. The symbol column holds
|
||||
// an ISIN for funds and shares and a bare ticker for crypto, whose ISIN-shaped
|
||||
// identifier appears only in the description. Exactly one identifier must be
|
||||
// findable, or the row is refused rather than attached to a guess.
|
||||
func tradeRepublicISIN(symbol, description string, required bool) (string, error) {
|
||||
candidate := strings.ToUpper(strings.Join(strings.Fields(symbol), ""))
|
||||
if domain.ValidISIN(candidate) {
|
||||
return candidate, nil
|
||||
}
|
||||
found := isinInText.FindAllString(description, -1)
|
||||
unique := map[string]bool{}
|
||||
for _, match := range found {
|
||||
if domain.ValidISIN(match) {
|
||||
unique[match] = true
|
||||
}
|
||||
}
|
||||
if len(unique) == 1 {
|
||||
for match := range unique {
|
||||
return match, nil
|
||||
}
|
||||
}
|
||||
if !required {
|
||||
return "", nil
|
||||
}
|
||||
if candidate == "" {
|
||||
return "", errors.New("row moves a position but names no security")
|
||||
}
|
||||
return "", fmt.Errorf("symbol %q is not an ISIN and its description does not name exactly one", symbol)
|
||||
}
|
||||
|
||||
// tradeRepublicIBAN resolves the account a transfer settles against: the
|
||||
// export's own column when it has one, else the IBAN the description carries in
|
||||
// parentheses, else the account's configured settlement IBAN. Free text only
|
||||
// contributes a value that is shaped like an IBAN, so a description that names
|
||||
// no account contributes nothing.
|
||||
func tradeRepublicIBAN(column, description, fallback string) string {
|
||||
if iban := normalizeIBAN(column); iban != "" {
|
||||
return iban
|
||||
}
|
||||
if match := ibanInText.FindStringSubmatch(strings.ToUpper(description)); match != nil {
|
||||
return normalizeIBAN(match[1])
|
||||
}
|
||||
return normalizeIBAN(fallback)
|
||||
}
|
||||
Reference in New Issue
Block a user