Automated, scheduled collection of NAV prices and a decade of income-distribution history for a basket of Malaysian unit trusts, from four fund houses that publish no API.
12
funds tracked
4
provider sites
Hourly
collection cadence
2010–26
distribution history
The problem
A fund analyst needs a clean daily time series of NAV prices and a history of income
distributions across a dozen Malaysian unit trusts. None of the four fund houses involved
publishes an API. Each puts the numbers on a different website, in a different layout, behind a
different amount of JavaScript, and one of them renders search results into a floating overlay
that standard selectors never see. Collected by hand, this is an hour of copy-paste every day,
with transcription errors baked in.
What it is
A set of scheduled scrapers that drive a real browser through each provider's own site,
extract the authoritative figure, and append it to an Excel time series committed back to the
repository, so the repository itself is the database, with a commit trail behind every
data point. It runs hourly on GitHub Actions and emails a daily report.
How it works
What it captures
Beyond daily NAV, it builds a longitudinal record of every income distribution each fund
has declared, in some cases back more than a decade. Each stem below is one payout event the
scraper captured and dated.
Net distribution per unit, by ex-date, a sample of the collected distribution history across three funds (public data)
What it removed
Typing
Daily manual data entry across four provider sites is now unattended. The workbook updates itself and mails out each morning.
Gaps
A fund missing from the primary aggregator falls back to the fund house's own site; a per-fund status column makes a miss a known state rather than a silent hole.
Mismatch
For multi-currency funds the scraper selects the MYR share class explicitly and verifies the currency on the page before recording, rather than trusting search-result order.
Honest limitations
It depends on each provider's live DOM. When a site is redesigned the relevant strategy breaks and needs re-selecting, the unavoidable cost of scraping instead of an API.
NAVs are recorded as published; an aggregator and a fund house can legitimately show slightly different prices, and the tool does not reconcile them.
History only goes as far back as each site exposes; there is no backfill beyond that.
The published repository stores the collected public workbooks; it is a convenience cache, not an official data source.