Measure before you choose: why I start with boring technology
Data-heavy products attract exciting infrastructure. I usually start with the simplest setup that works, and I make that choice with numbers, so it is easy to explain and easy to revisit.
Fariz Wibisono2 min read
When a product handles a lot of data, it is tempting to reach for impressive tools from day one: a specialised analytics database, a cluster of servers, many small services. Each of those is a good tool. Each is also one more thing to run, back up, monitor and keep in sync with everything else.
On the rebuild of Invictus I chose the plain option on purpose, and I made each choice with a measurement next to it.
One database instead of two
The heaviest data is a record of every trade on every pool we follow. We tested one real hour of traffic in three setups: an ordinary Postgres database, the same database with a time-series extension (TimescaleDB) that compresses old data, and a specialised analytics database (ClickHouse).
The specialised database stored the data most compactly. The compressed Postgres came close: a month of data shrank from about 80 GB to about 22 GB. Since we needed Postgres anyway for users and app data, adding a second database would have meant a second system to operate for a modest saving. So we kept one.
One server, split only when needed
Every part of the system, the website, the API and the background workers, runs from the same package on one server, each with its own role. That gives one thing to build, test and roll back. One server is a single point of failure, so backups go to a different provider, and each next step, like a second server or a database copy, has a clear signal that would trigger it.
Don't pay to refresh what nobody is looking at
The biggest cost in a data product is often not storage. It is refreshing information that nobody is looking at. So data that someone is viewing right now updates constantly, data that matters for alerts updates in the background, and data that can never change is read once and kept. A part of the system nobody is using costs almost nothing.
Why this matters beyond one project
It also makes conversations easier. When someone asks why we did not use the exciting option, the answer is a number and a condition, not a matter of taste.