Over-optimizing your software architecture: Common Blunders
We’ve all been there. Nobody is born knowing how many requests can postgres handle concurrently. We’ve all tried, optimized a little more than required, and recorded our observations in a small diary inside our head. I call it the “formula-diary”. Every time we face with a similar situation, without assessing the need, we tend to visit our formula diary and implement the solution we’re already comfortable with. It’s a common blunder, and it’s forgivable in most cases. The real cost comes when the size of your team grows.
In this blog, I’m going to share about the mistakes I made when building softwares, particularly related to eagerly optimizing my stack without thinking about the consequences or price. I’m honest and open about it now, but in my early days, I was a little defensive about my design, so I didn’t invite much feedback. With that said, not that my architecture sucked or burned an entire quarter of company profits, but it certainly could’ve been “cheaper”.
Blunder 1: The Redis Obsession
..Or any kind of cache, for that matter. Usually, we tend to tie the idea of “caching” with the “limitations” of our database . As ironical as it may sound, we trust a non-persistent, in-memory store more than a persistent layer of database. And in our heads, we’re completely under-confident about the performance of now commonly used NVME SSDs, which is by far the fastest SSDs ever to be created. This is what leads us to introduce cache very early in our software architecture. Be it Redis, Elasticache or memchaced, we layer it on top of our database queries to store and retrieve redundant queries fast. But at what cost ?
Always ask the following questions before introducing cache into your system:
- How many concurrent reads / writes is my database currently performing ?
- Can the problem be solved just by vertically scaling my database server ?
- Is there only one or two queries that are consuming a majority of my database resources ?
- Can I index my tables or create simple views that can eliminate the need for using a cache ?
This can help one in not eagerly moving to a cache based solution and saving hours by not modifying code, creating cache-invalidation strategies and figuring out the causes of stale data. And trust me when I say it, you will come across these problems when you introduce cache.
Blunder 2: Multi-Threading the life out of your backend application
When I learned first about High Performance Computing, I came across the term multithreading in a way in which it made sense. Multithreading is simply leveraging the multiple “cores” of your system for solving a complex problem that may otherwise take longer. For eg. If you want to compute sum of all the columns of a large 10,000 x 10,000 matrix, sure, go ahead and use multithreading to calculate columns in parallel. However, if all you need is to read and process a bunch of small files in a single API request, then introducing multithreading might be an overkill. It is not only hard to debug, but it makes your code difficult to understand and extend.
Here are the questions I’d ask myself before writing a multi-threading / batch processing utility:
- Is it taking minutes to calculate / process a request for me to think about multithreading in the first place ?
- Can I eagerly calculate the result or run a cron job to periodically calculate the results beforehand ?
- Is my system even capable of handling multi-threading ?
Blunder 3: Going completely cloud-native
Cloud is now seen as a silver bullet for every architectural problem in software development. Be it lambdas or a scale-as-you-go database, people are jumping on cloud without really checking if they even need it in the first place.
Sure, cloud offers things like ease of use, pay-as-you-go pricing model, on-demand scalability & reliable disaster-recovery capabilities. However, how much of that does your business really need. I’m not against using virtual machines on cloud, since everything is built on top of it. But I am against jumping on the ship of using cloud-native offerings at the first sight of trouble.
These are some of the questions to ask before using cloud-native features like serverless functions, message queues, cloud-native databases, cloud-native caches etc :
- Will it be cheaper considering the projected users of my application to move to cloud ?
- Is the increased resource cost of cloud justified against the effort reduction it will provide in terms of maintenance of my application infrastructure ?
- What does cloud do better that my team cannot do by themselves ?
Asking these questions are important, since they determine how sensibly you’ve planned your application architecture, saving maintenance cost and resources.
