A few weeks ago, I was involved in a project at Airbnb. We needed to design a brand new feature that needed to handle up to 5,000 requests per second. This project, being new, has quite a few requirement uncertainties. However, one of the known requirements was being able to handle more load than most other features at Airbnb.
There was an internal discussion around how to implement this feature. There were two approaches: Approach 1 involves setting up new internal services, new load balancers, and bunch of new infrastructure code to be written. This could take 1-2 months for an engineer to implement, but it will lay a foundation for future projects having similar load requirements. There was also Approach 2, which involves a bunch of small hacks, quite a few limitations, writing code that was not very reusable or extensible beyond the current scenario, but it will only take 1 week to implement for an engineer.
What is the right approach?
Engineers normally go through two phrase of training. The first phrase was the formal education. At school, they are taught to write quick and clever algorithms to solve a discrete yet hard problems. This is why computer science grads typically interview better than seasoned engineers. However, writing good interview code does not equate to being a good engineer. This is why engineers typically go through a second phrase of training, where they learn on the job to write readable code, design scalable architecture, and follow the best practices. A good engineers learn two skills on the job: being a good communicator and being able to handle complex systems.
A seasoned engineers can typically think more holistically about a solution. They will ask the following questions: what if the load is increased by 10x? Can the system be understood by new hires? Can the design maximize extensibility and reusability?
While these are all good questions to ask, it is easy lose sight of the root of all these questions, which boils down to one single fact: the future is unknowable.
In fact, all the questions we ask to design a system is to address this problem. If we know the future, then there is no need to make a system reusable, handle 10x load increase, or even understandable. We can simply write the solution to all the problems there will ever be. Why does a system need to be extensible if there are no new scenarios to extend to?.
I follow this up with two corollaries:
- Products must be able to adapt to changes, when the requirements change
- Most requirement changes cannot be known until a product is built
The first point is easy to understandable. As engineers, we loathe requirement changes, as it adds more effort and lengthens the development time. As a result, engineers tend to build product in anticipation of product changes. The fact that products are extensible is often to future proof changes. The second point, however, is not often understood. Engineers often think of requirements as the specifications written down by the product manager. However, product managers can never know precisely what the users actually want. What they do is to approximate the user requirement in the forms hypotheses, in hopes that the product built by engineers actually capture user’s value proposition. Because these requirements are hypotheses, the more complex they are, the more likely it is going to have wrong assumptions, and the less likely it is going to be useful. Products that are built to big and complex requirements often fail because of that. This is why we want to build the simplest products possible, so we can prove that the requirements are actually correct before we go further. Otherwise, anything we build will lengthen the time it takes to correct the requirements and is wasteful.
The reason extensible products often follow the first corollary but runs against the second one is that efforts to make things future proof is usually unnecessary. If a product in its lifetime never becomes extensible or scalable, then this high quality code will need to be thrown away. Imagine that someone took one year to write a beautiful web site with all the right practices and industry standards only to discover the right product is an iPhone app. How much waste would this produce?
So, what happened to the project I was involved at Airbnb? We built a simple version, and we quickly found out that this version was not what the user wanted and we quickly abandoned it. The engineering effort was only about 1 week. Some engineers started a separate track to build a scalable solution for both this project as well as other use cases. However, all these use cases disappeared over time. After about two months, the development effort on that project also halted.