Reading through the article I was thinking back to the era where we added Hadoop to our infrastructure, that caused such an overhead that it was 3 times slower than the previous, pragmatic single node solution 😂. But we learned from it.

Great article! If I may add: from my perspective it helped a lot to ask the Spice Girls questions to your stakeholders as often as possible (So tell me what you want, what you really, really want). This question delves into the underlying motivation, the WHY? — the ultimate problem the stakeholder would like to solve.

With a better understanding of the WHY, it is often easier for Data Engineers to come up with a much simpler solution than initially anticipated. Like: turns out the 1 second respond time for a report was just a random number, they are fine with everything faster than 1 minute 🧐.

Thanks for sharing such an important topic, +1 for simplicity 🙌.