Credit for the title of this post can be taken by dancinghands.com. I’m learning how to play the bongo, and rolling down the hill describes one of the many Latin rhythms. It also applies to data science.
In the quest for real-time processes, some unexpected help came from an enterprise architecture model. These days everything is agile, and architecture is seen as the work of charlatans. I do agree that architecture can go terribly wrong. At the same time though, not all quality stems from weekly iterations. The model I am referring to is depicted below. It gave me insight to what I saw happening.
Once you decide that real-time processes are important (also look at how real-time is real-time), requirements start to trickle down. It is likely that the information architecture needs adjusting in the direction of events and micro-services (more on this later). This in turn will impact the systems and component designs. Real-time systems require high uptime, so application management and infrastructure is likely to change to. This is in essence what I have seen happening in the last year. Rolling down the hill.
Summarizing, the following qualities are required in the layers:
- Business processes – real-time or as fast as needed (might be weekly)
- Information architecture – avoiding hotness of databases and schema changes, focus on integrating using key indexes like contract number, address, postal code etc.
- Application layer – systems are placed in highways of events, keep data streaming, state becomes localized, resilience, idem potency
- Infrastructure layer – Scalability and fault-tolerance, don’t let success become you future problem
I will leave details to your imagination.