System integration
simplification is a HOAX

In the realm of system integration and software development in general, simplification has been a reoccurring subject. But unfortunately there is no such thing as a “Magic Wand”. The last 10 years we have been talking a about rest API’s to replace the Enterprise Service Bus, noSQL databases to replace Relational Databases and REST and GraphQL to replace SOAP. All in the name of simplification. When Facebook, LinkedIn and Twitter come up with a great solutions for their problems, the tendency is to follow this even if your use case is totally different. The tools created are superior solutions for the problems they face, the question is are they also superior for your use case?. I personally think there is a place for all techniques mentioned, the success of the chosen technique depends on the problem to solve. This has nothing to do with simplification but on choosing a solution compatible with your problem!

Complexity that is removed from one place tends to pop-up somewhere else!

Changing the rules

“Simplification” is possible but there is always a tradeoff, you will change the rules and understand and accept the consequences. In general, complexity that is removed from one place tends to pop-up somewhere else! Ill-considered simplification plans often leads to unstructured workarounds in your and adjacent systems.

Why should I use an integration layer as I can easily connect to the applications internal API?

Service Oriented Architecture

Service Oriented Architecture has been around for over 20 years and is one of the most successful integration architectures. It results in a simplification of your landscape but there is added complexity. Creating reusable services takes more thinking ahead; Changing the interface will impact multiple consumers; Load is centralized to one generic service; and so on. To implement Service Oriented Architecture you could either use an Enterprise Service Bus or HTTP based (API Gateway). Moving from ESB to API is often promoted as a simplification and in some cases this could be true, but mostly it is a change of flavor. The reason REST API’s are pushed is because portals and apps are using a REST API’s backend. The backend applications more and more support this. What I personally find worrying is that the integration layer is removed in the name of  simplification. Why should I use an integration layer as I can just connect to the applications internal API?

Flexibility over optimization

Integration components versus application provided API’s

Connecting your consumers directly to the API provided by your application sounds great, one less component to manage. From a Service Oriented Architecture this should raise red flags. The integration layer is there for a lot of reasons, it is the flexible part in your application chain. Decoupling applications from your “business” is an enabler for migration; it translates application language and protocol to your enterprise language and protocol to make your services reusable for the whole organization; Organisations using the Frank framework it will miss the “a peephole” in your black box application chain, making it difficult to manage it. The trade-off for this “simplification” is you lost flexibility and it will take us back 20 years in time. SOA Manifesto states: “Flexibility over optimization”.

Relational versus NoSQL database

Another good example of a politicalized discussion is the move from relational databases to NoSQL databases. The same story here, you will need to accept the drawbacks. No-SQL* can be a perfect fit for Facebook or Twitter use cases but in many cases, relational databases are still the best option. So what are the main differences? The Relational way is to design your database around your data and structure your database accordingly. You limit redundancy and guarantee data consistency. In a NoSQL approach, you will design your database around the need of your application, Therefore you will need to focus on what queries your application is going to use before you model your database. Furthermore, you will be responsible for the data consistency by writing backend scripts. NoSQL database are optimized in scaling and maintaining performance with large dataset. Opposed to Relational databases are good at keeping data consistent and prevent data redundancy.

* NoSQL databases come in many flavors from key-value stores to document stores.

What capabilities does a queue bring to your integration landscape?

Queueing versus REST http

The latest thing within system integration is changing from a queue based integration to a http (API) based landscape. It seems like a quick win and it will simplify your landscape and save on maintenance costs. But what capabilities does a queue bring to your integration landscape?

Persistency / Reliability / Message order

On a queue we can persist a message (store it) this way we can create “exactly once” delivery and in a single-threaded situation it could even keep message order for you. If you want to create exactly one delivery without a queue you will need to store your message at the service consumer side for potential retrying and to make sure you do not get duplicates in your backend system you will need to implement a reliability pattern. The most obvious being a duplicate check within your service or backend application.

Load balancing, Throttling

If a service provider or consumer on a queue has time to handle a message it will connect to a queue and ask for the next message. This is an implicit load balancing and throttling mechanism. To create the same functionality in an “HTTP world” you will need a separate component to handle throttling and load balancing. Next to this, your application needs to provide some kind of health check endpoint so your load balancer can defer request when a consumer node is down.

Authorization

You can authorize an application to write to or read from a queue. In a HTTP situation this functionality will be moved to an API gateway or to the service provider.

This example shows us that simplification is never simple. You need in-depth knowledge about the capabilities provided and used in your landscape before you even start thinking about replacement.

Exactly once pattern

Simplification is about how you handle your complexity while maintaining your flexibility

Approach to “simplification” in system integration

Simplification starts by implementing an architecture and tooling that will help you manage the complexity of your landscape. Furthermore  simplification is all about making choices and understanding and accepting the consequences. Simplification is about how you handle your complexity while maintaining your flexibility.

First, determine the key capabilities and the amount of flexibility you will need to support your business (processes). The price for flexibility is called complexity. Therefore implement as much standardisation as possible, standardization will lead to shorter development time and lower cost of ownership.

Second, model your services around your business needs instead of the available application (interfaces) and most importantly determine your long-term goals/architecture.

Third, take an agile step by step approach, start with a migration plan. One of the biggest mistake would be trying to move to your end state in one big step. The step-by-step approach will allow you to create workarounds an remove potential technical debt and slowly migrate to “perfection”.  While going from current to target state make sure to pick the low-hanging fruit! Optimize and clean your landscape, I am sure you have buildup a lot of technical debt in the last 5 years. Remove duplicate functionality, old migration components, push your service consumers to a single version of your services and most importantly standardize your components.

Realize today’s brand new solution is tomorrow’s legacy system, make sure your components are loosely coupled and are prepared for the next migration

Jeroen Jansen
van Rosendaal
Founder & Architect

Get in contact for a free consultation!