All about integration

Structured data and application integration is a challenge for many organizations. Companies often start by setting up one system, such as an ERP, for their most important processes. As the organization grows, the need for other applications also arises, for example for the HR or marketing department. 

All the different systems can run separately, but ultimately you want them to communicate with each other. Linking one or two systems is no problem, but when dozens or hundreds of applications have to talk to each other, it becomes a challenge.

When you keep connecting everything, at some point you will end up with a tangle of APIs and other integration components that connect systems and applications. That's a hell of a job to maintain – not to mention the decoupling of old applications. 

That's why we share all our best practices for data integration and application integration on this page, so that you can get started in a structured way. As a result, you should have an integrated infrastructure with systems that work together seamlessly.

Getting started with integration: first gain insight into the business processes

As an organization, you work with a vision and long-term goals. This also includes a flexible infrastructure that is ready for the future. That is why it is good to first fully map out your application landscape and your business processes.

How are your business processes structured and which systems and applications support them? It is important to design the architecture around your business operations and not from your backend applications. You then make much more targeted choices about which connections are really necessary. In addition, you gain insight into the role of a system and you can clean up any 'messes' in your IT landscape.

You should also map out the data to be unlocked: which data should go where and why? Where is data stored? Only when you have insight into your data streams, applications and systems can you start developing a structured integration strategy.

Ultimately, a Service-Oriented Architecture (SOA) is a good goal. Here you design the connections based on business processes and therefore not on the applications that you have running. This way you avoid becoming dependent on the applications you use and a structured and efficient infrastructure is created.

 

Integrate on-premises with cloud applications

So, as a company you want to structure your system integration. However, many organizations still work with a mix of on-premises systems and online applications. You may want to migrate one or more systems to the cloud first, before you start with data and system integration. Whether you are cloud ready depends on how flexible your applications are. The cloud offers the advantage that you can turn off systems and purchase additional computing or storage capacity when necessary. Applications that can handle this flexibly are more suitable for the cloud.
 

It is also advantageous for enterprise, data and system integration when you work with cloud agnostic applications. Cloud agnostic means that your application always works, regardless of the cloud platform. If that is not the case and you are bound to one cloud platform, we call that cloud native. If you are cloud agnostic, it is easier to use (open-source) data integration tools. Ultimately, integrating systems is always possible, regardless of whether they are in the cloud or not.

Download the free E-book

Technical integrations with (REST) APIs

Now that you have mapped out your IT landscape and checked whether all your applications are ready for the cloud (if you still want to migrate them), we dive into the technical part: the APIs! We compare several protocols with each other, so that you get a better picture and can better select what suits your needs.

An API (Application Program Interface) is a connection between two applications and it ensures that data exchange is possible. REST stands for Representational State Transfer and is used to develop HTTP services (webservices), often with an OpenAPI-specification (OAS). The big difference between API and REST API is that an API is written specifically for a platform or application, while a REST service is built on the principles of HTTP and can theoretically also work on other protocols. In any case, you want to build a robust (REST) API that does not collapse, falter or slow down, for example when there is a lot of traffic on your website.

SOAP and REST

Then we have SOAP: a somewhat older protocol that has been used for communication between web services for a long time. Indeed, it is the predecessor of REST. Compared to SOAP, REST is simpler and more modern, which can create the need to migrate from SOAP to REST. A complete migration is quite complex, because SOAP is like an envelope with data, while REST is built from resources in collections. This means that the functional structure of your data changes when you completely switch from SOAP to REST. To solve this, an integration component is often chosen, on which both the SOAP and REST interfaces are available.

Finally, there is GraphQL: a query language in which you specifically indicate what information you want in the response. In contrast, REST is an architectural style that returns an overview of data, divided into objects. The difference is that REST retrieves much larger packets of information than is sometimes necessary, while GraphQL returns very specific responses. Which solution is better? That depends entirely on whether you need query functionality or work with an object model.

A 'one API fits all' does not exist

Unfortunately, there is no single REST API for all of your connections, because you build a REST API for a specific customer, i.e. for an application or end user. An API between applications works differently than an API between an application and end user. The difference is, for example, in caching and authorizations. It is therefore not unusual to have multiple APIs for an object model. Try to avoid developing a point-to-point API for each customer, otherwise you will quickly end up in a tangle of REST APIs. In addition, try to avoid these four pitfalls in system integration when you start working on application integration with REST APIs.

Applying an integration framework

Connections between systems and applications start with an API. But if you expand your APIs, it is time to think about an integration framework to prevent the aforementioned tangle of connections.

Build it yourself versus integration framework

You can program REST APIs completely yourself, but hardly anyone does that these days. There are enough handy frameworks available that will help you a lot with programming, such as the Spring framework for developing Java software. It consists of several open source libraries that belong together and have already been tested

An alternative is a ready-made integration framework: a total package in which all technical details, such as security, performance and documentation, have been taken into account. As a result, much less technical knowledge is required and you can easily build connections yourself. You can therefore spend your time on functional matters. As a result, you are much more productive with a ready-made integration framework than with, for example, the Spring framework.

Learn more

Benefits of system integration with a framework

You can see an integration framework as a power strip to which you connect all your systems and applications via a plug. You can imagine that this offers a lot of flexibility. When you want to adjust or phase out a system, you simply pull the plug, without it having a major impact on the rest of your IT landscape. But also when you want to integrate a new application, you can easily connect it to the integration layer. An integration framework therefore offers a great deal of flexibility to quickly scale up and down. In addition, you benefit from multiple options multiple options, such as dealing with granularity, flexibility during upgrades, orchestration of requests orchestration of requests and mapping, for example, for converting languages.

What should you pay attention to when choosing an integration framework?
  • Whether the functional layer is decoupled from the technical layer. This makes it easier to connect components to each other or to separate them.
  • Whether it is an open-source framework. In that case, you have no vendor lock-in, no high license costs and no need for expensive Java programmers, which is the case with Closed Source.
  • Whether you have to take out a mandatory subscription. With large parties such as TIBCO, MuleSoft or WSO2, this is often the case. This usually means that their open-source solution has limited functionalities and you will pay for the entire suite with additional solutions.
  • Whether there is good support. It is annoying when you have to wait a long time for patches, feature requests and a solution to incidents.
Learn more

How to set up an integration framework

Now that you have chosen to work with an integration framework, it is important to set up all processes properly. Of course, you do not want data to be duplicated or lost. You can read more about data loss later, but to prevent duplicates, there are various methods that you can take into account. Some examples are:

  • Executing a request exactly once. When a backend application is unable to avoid duplicates, the integration layer is the place to compensate for such a shortcoming. That is why each message is given an identifier. This can be used to check whether a message with that specific identifier has already been processed or not, so that a request is processed exactly once.
  • Ensuring guaranteed delivery. You want messages to be processed correctly, so that a request is answered properly. It also helps if you work with HTTP codes for a REST API, so that the user and the developer receive feedback on whether the answer was successful or not.
  • Setting up queueing. A queueing application stores the message until it receives a message from the receiving party that the message has been processed. Queueing makes it possible to exchange data with intermediate persistence.
  • The transactional processing of data. This allows an entire package of data to be processed at once and not partially, which in turn reduces the chance of duplicate data.
Download the step-by-step plan

Maintenance and management of existing integrations

A simple IT landscape with a core system and a few additional applications can be maintained easily enough. But as soon as it expands with multiple internal and external customers and the number of APIs increases, maintenance and management of system integrations becomes a daily task. When you then work in a glass box, including a transparent management environment, it helps to protect your organization against data loss.

Transparency is key for an administrator. An IT landscape that is in a black box is much more difficult to manage due to the lack of insight. In the event of similar incidents in different applications, you have to reinvent the wheel again and again. By adding an integration layer, you standardize a number of things, such as writing to a queue, reading it and calls from web services. You transform into a glass-box environment, as it were, which suddenly makes it measurable how your application chain is running and what potential bottlenecks are.

Data loss and recovery in system integration

If you fail to properly implement data integration, you run the risk of losing data. Failure at an external party, programming errors in your own application or a user error are also causes of data loss. Fortunately, it is usually the case that not all of your data is lost, but that some of it is missing. In that case, it is nice if you have set up your application landscape in such a way that you can quickly retrieve the data and provide it again.

 Three tips to prevent data loss:
  1. Implement an integration layer: you will then have the management and recovery of data arranged in a universal manner.
  2. Use error storage. Messages that have not been processed successfully are stored in the error storage. This gives an administrator insight into what the problem is and what went wrong, so that the process can be adjusted and improved.
  3. Work with an audit trail. With an audit trail, it is possible to follow a unique message through your entire application chain and monitor the processing process.

Always have insight into all your integrations

As an administrator, you want insight insight into the connections that exist, and insight when something goes wrong. An integration framework such as the Frank!Framework has an extensive management environment. This dashboard not only helps to trace unprocessed data, but also facilitates the administrator in many other ways.

Of course, you want to be able to monitor whether all processes are running correctly. This requires an overview of all connections within the integration framework, because then you can check exactly whether a message has been delivered correctly or processed correctly. Analyzing message flows is also part of a good management environment.

In addition to the fact that it is interesting to know that there is an exchange of data between two applications, you also want to see how much data is exchanged, how large those messages are and how long they take to go through your entire application chain. Is something going wrong? Then error logs indicate what went wrong, so that you can resend messages. A proper management environment allows you to solve 99% of problems yourself, without the need for a programmer.

Download our step-by-step plan and gain more control over your application chain in 5 steps

A case involving the insurance market

The Frank!Framework is an open-source integration framework and it is the basis for the integration of business systems and applications. Simply put, the Frank!Framework makes it easier for organizations to integrate all kinds of systems with each other, in order to expand and simplify work. In this blog you can read the 5 ways in which the Frank!Framework makes it easier for insurers.

Learn more

System integration with low-code platforms

According to Gartner, the adoption of low-code platforms is increasing by 20% every year. It is therefore not surprising that most organizations already use a platform such as Mendix or OutSystems in some form. There is more to this than simply purchasing a license. After all, how do you integrate your low-code platform with the rest of your systems? In this blog we will tell you how to do this.

Learn more

Keeping complex integrations, applications and microservices manageable and stable

When system integration becomes really complex, it requires a solid approach. There is dependency between applications and orchestration of APIs takes place. In addition, services and microservices enable scalability. You want to keep such an application landscape as stable and manageable as possible to ensure that all processes run as smoothly as possible. The Frank!Framework helps you with this. We have written an extensive blog about what you need to take into account and which best practices we have encountered.

About the Frank!Framework

With more than 40 experienced integration specialists, we have been working on complex and less complex integration issues since 2003. We provide enterprise, data and system integration with our Frank!Framework: a free open-source framework with ready-made integration solutions for affordable system integration.

Open Source without vendor lock-in

Every developer can get started with our (free) Frank!Framework, because it is an open-source solution. Thus, there is no vendor lock-in. The framework consists of more than 100 integration components for frequently used business and cloud applications. You can easily configure and link these building blocks yourself without programming. The Frank!Framework is continuously under development. When new components are ready, they are made available to all users, so that everyone can benefit from them. In this way, we give something back to the developer community. 

Affordable services

The building blocks are an accessible way to integrate data and systems. That saves a lot of time and money! For example, our integration specialists are available at rates as low as €70 per hour, while a Java programmer quickly costs between €100 and €150 per hour. And because the technical layer is separate from the functional layer, someone with functional knowledge can simply make a connection themselves. So you don't need expensive programmers for that.

Only the support you need

The Frank!Framework is open-source, and anyone can use it. If you ever need specialist knowledge, we are here for you.

You can purchase support from us for any conceivable issue, without having to have a number of licenses. Of course, you are not obliged to purchase support. As mentioned, you can solve 99% problems yourself using the extensive management environment of the Frank!Framework. Would you still like to use a support contract? We would be happy to discuss it with you. You determine the term yourself.

Learn more about Frank as a Service

Proof of Concept 

Of course, we are quite convinced of the power of our integration framework. A Proof of Concept (PoC), however, is an accessible way to discover whether the Frank!Framework is an answer to your problems and meets your needs. Whether it concerns linking two APIs or solving a particularly annoying file, we are happy to get started with it. Though it is often thought that a PoC is expensive, with the Frank!Framework a Proof of Concept is quite affordable and it allows you to discover the possibilities. You can therefore benefit from our integration knowledge without being tied to anything right away.

Solve your integration problems

Schedule a free consultation

Would you like to know more about what we and the Frank!Framework can do for your organization? Or would you like to brainstorm about a specific integration issue? Then schedule a free and non-binding (digital) consultation with one of our integration specialists. We are happy to work with you and provide a customized demo.

Contact us