Integration Basics

What is Data Integration?

In today’s world of technology and smart business decisions, data integration plays a significant role.

Integrating data generated from multiple applications and working on it has become the flagship of some of the IT projects run by various organizations around the world. Not only that, but the need for improving data accessibility, enhancing teamwork and collaboration, as well as the need for reports and dashboards have also given rise to the idea of data integration.

Data integration is the amalgamation of a systematic series of operations used to combine data from disparate applications into valuable information.

Data integration occurs when a variety of data sources are blended into a single database, offering users of that database efficient access to the information they need. DI aims at providing an integrated and consistent view of data coming from internal and external data sources.

There are several ways to integrate data:

Manual data integration A process by which an individual user manually collects necessary data from various sources by accessing interfaces directly then cleans it up as needed and combines it into one warehouse. This is highly inefficient and inconsistent and makes little sense for all but the smallest of organizations with minimal data resources.

Middleware data integration is an integrated approach where a middleware application acts as a mediator, helping to normalize data and bring it into the master data pool. Middleware comes into play when a data integration system is unable to access data from one of these applications on its own.

Application-based integration is an approach to integration wherein software applications locate, retrieve, and integrate data. During integration, the software must make data from different systems compatible with one another so they can be transmitted from one source to another.

Uniform access integration is a type of data integration that focuses on creating a front end that makes data appear consistent when accessed from different sources. The data, however, is left within the original source. Using this method, object-oriented database management systems can be used to create the appearance of uniformity between unlike databases.

Common storage integration is the most frequently used approach to storage within data integration. A copy of data from the original source is kept in the integrated system and processed for a unified view. This is opposed to uniform access, which leaves data in the source. The common storage approach is the underlying principle behind the traditional data warehousing solution.

Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on-premises and cloud-based processes, services, applications and data within individual or across multiple organizations.

Functional Components of iPaaS

Integration platform typically contains a set of functional components, such as: 

• Communication protocol connectors (FTP, HTTP, AMQP, MQTT, Kafka, AS1/2/3/4, etc.).
• Application connectors/adapters for SaaS and on-premises packaged applications.
• Data formats (XML, JSON, ASN.1, etc.).
• Data standards (EDIFACT, HL7, SWIFT, etc.).
• Data mapping and transformation.
• Data quality.
• Routing and orchestration.
• Integration flow development and lifecycle management tools.
• Integration flow operational monitoring and management.
• Full lifecycle API management.