How our pocket friendly In-House Integration Solution is helping clients in migrating and integrating data from a heterogeneous data sources to another homogeneous / heterogeneous data source(s)?

One of the biggest bottlenecks for an organization is to migrate and integrate from multiple legacy and external applications to a single point. In ideal scenario, migration and integration of data from multiple data sources to an application having different data sources takes up ~70% of the implementation efforts and cost. To overcome this problem and help our clients in saving both time and money, Emeis has a unique in-house integration solution tool named as B2B. Our B2B integration tool helps in migrating and integrating data from heterogeneous data sources to another homogeneous / heterogeneous data source(s) taking into account all the aspects like volume of data, types of data sources, different schemas, performance, security concerns, resource constraints, scalability etc.

Modes

The tool can be configured to work in different modes. It is supporting following scenarios.

• Push : Real Time
  • Direct communication with the configured end points take place in synchronous manner which also provides the returned objects from the integrated end point. The tool needs to be configured with the end point details followed by the data mapping for data transformation. More details can be found in the designated sections.
  • Distribution mode is for the scenarios where data needs to be distributed/replicated to multiple destinations in real time without impacting the main application’s performance. The tool receives the data to be distributed from source and pass it to all destinations after transformation.
• Pull : Scheduled
  • Sync and Migration is supporting the data synchronization and migration requirements where initial load of data into the systems has been addressed. It further provides the facility to keep the data synchronized between the systems on a configured data interval basis.

In all modes, the data mapping for data transformation rules are configured using the common mapping and configuration facility. More details below.

Sources/Destination type

Our B2B integration tool has the capability to handle multiple data sources of different types like

• Excel/CSV/Json (share at FTP/SFTP, shared folder, etc.)
• Database
• API
• CRM
• Custom (Customer in plugin custom code to solve special scenarios).
• Integration tool has a master page to configure details:

Mappings

It has mappings facility for each mode. There could be multiple mappings within a mode/job. On high level, it can be outlined as follows:

Such mapping are done through user interface which maps the source with the destination objects.

Items in mapping on high level are as follows.

1. The source column name
2. The source column data type
3. The source column data length
4. The validation to be applied on the source column
5. The destination column name
6. The destination entity name will be mentioned on primary level.
7. Lookup columns parent entity column name.
8. Default Value (in case the source column is NULL) to go to a destination attribute.
9. Validation configurations like length, max values, complex cases using regex, etc.
10. Custom validation at field or object level using plug and play of custom code.

Background Service       

A service run in background and keep checking the timings for the schedule and triggers the sync as per the configuration. It could be nightly, hourly, etc.

Notifications     

The distribution list defined on the master Job level gets the email notifications for success and failure of jobs like the one below.

The details of failures is logged in Log repository and the complete import log is part of Import logging framework.

Logs     

UI is available which shows the stats of the action going on in the background. Three broad categories for each job are defined: Warning, Success and Error count in a job. Rows with warnings will be added with the number of records imported/updated.

Reconciliation Report    

Reports are available to monitor the on-going background activities. These can be an attachment of the outgoing notification mail, if configured.