Compared to two decades ago, when data Integration architecture was a more straightforward operation for most organizations, data integration has become the connective tissue that holds the modern IT environment together. In the past, it was a bit like a small-town road system, with few connection points, and no traffic jams to be concerned about.
The IT environment of today looks very different. On one level, the enterprise is simply dealing with more requirements for data integration. Companies have deployed multiple ERP systems, CRM, marketing automation, point-of-sale systems, mobile applications, etc. The small-town road system of yesterday has been replaced by a network of highways and interchanges that need to accommodate higher traffic volumes, with substantially more sophisticated security and reliability mechanisms.
It is easy to see how data integration has developed into a key competency for business success. Here are some data integration techniques and best practices for planning your roadmap.
Begin with the end in mind
Anthony Scriffignano, Chief Data Scientist at Dun & Bradstreet, advises: “Never lead with a data set; lead with a question.” Every data Integration architecture should begin by establishing clear objectives for the project. Is the organization trying to optimize operations and gain efficiency? Are you aiming to understand your customers and prospects better to gain a competitive advantage? Does the business need to modernize its landscape and be better prepared to adopt new technologies?
Well-executed data integration projects can yield tangible results. What results are you targeting, and how will you measure them? There are many reasons to embark on a project to improve data integration. It is vitally important that IT leaders be clear about their reasons for doing so.
Decide which data sources to include
The business case should drive decisions regarding which data to include.
Traditional mainframe and IBM I systems continue to play a key role in the operations of most large enterprises and even many small and mid-sized companies. These systems hold core transaction data essential to most data integration initiatives.
Next, companies should look at the disparate software systems throughout their landscape and identify the role that data from each of those systems should play in meeting the objectives laid out in the business case.
Finally, look beyond the organization’s boundaries to determine whether and how you will incorporate data from external sources. This could include third-party data for analytics (such as consumer location data and traffic analysis that can drive site selection decisions for a retail store).
Minimize complexities in data integration techniques
Data transformation is fraught with complexity. For example, mainframe data can be especially challenging due to some anomalies associated with variable length records and COBOL copybooks. To make matters worse, it is getting more and more difficult to hire people who have the skills required to make sense of these complexities and the skills to handle newer technologies.
Enterprise-grade integration tools can simplify the process by managing complex data types behind the scenes. IT can focus its attention on the important matters of designing and validating the integration roadmap instead of mastering the arcane details of mainframe data.
Determine data communication methods
There are numerous things to consider when determining how data will be communicated. For example, real-time integration is the gold standard for which most organizations aim. Also, batch-mode integration can adequately address several scenarios and even be built to meet an “almost real-time” standard. It is important to consider both current and future volumes of data to assess whether pipeline capacity will be adequate to handle the traffic.
Ultimately, the communication method you choose will depend on your objectives. Where do you want the data to live, and what do you plan to do? Again, it comes back to the business case. Suppose your objective is to consolidate information from multiple sources into a big data platform for analysis. In that case, your strategy will look very different than if you are pushing changes on a customer record from a legacy system to a marketing automation system.
Consider future needs
Today, businesses have access to a range of new technologies unknown five to ten years ago. We have seen an explosion in the amount of data available. Thus, innovation has followed suit, as businesses have discovered new ways of using new technologies and data to increase efficiency and drive competitive advantage.
Data integration is generally not a ”once-and-done” project in today’s world. As the organization evolves, new technologies emerge, and different information sources become available, you need to reassess and realign your data integration architecture techniques periodically to serve business objectives.
Conclusion
In conclusion, companies must establish an integration architecture strategy for robust, adaptable tools and frameworks that can support their organizations. Also, new technologies emerge organizational needs evolve, and new opportunities come to light.
