Challenge: Provisioning Data for Analytics

Overview

Organizations across industry domains struggle with the data provisioning process to support detailed statistical analysis. This analysis performed via specialized data analysis professionals and software is subject to constraints due to the structure of the sourcing applications and databases. Problems with data quality, integrity and consistency are costly and cannot be remediated solely byimplementation of “best of breed” software solutions. Provisioning Data for Analytics, takes a closer look at common root cause issues and offers a qualified perspective on addressing this subject area.

Business Context

Many businesses are grappling with the problem of getting correct complete and accurate data forAnalytic Analysis. Due to the events of the past decade and the evolution of the financial servicesindustry data analytics has become the “lynch pin” for managing risk, meeting compliance standards andenabling regulatory reporting.  Not only does back-end data analysis address oversight requirements it contributes significantly to thefuture of the enterprise.  The analytical tools have unique statistical reporting capabilities which have also become an integral part of Customer Relationship Management (CRM) and Product/ServiceMarketing functions. In fact due to the lack of reliable databases, to support diverse business needs, data captured for a specific purpose such as risk management is frequently repurposed for marketing or financial reporting needs. To address information needs we have observed that businesses have developed stand alone, independent extract, transform and load (ETL) processes specific to each analytic function. This approach compensates for factors such as:

• lack of an Enterprise Data Warehouse (EDW).
• inability to rely upon the EDW they have created.
• need to supplement or enrich the data from the EDW per each analytic process.

The impact to the business is huge. Operational and strategic functions become limited in their abilityto drive business decision making and meet third party and compliance reporting requirements.Frustrated data and business analysts report:

• An inability to get the information needed to make decisions
• Different decision support processes generate inconsistent data results
• Failure or inability to readily locate the data they need to meet business requirements
• Data Transformation processing takes too long and costs too much
• Difficulty loading the data you need into the analytic engine

Due to an ever competitive business climate and significant pressure on operational costs and investment it has become critical to address the overall data processes which support the business analysis functions. Major concerns are not only the direct costs but the timings and quality of the information delivery processes.

Challenges to Reengineering the Analytics Process

Many analytical functions are supported by a myriad of feeds, conversions, manual and automated data remediation processes, applications and software products. As stated earlier, these have been deployed not taking an architectural based design approach. The reengineering of these data processes is non-trivial and is not purely an architecture based project but the complexity may be significantly reduced based on architecture directions.  In summary the business objectives of a re-engineering effort include:

• Ability to get the answers you need to make decisions in a timely way
• Reduction in costs to maintain multiple ETL processes and tools as well as versions of the data
• Eliminate the probability of analysis producing erroneous results leading to bad decisionsData Quality
• Achieve acceptable data integrity, accuracy and precision
• Reconcile or eliminate timing issues between components• Implement a consistent view of information
• Eliminate Inconsistent information from the sources due to loss of accuracy during transformation

HPSquared recognizes the business needs and the challenges facing these problems. We are able to assist in defining an approach and implementing solutions which will over a series of releases deliver improvements to your firm’s Analytics’ function performance. Our objective is to fast track the redesign of the enabling data architecture  and to simplify ETL processing with the end result of improving business decision making capabilities.  To serve you we offer expertise in:

• Data and Information Architecture• Development of Data Warehouse Solutions
• Implementation of Data Governance and Stewardship Processes and Organization
• Development of data quality improvement programs
• Subject matter expertise with leading analytic vendor product offerings, ETL solutions, and master data management applications and strategies

 

MDM and Its Relationship to the Information Management Maturity Model

A Point of View Provided By:

Philip Teplitzky (phil.teplitzky@hpsquaredllc.com)

Marc Hurst  (marc.hurst@hpsquaredllc.com)

March, 2010

Key Words: methodology, data quality, data governance, analytics, enterprise data, information management

Introduction

Master Data Management (MDM) initiatives have become commonplace as enterprises attempt to establish a higher quality basis for business performance analytics.  A myriad of approaches have been taken to establish higher quality reference data.  Dominant database players such as Oracle, IBM and SAP have jumped in and have offered solutions or have acquired MDM software companies.  MDM is often viewed as a relatively rapidly applied band-aid to cover up severe and persistent wounds.  When management buys into the quick-fix the problem does not go away.  An overall information management environment is not a single dimension it is a multitude of systems, including automated and manual processes.  These components are critical to an effective MDM initiative.

Our analysis is to facilitate discussion about how to make MDM successful.  By discussing a high level view into what it is and linking it to criteria incorporated into the evolving Information Maturity Models.  With our objective is to discuss the need for a holistic information management plan which will better position the organization for the benefits which MDM offers.  Thus this is in part an attempt to bring forth a reality check for those who are embarking on this journey.

Objectives:  Why This Matters

Having led teams in the data space for several decades we have seen a myriad of circumstances around data management.  Initially the development of enterprise operational data bases, custom designed and supporting custom applications was the trend.  This transitioned into point solutions with little if any data sharing across applications.  ERPs came into dominance enforcing their view of information.

Resulting was a high degree of variation in defining the business rules, format and context of the primary business objects.  Each application became its own universe and to enable a broader view large-scale data warehouses or ERPs were implemented.  These did not resolve data problems just met different business needs.  As data quality and MDM solutions became prevalent IT groups pitched that a solution to data problems was available.  These solutions provide high quality technology to provide insight into business performance and to a degree address data quality issuers.  Yet the MDM/DQ initiatives are frequently treated as a single self-contained event.  Augment the staff by hiring outside consultants, purchase software, develop data transformation modules and build new data stores.

What has been learned is that these processes are not discreet one time initiatives.  They are continuous in nature not only adding domains of subject areas in subsequent phases but establishing a management system which monitors activity.  In addition root causes for the problems are traceable to source applications, variation in business processes and no oversight or governance.   What is clear is that MDM is more than a solution it is a catalyst for new business process; workflows and data governance for information management.  Information is a critical asset to the business subject to internal and regulatory standards and compliance and thus the appropriate environment or ecosystem should be provided.

MDM is not an IT solution; it is a business wide program. It is complex supporting multiple consumers of information and charged with establishing a degree of order amidst chaos.  In addition it offers an opportunity to simplify dataflow and architecture supporting the linkage between front office and back office information needs.

DQ and IMM are related with dependencies; IMM best practice evolution creates ecosystem for DQ and MDM benefits to be achieved.

Getting on the Same Page:  Address Terminology

Everyone has to be on the same page, or at least in the same area code.  There is no standard formula for programs of this magnitude.  Every business has its unique business priorities and needs.  When addressing MDM the effort to establish a corporate vocabulary and also a common view of the objectives and the end state goals, is mandatory.

It is one of the problems of discussing this subject, everyone has their own definition and understanding – we need “Definitional Governance. This is non-trivial as the ramifications of an MDM program are immense.  From an IT view the variations between the pundits, vendors, analysts and thought-leaders on the meaning of terms is significant.   Thus the need for proprietary definition within the business itself,  moderated by the realities of what the technologies do,  what infrastructure is in place, standards and the propensity of the overall IT and business to readily adapt to change.

Here is just a subset of the terminology applicable to an MDM program:

• Master Data Management
• Meta Data Management
• Data Quality
• Information Quality
• Operational Data Store
• Enterprise Data Warehouse
• Information (Data)  Management Maturity

Additional definition and clarification address “conceptual” concepts and terms such as:

• Data vs. Meta Data
• System vs. Meta System
• Continuous / Business Process Improvement
• Control Systems and Feedback
• Self Correcting systems
• Information/Data Quality: Definition and Measurement

Once the objectives, scope, objectives and goals are established around key terminology you will have standard definitions.  As the initiative commences we recommend that the project:

• Establish expectations,
• Develop business cases
• Syndicate  goals and progress continuously

An Interaction Model for Information Management

The performance and quality depends on a well-defined set of relationships establishing what we are calling the Interaction Model for this analysis.  We propose the following components as significant to inter dependencies and to meeting MDM expectations:

• Operational System: that process  the inputs/transactions
• Data Storage: the data bases that maintain the semantic model
• Data Cleansing Systems : applying edit and validation and business rules
• Data Governance and Stewardship Functions: oversee the application of rules and processes as well as applying acceptance and remediation of information
• Meta Management System:  that monitors the operation of the Operational Systems

Implications of the Interaction Model lead into a discussion of the Information Management Maturity Model.  This model, also known as the Data Maturity Model, has become a major focus across industry.  There is no single standard definition of the phases and capabilities but significant progress has been made by industry organizations and consultancies.  We have stayed close to the ongoing work of the Enterprise Data Management Council working closely with Carnegie Mellon University on incorporating and integrating extensive field research for its Data Maturity Model.

The Information Management Maturity Model (IMM) is a work in process.  By no means prescriptive yet it is evolving rapidly.  The Enterprise Data Management Council has taken a broad scale approach to establishing the paradigm as a framework for making data and information more reliable, consistent and useful.  Other consortiums and groups addressing the IMM include: DAMA, MIKE 2.0, and MAG.

Lakefront Data a leading consultancy and thought leader offers their view of IMM providing an evolution of capabilities until maximum maturity factoring in the following dimensions or sectors:

Master Data Management (MDM) and Information Management Maturity

MDM cannot be created in a vacuum; it is part of a larger Information Process – an Information Ecosystem. The IMM outlines the capabilities across Information Management domains thus establishing the ability of the company to successfully address major initiatives.  MDM is a major initiative.

MDM is also a continuous process.  It happens each day in response to business events and transactions.  Its success is predicated on the effectiveness of many systems.  These systems are all embedded into the IMM which will rate an entity’s ability to execute effectively in information management.  This relationship leads to the following assumptions:

• The higher the level of Data Maturity the higher the probability of having a more effective and efficient MDM environment
• By inference if  you ignore the Data Maturity domains you will not have an effective MDM process
• The higher your level of Data Maturity implies the a greater impact of the MDM functions

I came across this today, according to industry pundit Phil Simon (in his post “Data Quality Lip Service” from March 4, 2010 on Dataflux’s blog site) management gives lip service to data quality.

“They (management) realize that they are often powerless to effect standards and consistent DQ practices throughout the organization. The bigger the organization, the more difficult it is for change agents and DQ proponents to put their ideas into action. Building consensus within a department is often difficult enough. Throw multiple departments and multiple countries into the equation and your job just got exponentially harder”.

MDM requires fundamental change to so many complex processes, multiple business units, geographies and functional departments.  This speaks for establishing scope for MDM programs which is achievable and offering reasonable return of investment while also being a building block for further MDM projects.

Conclusions: Making an MDM Program Successful

We offer the following set of objective based activities to aid an organization in its journey into an MDM program.

• Create a formal Taxonomy and Ontology of terms to limit ambiguities – avoid the I thought you meant syndrome
• Benchmark against the EDM Council and DAMA Maturity Models – Understand target state and the roadmap to get there
• Define MDM goals and objectives based on business priorities
• Determine what your target level of Information Maturity is to provide an acceptable level of MDM quality
• Establish an achievable remediation program
• Establish a Business Process Redesign function to implement the remediation plan
• Establish an appropriate MDM performance management process (KPIs/KPMs and tracking methods)
• And most importantly: Going Forward:  Measure, Fix and Re-Measure: the processes, systems, organization and information management ecosystem that enable MDM

Thank You Visit Us at hpsquaredllc.com