Data Management

 

Data Management is a group of activities relating to the planning, development, implementation and administration of systems for the acquisition, storage, security, retrieval, dissemination, archiving and disposal of data. Such systems are commonly digital, but the term equally applies to paper-based systems where the term records management is commonly used. The term embraces all forms of data, whether these datasets are simple paper forms, the contents of relational databases, multi-media datasets such as images, or research data.

Key Data Management activities include:

  • Data Policy development;
  • Data Ownership;
  • Metadata Compilation;
  • Data Lifecycle Control;
  • Data Quality; and
  • Data Access and Dissemination.


Measurements of Data Quality

Every organisation is unique, but there are a number of quantitative Data Quality measures that are universal:

DQ Quant. Measure Description
Completeness The degree to which all required occurrences of data are populated
Uniqueness The extent to which all distinct values of a data element appear only once
Validity The measure of how a data value conforms to its domain value set (i.e., a set of allowable values or range of values)
Accuracy The degree of conformity of a data element or a data set to an authoritative source that is deemed to be correct or the degree the data correctly represents the truth about a real-world object
Integrity The degree of conformity to defined data relationship rules (e.g., primary/foreign key referential integrity)
Timeliness The degree to which data is available when it is required
Consistency The degree to which a unique piece of data holds the same value across multiple data sets
Representation The characteristic of Data Quality that addresses the format, pattern, legibility, and usefulness of data for its intended use in addition to quantitative


Data Quality measures should also consider qualitative measures. Some examples include:

DQ Qual. Measure Description
Business Satisfaction Measures The increase/decrease in business satisfaction based on surveys
Collaboration/Improved Productivity Measures Percent of times the Data Governance Council detected and eliminated redundant intra- or inter-departmental projects/initiatives
Business Opportunity/Risk Measures Business benefit gained due to quality data or business risk realized due to questionable data. Increase in competitive analytics due to data availability and Data Quality improvements
Compliance Measures Users with access to update/influence the master data are restricted to only those employees who have need and have been approved access as part of their job functions

It is very important to establish the measures of Data Quality most important to UNSW. This is required to establish a baseline for the quality of your data and to monitor the progress of your DQM initiatives. The other foundational components of the Data Quality Cycle are to Discover, Profile, Establish Rules, Monitor, Report, Remediate, and continuously improve Data Quality.


Components of DQM

Once in place, these key components provide robust, reusable, and highly effective DQM capabilities that can be leveraged across the enterprise:

DQ Component Description
Data Discovery The process of finding, gathering, organising, and reporting metadata about your data (e.g., files/tables, record/row definitions, field/column definitions, keys)
Data Profiling The process of analysing your data in detail, comparing the data to its metadata, calculating data statistics, and reporting the measures of quality for the data at a point in time
Data Quality Rules Based on the business requirements for each Data Quality measure, the business and technical rules that the data must adhere to in order to be considered of high quality
Data Quality Monitoring The ongoing monitoring of Data Quality, based on the results of executing the Data Quality rules, and the comparison of those results to defined error thresholds, the creation and storage of Data Quality exceptions, and the generation of appropriate notifications
Data Quality Reporting The reporting, dashboards, and scorecards used to report and trend ongoing Data Quality measures and to drill down into detailed Data Quality exceptions
Data Remediation The ongoing correction of Data Quality exceptions and issues as they are reported