With the increasing flood of data, the importance of data quality and data integrity is growing. Both concepts are essential if the raw material “data” is to be used profitably. Every company and every institution works with data – be it in accounting, customer management or in the analysis of business-relevant KPIs. Even a simple example from accounting shows how crucial it is to record and control data precisely: missing values for three months of incoming payments would be more than problematic.

The more data-driven companies become, the more important it is to systematically monitor their data. The closely related, yet different concepts of data integrity and data quality play a key role here. This article explains the differences between the two terms and shows how companies can improve their data quality and integrity in a targeted manner.

Data integrity vs. data quality

Data integrity and data quality are best distinguished by their underlying objectives. The purpose of data integrity is to ensure the reliability, consistency and security of data. It includes ensuring the trustworthiness of data and the consistency of relationships between data sets.
Data quality, on the other hand, guarantees the usability and added value of data. It refers to ensuring the accuracy, completeness and informative value of data. There are overlaps between ensuring data integrity and data quality. For example, missing or duplicate values pose a problem for both data integrity and data quality.

The following scenario can help to clarify the difference between the two terms: If a project requires data in a certain format and a data scientist formats the raw data accordingly by overwriting it, this can improve data quality in the short term – but at the expense of data integrity. An example: If a column with monetary values in multiple currencies is overwritten directly so that only values in a single currency are included, the consolidated data appears immediately applicable. However, the original values are lost. Since exchange rates are time-dependent, it may be impossible to correctly reproduce or restore the original values. This not only impairs data integrity, but can also have a negative impact on the future usability of the data – and therefore its quality.

Data integrity vs. data quality

The ALCOA principles are a proven approach to ensuring data integrity. These originally come from the life sciences, but have also become established as fundamental principles in the data sciences.

The ALCOA principles define central requirements for the quality and traceability of data:

• Attributable: Data must be clearly assigned to a source or person and be traceable through mechanisms such as timestamps and automatic logs.
• Legible: Data should have a fixed memory address and a standardized format so that it remains permanently readable.
• Contemporaneous: Data must be documented exactly at the time it is recorded – not before and not after.
• Original: Primary data must be preserved in the course of further processing and must not be falsified by subsequent changes.
• Accurate: Only error-free data that has not been subsequently edited may be recorded; corrections should be documented in a comprehensible manner.

The ALCOA principles are often extended (ALCOA+ or ALCOA++) – for example, to include Complete, Consistent, Enduring and Available. However, the core principles remain unchanged and form the basis for integrity and reliability of data.

Principles of data quality

While ALCOA ensures that data is correct, unchanged and traceable, this alone is not enough to guarantee high-quality data for analytical or operational purposes. This is where the DAMA-DMBOK (Data Management Body of Knowledge) framework of the Data Management Association (DAMA) comes in.¹ comes into play. The DAMA framework is often used in companies to ensure that data is not only integrity-assured but also of high quality. It supplements the data integrity principles with dimensions that ensure the suitability of data for its intended purpose:

1. Accuracy: Does the data correspond to reality?
2. Completeness: Is all required data available?
3. Consistency: Is the data consistent across different systems?
4. Timeliness: Is the data current enough for its use?
5. Validity: Does the data comply with the defined rules and formats?
6. Uniqueness: Are there duplicates or unnecessary redundancies?

Challenges during implementation

Despite proven principles such as ALCOA+ and the DAMA-DMBOK framework, there are numerous challenges in practice when it comes to ensuring data integrity and data quality. Typical problems that need to be regularly checked and rectified include

• Lack of automatic backups: risk of data loss and lack of traceability
• Missing or duplicated values: Impairment of data consistency and accuracy
• Outliers or implausible values: risk of incorrect analyses and decision-making processes
• Incorrect column formatting: Difficult processing and lack of standardization
• Incompatibility of data from different sources: Challenges in integration and analysis

To overcome these challenges, there are now a large number of automated solutions that are either rule-based or controlled by machine learning.

One example of such automated quality assurance is the platform Great Expectations². It makes it possible to define explicit expectations for data, for example:

• Uniqueness of values (no duplicates)
• Compliance with defined value ranges (e.g. sales values between 0 and 1 million)
• Format specifications for certain columns (e.g. date formats or numerical values)

Great Expectations can be used flexibly – both in cloud environments and in local systems – and can be seamlessly integrated into existing data pipelines. The results of the quality checks are automatically logged and transmitted, including automated email notifications in the event of critical deviations.

Challenges during implementation

To systematically increase data quality and integrity, companies should implement the following measures:

• Inventory of current data quality (audit of data sources, gap analysis)
• Introduction of standards and governance guidelines (e.g. metadata management, access controls)
• Integrate technical solutions (automated checking mechanisms, data quality reporting)
• Training and sensitization of employees (avoidance of human error, promotion of data-driven decision-making).

Conclusion

Data quality and data integrity are crucial for companies that want to be data-driven. Both concepts are closely linked, but pursue different goals. Finding the right balance between data quality and integrity is essential to provide reliable, secure and usable data.

How integrity and quality can or must be ensured depends crucially on the current data infrastructure, the regulatory environment and specific use cases. However, due to the advanced development of methods in the data sector, tried and tested solutions can be used in most cases to make data more secure, reliable, useful and easier to use.

Footnotes

¹Data Management Body of Knowledge: dama.org

²Great Expectations: greatexpectations.io

Further sources

Alosert, H., Savery, J., Rheaume, J., Cheeks, M., Turner, R., Spencer, C., Farid, S., & Goldrick, S. (2022). Data integrity within the biopharmaceutical sector in the era of Industry 4.0. Biotechnology Journal, 17, e2100609. https://doi.org/10.1002/biot.202100609