“Data profiling” is not something we do in our personal lives, and it is not something that the majority of business people – or have ever done – in their professional roles. So, when it comes to explaining the benefits of data profiling to business people, there is no foundation of past experience that can be leveraged.
Indeed, business people may think it is some concept that is used by IT for their purposes, but they do not know why. This is made worse if data profiling is presented as an abstract capability which is not related to business people’s everyday lives. And perceptions can be even more negative if presentations of data profiling include a lot of technical jargon that is not easy to understand.
Yet, data profiling can bring tremendous benefits to many business areas and the technology involved is much more accessible to businesspeople in terms of ease of use than it has ever been. So let’s take a brief look at five major business benefits that data profiling can bring.
1. Easily Find Data That You Need
More and more data processing is being done in the business. Many business analysts have learned SQL and many are learning Python. Reporting packages with great user interfaces are being adopted too. Gartner estimates that by 2023 Citizen Developers in the business using no-code/low-code tools will outnumber professional IT developers by 4 to 1. [https://venturebeat.com/2021/10/22/gartner-citizen-developers-will-soon-outnumber-professional-coders-4-to-1].
The problem is that while businesspeople have all these tools available and know how to use them, they cannot simply “see” where the data is for each requirement they want to solution. This is like the data discovery problem that is well known for data scientists – but on a massive scale.
Imagine you are a retailer, and you want to offer a gift card – in Bitcoin – via a social media marketing campaign. Where are the social media handles of the existing customer base? Maybe hidden in fields like “Address Line 3” dispersed across a variety of odd tables in various databases. Without data profiling, there is no way to find out, and the campaign is not going to get off the ground effectively or efficiently.
2. Quick Check if the Data Is Good Enough for Your Use
Data quality is often described as fit for use. This certainly includes objective defects like coding errors and misspellings, but the data has to fit the business requirement at hand to be “fit for use”. This has to be judged every time for every business requirement. It will vary from one requirement to the next.
Let’s suppose the Treasury department of our retailer wants to securitize its credit card receivables. That is, it wants to bundle up these debts and sell them as bonds to get working capital. Data profiling might show that for each cardholder the State of Mailing Address is always filled out, but the State of Residence has a lot of missing values. This is because the company has only been concerned about mailing cardholders their statements and similar documents so far, but has never used the State of Residence field for anything.
Now, the investment bank the retailer is working with has told them that the portfolio of acceptable debts cannot include a dollar total of more than 15% for any State of Residence. Only data profiling is going to help our retailer get around the issue of missing State of Residence data. However, the profiling can be done quickly and easily, and businesspeople can use the results to construct an acceptable debt portfolio.
3. Prove That You are Managing Personal Information
Business units everywhere are suddenly finding themselves accountable for proper management of personal information (PI) with the explosion of data privacy laws like the GDPR and CCPA. This was not something they had to do in the past.
A first step is to know if a database even has PI in it. If it does not, then there is no need to be concerned about the data privacy laws. If the database does have PI, then more work needs to be done. With data privacy, the presence or absence of PI has to be proven. It is not possible to rely on, say, someone’s assumption or opinion. And even if PI is present, if it is transitory in nature, then data privacy laws generally do not apply.
Data profiling can prove if PI is in a database or not, and if it is present, just how transitory it is. Businesspeople now have an objective proof of whether they need to apply data privacy rules or not. Furthermore, this proof has to be periodically updated – which data profiling can easily do.
It is difficult to see how business units can avoid using data profiling if they want to manage their PI obligations rather than being told what to do by some other group – but still remaining accountable if anything goes wrong.
Of course, there is a vast amount more to managing PI than this first step, but all along the way data profiling can help and accelerate the process.
4. Keep an Eye out for Changes
Today, more than ever, businesspeople rely on the stability of their data environments. Traditionally, business units have been at the mercy of IT in terms of structural and other high-impact changes. Because no one knows the full set of interrelationships between systems, a change implemented in one environment can produce an unexpected change in some downstream environment that was not realized. This is usually detrimental.
What business units can do is implement repetitive data profiling across the environments that matter to them. Today, data profiling tools tend to only look at changes after an initial scan, so while the initial scan may take a while the subsequent ones are much faster. This enables the business to get notified if the data profiling tool sees that a column has disappeared, or been renamed, or now has a different datatype. Similarly, the content of code tables can be monitored – the tool can detect the addition of a new record to, say, the Customer Type table – something that would be expected to be important.
If the business gets notifications of changes they were not expecting, they can take action promptly, rather than waiting for lots of things to fail and then trying to figure out why it is happening. Proactive change management at the metadata level can help avoid inaccurate reporting and processing in downstream systems.
5. Get Rid of Data Deadwood
As business units become more accountable for the data environments that they own, and as data migrates more and more to the Cloud, data administration tasks become influential to the business. While data storage is cheap, it is not free, and huge data volumes mean big bills from Cloud providers.
One thing the business can do is identify tables and columns that are not used. They are empty, or have information that never changes, or are never used for anything. These are areas where data “dead-ends”. Temporary tables left over from development work are one example.
Data profiling brings visibility to the data environment. The business can use it to identify columns and tables that can be removed. Obviously, this needs to be done in a controlled manner, but it is now possible for the business using data profiling tools to identify them. We can expect business units to be doing more and more data administration in the future and with the current growing data explosion data profiling will be instrumental in enabling the business to have quick transparency to help them make the right decisions about managing their data.
Taking the Next Step
Our brief review of these five business benefits of data profiling by no means exhausts the potential. It should also be remembered that every business unit will have some special needs that are rarely seen elsewhere – but which are important, and part of the solutions can include data profiling. This is why it is so critical to help the business understand how to apply data profiling benefits in their particular environments.
At Global IDs, we are passionate about providing powerful data profiling tools that are easy to use by the business. We bring a unique data profiling capability with predictive data quality recommendations and fingerprinting. Contact us to find out more at www.globalids.com