Data Cleaning for AI Projects: Methods FDEs Use on Messy Customer Data

What Is Data Cleaning for AI Projects?

Data cleaning for AI projects is the process of detecting and correcting errors, inconsistencies and gaps in data so that models and AI features built on it behave correctly. It covers fixing formats, standardising values, handling missing entries, removing or merging duplicates, treating outliers and correcting wrong labels.

A systematic literature review by Côté and colleagues (Polytechnique Montréal), covering papers from 2016 to 2022, groups the field into six activities: feature cleaning, label cleaning, entity matching, outlier detection, imputation and holistic cleaning. That is a useful map, and the methods below follow it.

Does Data Cleaning Actually Improve AI Results?

Usually, but not automatically. The CleanML study (Li et al.) tested how cleaning five error types, namely missing values, outliers, duplicates, inconsistencies and mislabels, affects machine learning classification. Its findings were mixed: the impact varied by dataset and was not always positive, and it differed by error type. For example, imputing missing values usually helped, while removing duplicates more often had an insignificant or even negative effect.

The practical lesson is to treat each cleaning step as a hypothesis. Clean, measure on a held-out set, and keep the change only if it helps or if it is needed for correctness. "Cleaner" is not the same as "better for this model".

Why Customer Data Is Especially Messy

Customer data comes from many hands and many systems: sales reps typing into a CRM, web forms, payment systems, support tools and spreadsheets. In an Indian context, common patterns include phone numbers with or without +91 and leading zeros, names transliterated several ways, day-month-year dates mixed with year-month-day, free-text addresses, and mobile numbers shared across family members.

The FDE Method: Eight Steps in Order

The order matters. Fixing duplicates before standardising formats, for example, means your matching rules will miss obvious pairs.

1. Profile before you touch anything

Measure what you have: row counts, null rates per column, distinct values, ranges, formats, and the most common values. Compare against what the business says the data should be. Profiling often reveals the real problems, such as a column that is 70 percent empty or a status code with twelve undocumented values. This belongs in the discovery stage of the FDE project lifecycle.

2. Ask the data owner what the columns mean

No tool tells you that "status 4" means "closed by phone". Sit with the person who uses the system. Record definitions, known quirks and which source is authoritative for which field. This step avoids more errors than any code.

3. Fix types and formats

Convert text to proper dates, numbers and booleans, and standardise encodings and units. The classic trap is ambiguous dates. "03/04/2025" is 3 April in a day-first system and 4 March in a month-first system. Confirm the convention with the source owner instead of guessing, and parse each known format explicitly.

python

import pandas as pd

Notice that the unparsed flag keeps failures visible instead of silently dropping them. A blanket dayfirst=True setting can misread ISO-formatted dates, which is exactly the kind of quiet error that corrupts a model.

4. Standardise values

Trim whitespace, normalise case, map synonyms to one canonical value, and strip formatting from identifiers such as phone numbers, keeping only digits and validating length. Maintain mapping tables for categories (for example, city names) that people can review and extend. Keep the original column next to the cleaned one so you can trace and undo changes.

5. Handle missing values deliberately

First ask why the value is missing. A blank might mean "not applicable", "not collected", "refused" or "system error", and each has a different meaning. Then choose:

• Leave it missing and let the model or a missing-indicator feature handle it.
• Impute with a median, mode, group-based value or model-based estimate.
• Drop rows or columns only when the loss is small and not biased toward one group.
• Go back to the source when the field matters and can be recovered.

Always add a flag column showing which values were imputed. Also check sentinel values such as 0, 9999 or "N/A" that stand in for "unknown".

6. Deduplicate and resolve entities

Exact duplicates are easy. The harder problem is the same real-world customer appearing as different records. Entity resolution links them using several signals: normalised name, phone, email, address and date of birth. Typical approach:

1. Block records into candidate groups by a cheap key (such as the last four digits of a phone number) to avoid comparing everything with everything.
2. Score pairs using string similarity and exact matches on strong identifiers.
3. Decide with thresholds: auto-merge high-confidence pairs, send the grey zone to a person, and keep low scores separate.
4. Build a golden record with explicit rules for which source wins per field.

Wrongly joining two customers is usually worse than leaving a duplicate, so keep merges reversible and logged. Send uncertain pairs to human review, using the approach in LLM guardrails and human-in-the-loop design. Duplicates also cause leakage: if the same customer lands in both training and test sets, your evaluation will look better than reality.

7. Check outliers and labels

Look for impossible values (negative ages, future birth dates) and extreme but possible ones (a very large order). Do not delete outliers automatically; some are your most valuable cases. Investigate, then correct, cap, flag or keep.

Labels deserve equal attention. Northcutt, Jiang and Chuang's work on confident learning argues that label quality is often overlooked compared with model predictions, and offers a method for finding likely label errors in a dataset. In customer data, wrong labels often come from process quirks, such as a "churn" flag set by a billing batch job rather than by customer behaviour. Review a sample of labels with someone who knows how they were created.

8. Validate continuously

Cleaning is not a one-off script. Write automated checks that run on every data refresh: required fields present, values in expected ranges, no sudden change in null rates or row counts, no new unseen categories, and no schema changes upstream. Agree these as a contract with the data owner, an idea we cover in the broader context of enterprise pipelines. Fail loudly and alert someone, because silent drift is what breaks models months after launch.

Document Everything

Keep a cleaning log: each rule, why it exists, who approved it and what it changed. This helps with audit, debugging and handover, and supports privacy and accountability duties. Customer data often contains personal information, so involve the customer's data protection owner early and check applicable law, such as India's Digital Personal Data Protection Act, with their legal team. Our guide to AI governance for forward deployed engineers covers this in more detail.

A Worked Scenario (Hypothetical)

A lending company wants a model to predict which customers are likely to miss a payment. The FDE receives an extract of 200,000 customer records from three systems.

Profiling shows that the income column is mostly empty for one source, phone numbers come in four formats, and about a tenth of customers appear twice with slightly different names. The data owner explains that the value 0 in the "late payments" column means "not yet recorded", not "none".

The FDE converts dates with explicit formats, standardises phone numbers, replaces the 0 sentinel with a missing flag, and links duplicates using phone plus name similarity. Several hundred uncertain pairs go to a credit analyst. Splitting data by customer ID after resolution avoids leakage. A label check finds that "defaulted" was set by a batch job that also flagged accounts closed early, so the team redefines the label with the business. Automated checks are added to the daily load.

The model accuracy gain is modest. The bigger win is that the business now trusts the numbers. This scenario is illustrative, not a real case.

Common Mistakes

• Cleaning before profiling. You fix the wrong things.
• Dropping rows to make errors vanish. You may bias the sample.
• Over-merging records. Keep merges reversible.
• Splitting before deduplicating. Leakage inflates your results.
• No cleaning log. Nobody can explain the data later.

Skills to Build

If you come from a data background, our article on moving from data engineer to forward deployed engineer shows how these skills transfer, and FDE vs data scientist explains where the roles differ. For the full skills picture, read what technical skills an FDE needs, and for how this fits a delivery, see how FDEs help enterprises deploy AI. If you are weighing approaches, fine-tuning vs RAG shows how cleaning needs change by method. When data readiness is the real blocker, why AI projects fail explains the pattern.

TL;DR:

‍ Data cleaning for AI projects is the work of turning raw, inconsistent records into data a model can learn from and a business can trust. Forward deployed engineers (FDEs) do it in a fixed order: profile the data first, fix types and formats, standardise values, handle missing data, resolve duplicates and entities, check outliers and labels, then add automated validation that keeps running after launch. The goal is not perfect data. It is data that is good enough for the specific decision the AI will support, with every cleaning rule written down and every uncertain case reviewed by someone who knows the business.

‍

‍