Industrial companies looking to leverage AI do not have a data shortage, they have a shortage of data they can use effectively. Plants collect millions of readings a day covering flow, temperature, pressure, vibration and process state. They then invest heavily in moving that information into Snowflake, Databricks, Azure, AWS and other data-lake environments so analysts and AI teams can finally work with it.
But moving data into the cloud does not automatically make it usable. Frozen tags, missing periods, implausible sensor values and communication failures can derail even the most perfectly tuned AI initiatives.
Time-series data remediation is the process of correcting or replacing bad measurements so the result can be used for reporting, analysis or AI model training. It covers gaps where there are no values, readings that are physically implausible, frozen signals, and values that disagree between linked systems. Remediation produces a corrected copy of the dataset, and it does so without altering the historian.
This article covers what remediation can and cannot recover, why filtering is often a flawed answer to bad data, the methods available for remediation, and the conditions for doing it safely.
Why Detection on Its Own is Not Enough Anymore
Industrial data teams now have the tools to accurately detect, isolate and triage data quality problems. Detection tells you where and why the data cannot be trusted as it stands, which is essential, but it does not answer the question that comes next: What should happen to that data before it reaches an AI model, a report, a digital twin or a lakehouse.
High-impact use cases for data remediation include:
- preparing a training set for an AI model that will run on historian data
- feeding a reporting pipeline that produces figures other people act on
- migrating historian data into a data lake where downstream consumers will not have the original context
In each case someone has to decide what happens to the bad data points, and automated remediation turns that from weeks of manual preparation into a rule that runs against thousands of channels unattended.
Data Quality Problems Often Can’t Be Fixed at the Source
The decision would be simpler if the underlying data problems could be corrected where they originate, but two constraints usually prevent this.
- Ownership is the first. The team that consumes the data is often not the team that owns the instrument or the historian configuration, so a fix at source depends on another group’s priorities and schedule.
- Time is the second. A calibration fault from last March cannot be recalibrated retrospectively, so the readings it produced are the only readings that exist for that period.
Filtering Does Not Clean Your Dataset – It Shrinks It
Consider a team building an AI model for a specific pump or production line. They have three years of historian data, which sounds like an ideal training set until the filtering starts, because a flow meter flatlines intermittently, a pressure transmitter reports impossible values, and two systems disagree over several periods.
Excluding every affected window leaves a few fragmented months, often without enough representative operating conditions to build an AI model that holds up. Conversely, keeping the bad values leaves a model that learns a frozen reading as steady process behavior. Often, neither failure is visible to whoever uses the dataset, because there is no record of what was removed or replaced.
Remediation preserves the source, corrects only what can safely be made usable, and records every change. Data scientists can then see which values are real measurements and how much of the training window was reconstructed, which lets them judge how far to trust a model trained on it.
With the release of Aperio DataWise 4.5, industrial data teams can remediate anomalous data at scale, which means producing a corrected copy of the data while being explicit about what has been changed.
The Three Rules for Safe Remediation
When a data channel stops reporting for a period, any remediated value is an inference. That is acceptable, and often very useful, provided three conditions are met.
- The source data is never modified. Remediation should produce a separate copy so the historian continues to hold the record of what was actually reported, including its faults. Anyone who needs to audit the original dataset can still do so, and a remediation decision made for one purpose does not affect every other consumer of the same data.
- Every changed point is labeled. A reconstructed value that reaches a model or a report indistinguishable from a real reading is a serious problem, because the consumer has no way to weight it differently or to exclude it. Reconstruction itself is manageable, provided the reconstructed points are identified as such and whoever builds the model knows how much of the training window they represent.
- Every change traces back to a stored rule. If someone asks six months later why a value in the dataset differs from the historian, you need to be able to reproduce the transformation exactly, because the answer has to be defensible to a third party.
A remediated dataset that meets those three conditions can be used for model training with confidence, with a clear view of its contents and the reasoning behind each change.
Figure 2: Saved rules covering the same channels. Each one targets a different combination of data quality issues, so a rule built for a report and a rule built for a condition model can run against the same source without one overwriting the other.
How Remediation Works in Aperio DataWise
DataWise creates cleaned copies of your data without modifying the source historian, and the capability is built from two objects.
- Rules define which data quality issues to remediate and how. A rule holds a name, the issues it acts on, a severity range for each of them (on a zero to ten scale), the method to apply, and optionally a set of value ranges. Rules are reusable, so the same definition can be applied repeatedly across different channels and periods.
- Jobs apply a rule. A job runs one rule against a selected scope of channels over a specific time window, reports its progress as it goes, and produces the remediated output. The jobs list shows the status of each run, the number of channels involved, and the count of points remediated against the total.
Figure 3: A completed job across 1,552 channels, with 15.2 million points modified out of nearly three billion. Each channel can be opened to compare the original signal against the remediated version, with the underlying events listed, so the output can be checked before the data is used.
Opening a completed job shows each channel with the original signal and its detected events alongside the remediated version with the remediation marked, so you can see exactly what changed before you use the result. The remediated data can then be exported as a file or ingested directly by a downstream application or model.
None of this has to be worked out alone. Our team takes customers through these decisions across a range of process environments and use cases, so we can help you identify the channels most worth remediating, set severity thresholds that match what you are building, and structure your rules so that every dataset you produce is defensible.
Four Ways to Remediate a Flagged Measurement
DataWise offers four methods, and the choice depends on what the signal was doing and what the downstream consumer needs.
- Substitution (numeric value, string value) replaces the flagged point with a fixed number or string you set in advance. Use it when the true value is unrecoverable and you want the gap visible in the data, or when a constant is genuinely correct, such as zero flow on an isolated line. Whatever number you choose gets included in any downstream average or total, so pick one that fails obviously rather than one that blends in.
- Interpolation (linear) draws a straight line between the last good value before the gap and the first good value after it. Use it for short gaps in signals that move slowly, where the straight line is close to whatever the process actually did. The longer the gap, the weaker that assumption gets.
- Reconstruction (synthesize) builds replacement values from how the channel normally behaves, so the fill can follow the signal’s usual shape instead of a straight line. Use it for longer gaps, and for signals that cycle or vary enough that a straight line would look obviously wrong. The result is still an estimate, and downstream consumers need to know which points it produced.
- Removal drops the flagged measurement from the output, taking the timestamp with it, so the series is no longer evenly spaced. Use it when an incomplete series is acceptable and a substituted value would be worse than nothing. Unlike filtering by hand, the removal is governed by a stored rule, the dropped points are recorded, and the original stays in the historian.
To see DataWise remediation in action, follow this link to book a demo.
Frequently Asked Questions:
Time-series data remediation is the process of correcting or replacing bad measurements in sensor and historian data so the result can be used for reporting, analysis, or model training. It handles gaps, frozen signals, implausible readings, and values that disagree between systems, and it produces a corrected dataset rather than only a list of problems.
The terms overlap, and in practice, cleansing often means filtering out bad records, while remediation means replacing them with something usable. For time-series data, filtering leaves holes in a series where downstream consumers expect continuity, whereas remediation keeps the series intact and records which points were changed.
Interpolation is reasonable for short gaps in a signal that changes slowly, and it becomes unreliable as gaps lengthen or as the process becomes more variable, because it assumes the signal moved smoothly across a period when it may not have. The safer practice is to interpolate only where that assumption holds, and to label every interpolated point.
It is, provided the cleaning happens in a copy rather than in the historian itself. The original record stays available for audit, the changed points are labeled so consumers know what they are working with, and a decision made for one use case does not propagate to everyone else reading the same tags.
Remediated points should be labelled in the output so that anyone using the dataset can tell them apart from original readings. In DataWise, each job also shows the original channel with its detected events alongside the remediated channel with the remediation marked, so the changes can be reviewed before the data is used.
Yes it absolutely can be, provided the reconstructed points are identified as such and whoever builds the model has context for what has been remediated. The potential risk comes from reconstruction that arrives unlabeled, because the model then treats an infrence with the same confidence as a measurement.
There is no general figure, because it depends on how many instruments are affected and how the faults are distributed across the period. What matters more than the total is where the data losses occur, since a fault affecting one instrument during a plant startup can remove the operating conditions a model most needs to learn, even when the overall share of discarded data is small.