Process Historian vs Time-Series Database vs Data Lake for Plant Data

The word "data" is nothing new to industry. For a long time, the key decisions on a plant have come from live and historical data. Losing that data is more critical here than in almost any other sector, because it leads to wrong decisions, and wrong decisions end in lost production or degraded product quality. In the process industry, one of the most important disciplines is RCA, root cause analysis: it starts from an incident and drills all the way down to the underlying mistake, whether that mistake was made by a system or a human. None of that works without data you can trust.

Given how much rides on the data, how you log and store it matters as much as collecting it. As the data world has matured, plants have mainly three places to keep their data: the process historian, the time-series database, and the data lake. They aren't really competitors. They're answers to different questions, asked by different people. A historian answers an operator's question: what was tag FCV100.PV doing at 14:32 last Tuesday? A time-series database answers an engineer's question: what was the OEE (overall equipment effectiveness) across all four lines during a production run? A data lake answers a production analyst's question: here are five years of production data, let me find a pattern that improves quality.

So do you need all three? The answer is both yes and no. Most plants don't need to pick one. They need two of the three, and the whole skill is knowing which two, and what it costs to be wrong.

A note on who's answering this

One thing before the detail, because it should shape how you read the rest. I've spent years operating historians and building the pipelines that move plant data into modern stores, and the lesson that keeps repeating is that a data pipeline has to be designed around the goal it serves, not bolted on afterward. A central historian, in particular, is far more valuable than a single device's or a SCADA station's local historian, because it gives the whole plant one trusted record instead of many partial ones.

What a process historian is actually for

A process historian is a database built for one job: capturing every tag from the plant floor, at high frequency, effectively forever, without losing a sample. Think OSIsoft PI (now AVEVA PI), Wonderware, or the historian inside a SIMATIC or Honeywell stack. It stores data tag by tag, indexed by time, and its defining promise is that it never drops a sample. It connects to the OT (operational technology) world natively, over OPC UA or MQTT and device drivers, and it sits comfortably on the control network.

What it isn't good at is analysis across many tags, or joins with other data. Ask it the operator's question, what was this tag at this time, and it's unbeatable. Ask it the engineer's cross-plant question, and you start fighting the tool.

What a time-series database is actually for

A time-series database, such as InfluxDB or TimescaleDB, is built for a different question: aggregation across many series over time. It speaks SQL (or Flux), it has window functions, downsampling, and continuous aggregates, and it makes cross-tag, cross-line queries, the OEE-across-the-plant-this-week question, cheap and natural. It also lives comfortably in IT: cloud-hostable, familiar to any backend engineer, easy to wire to a dashboard.

The catch, and it's the part that's easy to underestimate, is the front door. A historian talks to the plant natively; a time-series database does not. You have to build and maintain the bridge that pulls tags off OPC UA and streams them in, and that bridge is a real system with real upkeep, sitting right on the OT/IT boundary. The database is the easy part. The connectivity is the part you'll still be maintaining in three years.

What a data lake is actually for

A data lake is the most flexible and the most dangerous of the three. At its simplest it's files, usually Parquet, on cheap object storage, queried on read with something like DuckDB or Spark. Its superpower is schema-on-read: you don't have to decide the shape of the data before you store it, so you can pool years of raw history, maintenance logs, quality records, and shift notes in one place and ask questions nobody anticipated at design time. That's exactly what a data scientist wants.

Its failure mode is just as real. With no schema and no governance, a data lake quietly becomes a data swamp: a vast store nobody trusts or can navigate. A lake rewards discipline and punishes its absence, which is the opposite of a historian's guarantees.

Process historian vs time-series database vs data lake: the comparison

So do I still need a process historian?

The honest answer to the question everyone actually asks, should I rip out my historian and replace it with a time-series database, is usually no. Not because the historian is better, but because you'd be solving the wrong problem. If your pain is that you can't do cross-plant analytics, the fix is to add an analytics layer, not to tear out the system of record your operators and auditors depend on.

Here's how I'd decide, by situation rather than by feature:

• Compliance-driven and single-site: keep the historian as the system of record. Add a time-series database or lake downstream if analytics demand it, but don't move the record.
• You need cross-plant OEE, KPIs, and dashboards: add a time-series database, fed from the historian or straight from OPC UA, and do your analytics there. Leave the historian doing what it's good at.
• Your data scientists want raw, joinable, multi-year history: that's a data lake, downstream of everything, never the primary capture.
• Greenfield, with modest compliance needs: you can often skip the historian entirely, capture to a time-series database, and archive to a lake. This is the one case where "replace the historian" really means "never buy one."

Where it goes wrong

Two failure patterns show up again and again.

The first is the swamp. A plant stands up a data lake because a corporate initiative said "get everything into the lake," pipes years of raw tags into Parquet with no naming discipline, no unified naming, and no owner, and eighteen months later has a multi-terabyte store everyone is afraid to query because nobody is sure what any of it means. The lake didn't fail; the governance did. A lake without unified naming and an owner is a liability, not an asset.

The second is the rip-out. A team, sold on a shiny time-series database, decommissions the historian, and only then discovers that the historian was never just a database. It was a database plus years of accreted OT connectivity: every driver, every tag mapping, and every quirk of every PLC, quietly maintained. They spent most of a year rebuilding that connectivity by hand before the new system matched what the old one did on day one. When you cost a migration, cost the connectivity, not the database.

The regulated-industry exception

There's one context where this isn't a free choice: regulated manufacturing. In pharma and food, data integrity isn't a feature, it's a legal requirement, and it has a name, ALCOA: data must be Attributable, Legible, Contemporaneous, Original, and Accurate. Auditors accept historian records because certified historians are built to satisfy exactly that: tamper-evident, time-stamped, with quality codes and an audit trail baked in.

The moment you move that data out, into a time-series database or a lake, you inherit the burden of proving the copy is as trustworthy as the original: no gaps, no silent transforms, a defensible chain of custody. I've sat on the validation side of this. It's doable, but it isn't free, and it's the reason regulated plants keep the historian as the system of record even when everything analytical happens downstream. If you're in a GMP environment, decide where your "original" lives before you decide anything else.

What most plants actually land on

Put it together and most plants don't land on one system. They land on a layered one: the historian (or a historian pattern) as trusted capture at the edge, a time-series database as the analytics and dashboard layer, and, if there are data scientists in the picture, a lake as the deep archive. Data flows one way, from floor to store to analysis, and each layer does the job it's genuinely good at. That's the same OT to cloud architecture I've written about elsewhere, viewed from the storage end. I've shown one lightweight version of the capture layer in monitoring plant data without an expensive historian.

One regional note, because it changes the shape for some of you. In the EU, data sovereignty and on-premise constraints are real, and the governance pressure building around the EU AI Act pushes teams to keep provenance and lineage on any data that will feed a model. That tends to favor on-prem or private-cloud versions of all three layers, and it makes the historian's built-in audit trail more valuable, not less. If your plant data might one day train or ground a model under EU rules, the boring, well-governed capture layer is the thing that saves you later.

The bottom line

None of these three is the winner, because the question was never which one. It's which two, in what order, for whose questions, and what it costs you if you guess wrong. Get that right and the technology almost picks itself.