Operational Data
Getting Value Out of SCADA and Historian Data
The data is already being collected. It is sitting in a historian that engineering can query and nobody else can.
We move production and asset data out of the control layer and into a place where finance, planning and operations can all read the same numbers, without touching the control network.
The actual problem
A historian is an excellent archive and a poor reporting system
It was built to answer a different question than the one the business is now asking.
A process historian is very good at what it was designed for: capturing a large number of tags at high frequency, compressing them, and giving a control engineer a trend to look at. That design does not make it a reporting database, and the gap shows the moment someone outside operations wants a number from it.
The symptoms are familiar. Production reporting is a spreadsheet somebody rebuilds every morning from a historian export. Finance and operations quote different volumes for the same period because one is reading the historian and the other is reading the accounting system. A question that spans wells, ERP and the maintenance system takes two days because it takes three people. And nobody can answer anything about last year without asking the one engineer who knows the tag naming convention.
None of this is a failure of the historian. It is what happens when operational data stays where it was captured, and the reporting need grows past what that place was built to serve.
How the work runs
Out of the control layer, without touching the control layer
The security question is the first one asked in every one of these engagements, so it is answered first here too.
Step 1
One-way egress, agreed with the OT team
Data leaves through a read-only path the OT and instrumentation team specifies and controls: commonly a historian replica, a data diode or a broker in the DMZ. Nothing we build writes back toward the control network, and the OT team keeps the ability to shut the path off without our involvement.
Step 2
Tags become things a business recognises
A tag name is a control-system identifier, not a business concept. We map tags to wells, pads, units and assets, attach the units of measure explicitly, and record which tag is authoritative when two disagree. This mapping is the asset. It is what makes every later question answerable.
Step 3
One place the numbers come from
Time-series lands in a cloud warehouse alongside production reporting, ERP and maintenance data, at a grain that supports both the daily operational view and the monthly financial one. Reports move onto it and the old ones get retired, because otherwise you now have three.
What it makes possible
Questions that become answerable
Each of these is a question operators already ask, and currently answer by hand or not at all.
Production surveillance that is current
Well and pad performance against expectation, refreshed on a schedule rather than rebuilt each morning. The value is less in the dashboard than in everyone reading the same figure at the same time.
Deferment that is measured, not estimated
Lost production attributed to a cause and a duration rather than reconciled at month end from memory. This is usually the first number that changes a conversation, because the total is almost always larger than the estimate it replaces.
Rotating equipment behaviour over time
Vibration, temperature and pressure history against failure and intervention records. Long histories are worth far more than high frequency here, which is exactly what a historian already has.
Field data beside financial data
Volumes, costs and interventions in one place, at one grain. This is the join that makes cost per barrel answerable without a reconciliation meeting.
Where things belong
Historian or warehouse
This is not a replacement. The two do different jobs, and the common mistake is asking one of them to do the other’s.
| Process historian | Cloud data warehouse | |
|---|---|---|
| Built for | High-frequency capture and compression of many tags. | Joining across sources and answering business questions. |
| Best at | Sub-second trends, control-room use, long retention of raw signal. | Aggregates, comparisons across assets, and history beside ERP data. |
| Who queries it | Control and process engineers who know the tag names. | Anyone, using well, asset and period names they recognise. |
| Joins to ERP and maintenance data | Awkward, and usually done by export. | The reason it exists. |
| Should it be replaced | No. It stays the system of record for raw signal. | It is where the reporting moves to. |
Being straight about it
Where this does not work
If the tag naming has no convention at all and nobody remaining at the company knows what the tags mean, the mapping step becomes archaeology and takes considerably longer than the rest of the project. This is survivable and worth doing, but it should be scoped honestly rather than discovered.
If the historian is only retaining a few months at full resolution, anything that depends on comparing behaviour to a failure two years ago is not available yet, and the first useful action is to change the retention policy and wait.
And if the OT team has not agreed the egress path, nothing above starts. That is the correct order. An analytics project that routes around the people responsible for the control network is a project that gets stopped later, at greater cost, and rightly.
Questions we get asked
Before you commit to anything
No. Data leaves through a read-only path that your OT team specifies and controls, and nothing we build writes back toward the control layer. Where a client has no existing egress path, the first piece of work is agreeing one with OT and security, before any analytics work is scoped.
No. PI stays as the system of record for raw signal. It is very good at that, and nothing here is. What we do is copy the tags that matter into a warehouse where they can sit beside ERP, maintenance and production reporting data and be queried by people who do not know tag names.
Far fewer than most operators expect. A useful production surveillance and deferment picture typically comes from a small fraction of the available tags. Starting with every tag is the most common way these projects get slow and expensive without getting more useful.
It is the main variable in the timeline, not a blocker. The mapping from tag to asset has to be built and validated regardless; poor naming means it takes longer and needs more time from someone who knows the field. We would rather size that honestly at the start than discover it in week three.
That is usually the right way to do it. One field or one facility, one clearly defined question, and a result the operations team agrees with before anything is extended. The mapping work done at the first site is most of what makes the second one fast.
Remote and intermittently connected sites need the pipeline designed for gaps and late-arriving data rather than assumed to be online, which changes how the ingestion and the reconciliation work. It is a normal condition in this sector, not an exception, and it is better designed for at the start.
Where to next
Related reading
Oil & Gas Data and AI Consulting
The wider picture: what data and AI work looks like across an upstream or midstream operator.
Oil and Gas Playbook
Six use cases with the data each one needs, how you would know it worked, and where it fails.
Data Engineering Services
The pipelines, warehouse and governance underneath all of this.