Energy AI Data Engineer — Real-Time Lakehouse

Applied Computing; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Job summary

Applied Computing is seeking a Data Engineer to architect and maintain pipelines turning high-frequency time-series, lab, and historian data into a scalable Lakehouse for deep learning and real-time LLM workloads.

You will work across AWS (EKS, S3, EBS, KMS, CloudWatch) and Databricks, ensuring data is contextualised, synchronised, and optimised for AI workloads while solving problems at the intersection of control systems, industrial data engineering, and AI enablement.

Qualifications

• Proficient in PostgreSQL including partitioning, indexing, and query optimisation.
• Strong Python skills for data processing, scripting, and orchestration.
• Hands-on AWS experience (EKS, S3, EBS, IAM, KMS, CloudWatch).
• Experience with Databricks and PySpark for large-scale processing.
• Familiar with time-series industrial data (DCS/SCADA, process historians).
• Experience with unstructured data sync in hybrid cloud/on-prem environments.

Responsibilities

• Ingest and contextualise data from OPC UA servers, process historians, sensors, alarms, and P&IDs.
• Build real-time and batch pipelines into a Lakehouse, enabling AI workloads.
• Orchestrate Databricks Lakeflow/Connectors for data integration into Lakebase/Lakehouse.
• Ensure data fidelity, schema evolution tracking, and data lineage across pipelines.
• Support AI workloads including time-series forecasting and retrieval-augmented LLMs.

Skills

PostgreSQL

Python

AWS

Databricks PySpark

Time-series data

DCS/SCADA logs

Hybrid cloud/on-prem

Tools

Kafka

Flink

Spark Streaming

MLOps stacks

Job description

Applied Computing is seeking a Data Engineer to architect and maintain pipelines turning high-frequency time-series, lab, and historian data into a scalable Lakehouse for deep learning and real-time LLM workloads.

You will work across AWS (EKS, S3, EBS, KMS, CloudWatch) and Databricks, ensuring data is contextualised, synchronised, and optimised for AI workloads while solving problems at the intersection of control systems, industrial data engineering, and AI enablement.