Cloud-Native Real-Time Data Platform Engineer

Simple Machines; Greater London; On-site; GBP 90,000 - 120,000; Full time

Job summary

Simple Machines is a global, independent technology consultancy that designs and builds modern data platforms, intelligent systems, and bespoke software across London and beyond. You will join senior engineers to own the architecture of cloud-native data platforms and help turn messy data into scalable products.

The role emphasises data mesh, data products and data contracts, with production-ready pipelines and secure, compliant platforms.

Qualifications

• Strong Python and SQL.
• Deep experience with Spark and modern data platforms (Databricks / Snowflake).
• Solid grasp of cloud data services (AWS or GCP).
• Demonstrated ownership of large-scale data platform architectures.
• Strong data modelling skills and architectural decision-making ability.
• Comfortable balancing trade-offs between performance, cost, and complexity.
• Built and operated large-scale data pipelines in production.
• Infrastructure-as-code experience (Terraform, Pulumi).
• CI/CD pipelines using GitHub Actions, ArgoCD.
• Data testing and quality frameworks (dbt, Great Expectations, Soda).
• Experience in consulting or professional services environments.
• Strong consulting instincts — able to challenge assumptions and guide clients toward better outcomes.
• Comfortable mentoring senior engineers and influencing technical culture.

Responsibilities

• Own end-to-end architecture of cloud-native data platforms.
• Design scalable data ecosystems using data mesh, data products, and data contracts.
• Make high-impact architectural decisions across ingestion, storage, processing, and access layers.
• Ensure platforms are secure, compliant, and production-grade by design.
• Design and deliver cloud-native data platforms using Databricks, Snowflake, AWS, and GCP.
• Apply modern architectural patterns: data mesh, data products, and data contracts.
• Integrate deeply with client systems to enable scalable, consumer-oriented data access.
• Build and optimise batch and real-time pipelines.
• Work with streaming and event-driven tech such as Kafka, Flink, Kinesis, Pub/Sub.
• Orchestrate workflows using Airflow, Dataflow, Glue.
• Process and transform large datasets using Spark and Flink.
• Design systems that perform in production - not just on paper.
• Work across relational, NoSQL, and analytical stores (Postgres, BigQuery, Snowflake, Cassandra, MongoDB).
• Optimise storage formats and access patterns (Parquet, Delta, ORC, Avro).
• Implement secure, compliant data solutions with security by design.
• Embed governance without killing developer velocity.
• Work directly with clients to understand problems and shape solutions.
• Translate business needs into pragmatic engineering decisions.
• Act as a trusted technical advisor, not just an order taker.
• Set engineering standards, patterns, and best practices across teams.
• Review designs and code, providing clear technical direction and mentorship.
• Raise the bar on data quality, testing, observability, and operational excellence.

Skills

Python

SQL

Spark

Data platforms

Data modelling

Terraform

CI/CD

dbt

Great Expectations

Tools

Databricks

Snowflake

AWS

GCP

Airflow

dbt

Great Expectations

Soda

Job description

Simple Machines is a global, independent technology consultancy that designs and builds modern data platforms, intelligent systems, and bespoke software across London and beyond. You will join senior engineers to own the architecture of cloud-native data platforms and help turn messy data into scalable products.

The role emphasises data mesh, data products and data contracts, with production-ready pipelines and secure, compliant platforms.