Senior DataOps Engineer: Platform Reliability & Automation

Wwshemi; Greater London; Hybrid; GBP 90,000 - 120,000; Full time

Benefits offered by this job

Equity

Competitive salary

Employer pension contributions

Private health insurance

Hybrid work setup

Gym memberships

Company retreats

Job summary

Valarian Technologies Limited is expanding its London hybrid team to build and operate data services and processing workloads for mission-critical workflows. You will work with data engineers and platform engineers across customer-controlled environments to ensure reliability and recoverability of data pipelines.

The role emphasizes hands-on Kubernetes, Linux, and Python tooling, with strong CI/CD and GitOps practices.

Qualifications

• Demonstrable experience operating production data pipelines or data platforms, with ownership of reliability and incident resolution.
• Strong practical Kubernetes, container and Linux skills, including troubleshooting workloads, networking, resource limits and persistent storage.
• Experience building automated delivery workflows using Git, CI/CD and declarative configuration; familiarity with Helm, GitOps or equivalent approaches.
• Python and shell scripting for automation, plus sufficient SQL to investigate processing failures and database performance.
• Experience with workflow orchestration, such as Airflow, Dagster, Prefect or an equivalent system.
• Understanding of reliable processing patterns: idempotency, bounded retries, duplicate delivery, incremental processing and safe replay.
• Experience implementing metrics, logs, dashboards and alerting, with meaningful service and data reliability targets.
• Practical experience with stateful systems, controlled migrations, backup and recovery, secrets management and least-privilege access.

Responsibilities

• Build and maintain CI/CD and GitOps workflows for data applications, pipelines, configuration and database changes, including validation, promotion and rollback.
• Develop reusable tooling for scheduling, dependency management, retries, checkpoints, replay and backfills.
• Establish monitoring and actionable alerts for pipeline freshness, completeness, processing latency, failures and resource usage.
• Operate workload-owned data services and coordinate with ACRA platform owners for shared database, messaging and storage services.
• Implement and exercise recovery procedures, including backup restoration, interrupted processing and failed upgrades.
• Integrate data workloads with ACRA’s identity, secrets and network controls, preserving enclave isolation and authorised access.
• Investigate production incidents, improve runbooks and automate recurring operational work.

Skills

Kubernetes

Linux

Python

Shell scripting

SQL

CI/CD

GitOps

Airflow

Dagster

OpenShift

Tools

Helm

Git

Airflow

Dagster

PostgreSQL

Prometheus

Grafana

Job description

Valarian Technologies Limited is expanding its London hybrid team to build and operate data services and processing workloads for mission-critical workflows. You will work with data engineers and platform engineers across customer-controlled environments to ensure reliability and recoverability of data pipelines.

The role emphasizes hands-on Kubernetes, Linux, and Python tooling, with strong CI/CD and GitOps practices.