Senior Data Engineer

IO Associates; England; On-site; GBP 60,000 - 80,000; Full time

Job summary

A leading tech recruitment company is seeking a detail-oriented Senior Data Engineer to join their team in the United Kingdom. The successful candidate will be responsible for building and maintaining robust data pipelines and applications, ensuring that data-driven projects are completed effectively. Candidates must hold active DV security clearance and demonstrate strong proficiency in Python, Linux systems, and AWS serverless technologies. This is a fully on-site, permanent role that involves collaborating with cross-functional teams in a high-security environment.

Qualifications

• Strong proficiency in Python for data engineering.
• Experience with Linux systems administration and performance optimization.
• Hands-on with Python orchestration frameworks like Dagster or Airflow.
• Familiarity with serverless AWS services.
• Knowledge of security and compliance standards in high-security environments.

Responsibilities

• Design and manage data ingestion pipelines.
• Develop end-to-end bespoke data projects.
• Monitor performance across Linux infrastructure.
• Implement scalable data processing solutions.
• Drive improvements in pipeline reliability.

Skills

Python proficiency

Linux systems administration

Docker containerization

Kubernetes familiarity

Knowledge of security protocols

Tools

Dagster

Apache Airflow

AWS

Job description

Senior Data Engineer (DV Cleared)Fully On-Site | Permanent

Note: Candidates must hold active DV (Developed Vetting) security clearance. This is a fully on-site role.

The Role

We're seeking a detail-oriented, analytical Senior Data Engineer to join a high-performing multi-disciplinary data team. You'll be essential to building and maintaining robust data pipelines and bespoke applications, ensuring data-driven projects deliver against demanding customer timelines and requirements.

Core Responsibilities

Pipeline Engineering:

• Design, manage, and optimise data ingestion pipelines and orchestration workflows

• Develop and maintain end-to-end bespoke data projects from requirements to production

• Monitor system performance and ensure high uptime across Linux-based infrastructure

• Collaborate closely with data scientists, analysts, product owners, and engineering teams

Technical Delivery:

• Implement scalable data processing solutions handling complex, high-volume datasets

• Drive continuous improvement in pipeline reliability, performance, and observability

• Support cross-functional delivery of mission-critical intelligence platforms

What We're Looking For

Core Technical Expertise:

• Strong Python proficiency for data engineering and pipeline development

• Linux systems administration and performance optimisation experience

• Hands-on with Python orchestration frameworks (Dagster, Apache Airflow)

Cloud & Infrastructure:

• Serverless AWS experience (Lambda, Step Functions, Athena, Glue)

• Docker containerisation - building/maintaining production-grade containers

• Kubernetes familiarity for container orchestration and scaling

Security & Compliance:

• Knowledge of security, encryption, and compliance standards for secure networks

• Experience delivering data platforms in high-security environments