Senior Data Engineer: Pipelines, PySpark, SC Clearance

Mayfair IT Consultancy Ltd

Greater London

On-site

GBP 60,000 - 120,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Mayfair IT Consultancy Ltd is recruiting Data Engineers for a major UK Government data programme. You will design, build and maintain robust data pipelines using Python and SQL to support large-scale data processing and quality control.

In this role you will transform datasets, ensure data quality, and deliver reusable engineering solutions while applying Git, automated testing and CI/CD practices. SC Security Clearance is required.

Qualifications

  • Strong practical Python programming experience.
  • SQL experience for data engineering and transformation.
  • Experience building, testing and maintaining data pipelines or complex scripts.
  • Data cleansing, preparation, transformation and quality validation.
  • Git, version control and collaborative engineering practices.
  • Code reviews, automated testing and technical documentation.
  • Ability to investigate, resolve and prevent technical issues.
  • Strong problem-solving and communication skills.
  • Ability to work independently and within multidisciplinary teams.

Responsibilities

  • Design, develop and maintain data pipelines, frameworks and systems.
  • Write clear, reliable and well-tested Python code.
  • Use SQL to prepare, clean, transform and validate data at scale.
  • Investigate data quality issues and resolve technical problems.
  • Develop reusable components and improve engineering processes.
  • Apply Git, code reviews, automated testing and CI/CD practices.
  • Document solutions and follow agreed engineering standards.
  • Collaborate with technical and non-technical stakeholders.
  • Identify opportunities for automation and continuous improvement.

Skills

Strong problem-solving
Communication
Independence
Team collaboration
Attention to detail

Tools

Python
SQL
ETL/ELT pipeline development
PySpark
Cloudera
GCP
CI/CD
Git
Automated testing

Job description

Mayfair IT Consultancy Ltd is recruiting Data Engineers for a major UK Government data programme. You will design, build and maintain robust data pipelines using Python and SQL to support large-scale data processing and quality control.

In this role you will transform datasets, ensure data quality, and deliver reusable engineering solutions while applying Git, automated testing and CI/CD practices. SC Security Clearance is required.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer: AI Pipelines & Streaming (SC Cleared)
Senior Data Engineer: AI Pipelines & Streaming (SC Cleared)

ByteHire Limited. • United Kingdom

Remote
GBP 90,000 - 130,000
Data Engineer
Data Engineer

Mayfair IT Consultancy Ltd • Greater London

On-site
GBP 60,000 - 120,000
Senior Data Engineer
Senior Data Engineer

ByteHire Limited. • United Kingdom

Remote
GBP 90,000 - 130,000
Senior Data Engineer - Secure, Scalable Pipelines (Remote)
Senior Data Engineer - Secure, Scalable Pipelines (Remote)

NPAworldwide • Birmingham

Hybrid
GBP 50,000 - 70,000
Flexible working
Hybrid/remote working
Learning & Development
+5
Senior Data Engineer for Secure Govt Data Pipelines (Cloud)
Senior Data Engineer for Secure Govt Data Pipelines (Cloud)

NPAworldwide • Manchester

Hybrid
GBP 50,000 - 70,000
Flexible working
Hybrid/remote working
Learning & Development
+5
Data Engineer
Data Engineer

Datacareers • City Of London

On-site
GBP 70,000 - 110,000
Senior Data Engineer — Pipelines, Lakes & AI Enablement (Remote)
Senior Data Engineer — Pipelines, Lakes & AI Enablement (Remote)

Scott Logic • West of England

Hybrid
GBP 65,000 - 95,000
25 days annual leave
Pension scheme
Life Assurance
+2
Data Engineer — Secure ETL Pipelines (London Hybrid)
Data Engineer — Secure ETL Pipelines (London Hybrid)

Marks and Spencer • City Of London

Hybrid
GBP 65,000 - 90,000
Data Engineer
Data Engineer

G Digital • Greater London

Hybrid
GBP 36,000 - 60,000
Employer pension 5% match
Training budget
Hybrid work model
Data Engineer - SC Cleared
Data Engineer - SC Cleared

Lorien • Greater London

Hybrid
GBP 92,000 - 138,000