Senior Data Engineer: Scalable Pipelines & Data Quality

Government Recruitment Service; Greater London; On-site; GBP 60,000 - 80,000; Full time

Job summary

A government agency is seeking a data engineer to design and operate ETL and streaming pipelines for large datasets. The role involves ensuring data quality and compliance with standards while mentoring junior engineers. You'll work with analysts and product teams to create reusable data products in the United Kingdom. Strong knowledge of data quality standards and experience in automation using CI/CD are essential for success in this position.

Qualifications

• Experience in building and operating ETL and streaming pipelines for large datasets.
• Strong knowledge of data modelling, lineage, and quality standards.
• Familiarity with cloud and on-premises platforms.

Responsibilities

• Design, build and operate ETL and streaming pipelines for varied datasets.
• Implement data quality and standards compliance.
• Integrate data from APIs and batch methods.
• Mentor junior engineers and enhance documentation.

Skills

ETL/streaming solutions

Data modelling

Data quality standards

CI/CD tools

Data integration

Job description

A government agency is seeking a data engineer to design and operate ETL and streaming pipelines for large datasets. The role involves ensuring data quality and compliance with standards while mentoring junior engineers. You'll work with analysts and product teams to create reusable data products in the United Kingdom. Strong knowledge of data quality standards and experience in automation using CI/CD are essential for success in this position.