Overview
We are seeking experienced Data Engineers to join a major UK Government data programme, developing and maintaining reliable data pipelines and engineering solutions.
You will use Python and SQL to build, test and improve data processes, prepare and transform datasets, maintain data quality and deliver high-quality, reusable engineering solutions.
Key Responsibilities
- Design, develop and maintain data pipelines, frameworks and systems.
- Write clear, reliable and well-tested Python code.
- Use SQL to prepare, clean, transform and validate data at scale.
- Investigate data quality issues and resolve technical problems.
- Develop reusable components and improve engineering processes.
- Apply Git, code reviews, automated testing and CI/CD practices.
- Document solutions and follow agreed engineering standards.
- Collaborate with technical and non-technical stakeholders.
- Identify opportunities for automation and continuous improvement.
Essential Skills & Experience
- Strong practical Python programming experience.
- SQL experience for data engineering and transformation.
- Experience building, testing and maintaining data pipelines or complex scripts.
- Data cleansing, preparation, transformation and quality validation.
- Git, version control and collaborative engineering practices.
- Code reviews, automated testing and technical documentation.
- Ability to investigate, resolve and prevent technical issues.
- Strong problem-solving and communication skills.
- Ability to work independently and within multidisciplinary teams.
Technical Skills & Technologies
- Python and SQL
- ETL / ELT pipeline development
- PySpark and large-scale data processing
- Cloudera or comparable enterprise data platforms
- GCP or equivalent cloud environments
- Data profiling, cleansing and quality management
- Reusable Python packages and engineering frameworks
- CI/CD, Git and automated testing
- Data integration and pipeline optimisation
Comparable enterprise technologies and transferable experience are welcome.
Seniority & Delivery Expectations
- Develop reliable, scalable and maintainable data solutions.
- Produce high-quality, tested code.
- Apply sound technical judgement and resolve problems proactively.
- Follow engineering standards and contribute to improvements.
- Communicate effectively and share knowledge.
Desirable
- UK Government or public-sector experience.
- Experience with large-scale or complex datasets.
- Strong PySpark, Cloudera or GCP experience.
- Reusable Python package development.
- Data governance, security and compliance knowledge.
Clearance Requirement
SC Security Clearance is required for this assignment.
Candidates should be able to confirm their current security clearance status and meet the required screening and onboarding requirements.