Python Data Engineer - ETL & Spark | SC/DV Clearance

Jobtailor; Greater London; On-site; GBP 70,000 - 100,000; Full time

Job summary

Jobtailor is seeking a data-focused Python engineer to design, build, and maintain high-quality data processing solutions in a secure environment. You will develop scalable ETL/ELT pipelines, handle large datasets, and collaborate with cross-functional teams to drive architectural decisions.

You will implement robust testing, monitoring and logging, ensure data quality and integrity, and contribute to data platforms while adhering to governance requirements.

Qualifications

• Extensive commercial experience building Python applications and services.
• Strong data engineering background with scalable ETL/ELT pipelines.
• Proficient in SQL and handling large datasets.
• Experience with PySpark, Spark, and Pandas.
• Familiar with Git, CI/CD and automated testing.
• Comfortable Linux environments and secure development practices.
• Active UK Security Clearance (SC/DV) is mandatory.

Responsibilities

• Design, develop and maintain Python applications, services and data processing solutions.
• Build and optimise scalable ETL/ELT pipelines for ingestion, transformation and delivery of data.
• Develop robust data processing workflows using Python and modern data engineering technologies.
• Work with large and complex datasets, ensuring data quality, integrity, availability and performance.
• Integrate data from APIs, databases, files and streaming systems.
• Develop reusable, well-tested and maintainable software components.
• Design and implement automated testing, monitoring, logging and error-handling mechanisms.
• Collaborate with data engineers, software engineers, architects and other technical stakeholders.
• Contribute to the design and evolution of data platforms, pipelines and supporting infrastructure.
• Improve system performance, scalability, reliability and automation.
• Participate in code reviews and contribute to engineering standards and best practices.
• Work within secure development environments and adhere to security, governance and compliance requirements.
• Produce technical documentation and contribute to architectural and engineering decisions.

Skills

Python programming

Data engineering

SQL proficiency

Git & CI/CD

Linux & secure envs

Production-grade delivery

Stakeholder collaboration

Security awareness

Education

Relevant degree or equivalent experience

Tools

PySpark

Spark

Pandas

Job description

Jobtailor is seeking a data-focused Python engineer to design, build, and maintain high-quality data processing solutions in a secure environment. You will develop scalable ETL/ELT pipelines, handle large datasets, and collaborate with cross-functional teams to drive architectural decisions.

You will implement robust testing, monitoring and logging, ensure data quality and integrity, and contribute to data platforms while adhering to governance requirements.