Senior Data Platform Engineer - Compliance & Data Quality

Pontoon; Chester; On-site; GBP 52,000 - 78,000; Full time

Job summary

Pontoon is seeking a hands-on Senior Data Platform Engineer to design, build and maintain scalable data platforms for Compliance Assurance in a regulated environment. You will own the end-to-end data engineering lifecycle, from source ingestion through streaming, processing, storage and automated controls.

This developer-led role requires strong Python/PySpark, Kafka and NiFi skills, experience with Hadoop/Hive, SQL and REST APIs, plus production reliability and collaboration with risk teams.

Qualifications

• Proven experience building data pipelines with Python and PySpark.

Responsibilities

• Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi.
• Integrate vendor platforms via REST APIs with secure authentication.
• Build data processing and storage solutions using Spark, Hadoop and Hive.
• Develop automated reconciliation, data quality and compliance controls.
• Implement robust data flows with retry mechanisms, error handling, replay, recovery and monitoring.
• Establish canonical data models, data lineage, validation rules and reliable data structures.
• Develop automated tests to validate data accuracy, completeness, performance and processing reliability.
• Use Git, code reviews and CI/CD pipelines to manage code changes and promote solutions through Development, UAT and Production.
• Troubleshoot production issues, improve pipeline performance and establish operational monitoring and support procedures.
• Collaborate with architects, platform engineers, control owners, risk teams and vendors to deliver reusable engineering solutions.
• Explore AI-assisted development tools such as GitHub Copilot while maintaining appropriate security, quality and governance standards.

Skills

Python

PySpark

Apache Kafka

Apache NiFi

Hadoop

Hive

SQL

REST APIs

Git

CI/CD

Data quality

Data reconciliation

Troubleshooting

Communication

Tools

Azure

GitHub Copilot

Jenkins

Job description

Pontoon is seeking a hands-on Senior Data Platform Engineer to design, build and maintain scalable data platforms for Compliance Assurance in a regulated environment. You will own the end-to-end data engineering lifecycle, from source ingestion through streaming, processing, storage and automated controls.

This developer-led role requires strong Python/PySpark, Kafka and NiFi skills, experience with Hadoop/Hive, SQL and REST APIs, plus production reliability and collaboration with risk teams.