Senior Data Platform Engineer - Compliance Assurance

Pontoon; Chester; Sur place; GBP 52 000 - 78 000; Plein temps

Résumé du poste

Pontoon is seeking a hands-on Senior Data Platform Engineer to design, build and maintain scalable data platforms for Compliance Assurance in a regulated environment. You will own the end-to-end data engineering lifecycle, from source ingestion through streaming, processing, storage and automated controls.

This developer-led role requires strong Python/PySpark, Kafka and NiFi skills, experience with Hadoop/Hive, SQL and REST APIs, plus production reliability and collaboration with risk teams.

Qualifications

• Proven experience building data pipelines with Python and PySpark.

Responsabilités

• Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi.
• Integrate vendor platforms via REST APIs with secure authentication.
• Build data processing and storage solutions using Spark, Hadoop and Hive.
• Develop automated reconciliation, data quality and compliance controls.
• Implement robust data flows with retry mechanisms, error handling, replay, recovery and monitoring.
• Establish canonical data models, data lineage, validation rules and reliable data structures.
• Develop automated tests to validate data accuracy, completeness, performance and processing reliability.
• Use Git, code reviews and CI/CD pipelines to manage code changes and promote solutions through Development, UAT and Production.
• Troubleshoot production issues, improve pipeline performance and establish operational monitoring and support procedures.
• Collaborate with architects, platform engineers, control owners, risk teams and vendors to deliver reusable engineering solutions.
• Explore AI-assisted development tools such as GitHub Copilot while maintaining appropriate security, quality and governance standards.

Connaissances

Python

PySpark

Apache Kafka

Apache NiFi

Hadoop

Hive

SQL

REST APIs

Git

CI/CD

Data quality

Data reconciliation

Troubleshooting

Communication

Outils

Azure

GitHub Copilot

Jenkins

Description du poste

Job Title: Senior Data Platform Engineer - Compliance Assurance

Contract: 12 Months

Location: Chester -3 days per week onsite

Rate: Highly Competitive

Status: Inside IR35

Working pattern: Fulltime

About the role

We are looking for a hands-on Senior Data Platform Engineer to design, build and maintain scalable data platforms that support Compliance Assurance within a regulated enterprise environment.

You will take ownership of the end-to-end data engineering lifecycle, from integrating source systems and ingesting data through to streaming, processing, storage and automated compliance controls. You will develop reliable, production-grade solutions that identify missing records, data mismatches, configuration issues and other exceptions, ensuring data quality, traceability and auditability.

This is a developer-led engineering role, ideal for someone who combines strong data engineering skills with software development expertise and a focus on production reliability.

Essential skills and experience

• Strong hands-on experience developing data engineering solutions using Python and PySpark.
• Practical experience with Apache Kafka and Apache NiFi for streaming, ingestion and orchestration.
• Experience working with Hadoop, Hive or comparable enterprise data-lake technologies.
• Solid knowledge of SQL, data transformation, data modelling and large-scale data processing.
• Experience integrating REST APIs, including authentication, pagination, rate limiting and error handling.
• Understanding of data reconciliation, data quality validation, exception handling and audit evidence.
• Experience using Git, automated testing, CI/CD and controlled software release processes.
• Ability to troubleshoot distributed data pipelines and support production systems.
• Strong analytical, problem-solving and communication skills.

Desirable experience

• Experience in financial services or another regulated enterprise environment.
• Exposure to voice, recording or collaboration platforms such as Microsoft Teams, Cisco, Zoom or NICE.
• Experience with Microsoft Azure, Microsoft Graph, Entra ID or service principals.
• Familiarity with monitoring and visualization tools such as Grafana, Splunk, Tableau or Power BI.
• Experience using GitHub Copilot or other AI-assisted software development tools.
• Exposure to agentic AI, workflow orchestration or retrieval-augmented generation.

Key responsibilities

• Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi.
• Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms.
• Build data processing and storage solutions using Spark, Hadoop and Hive.
• Develop automated reconciliation, data quality and compliance controls to identify missing, duplicate or inconsistent records.
• Implement robust data flows with retry mechanisms, error handling, replay, recovery and monitoring.
• Establish canonical data models, data lineage, validation rules and reliable data structures.
• Develop automated tests to validate data accuracy, completeness, performance and processing reliability.
• Use Git, code reviews and CI/CD pipelines to manage code changes and promote solutions through Development, UAT and Production.
• Troubleshoot production issues, improve pipeline performance and establish operational monitoring and support procedures.
• Collaborate with architects, platform engineers, control owners, risk teams and vendors to deliver reusable engineering solutions.
• Explore AI-assisted development tools such as GitHub Copilot while maintaining appropriate security, quality and governance standards.

What we are looking for

We are seeking an engineer who enjoys solving complex technical problems and taking ownership of solutions from design through to production. You should be comfortable writing code, working with distributed data technologies, investigating data inconsistencies and building reliable systems that meet enterprise compliance requirements.

You will bring a practical engineering mindset, attention to detail and a commitment to delivering secure, scalable, testable and maintainable solutions.

Why join this role?

Work on strategic enterprise data platform engineering.

Build reusable ingestion, streaming and processing capabilities.

Develop automated controls that support compliance, data integrity and auditability.

Work across modern data engineering and software delivery technologies.

Contribute to a platform designed to support future integrations and evolving business requirements.

Pontoon is an employment consultancy.

We put expertise, energy, and enthusiasm into improving everyone's chance of being part of the workplace. We respect and appreciate people of all ethnicities, generations, religious beliefs, sexual orientations, gender identities, and more. We do this by showcasing their talents, skills, and unique experience in an inclusive environment that helps them thrive. If you require reasonable adjustments at any stage, please let us know and we will be happy to support you