Data Engineer - Real-Time Pipelines & Kafka

LA International; Greater London; On-site; GBP 60,000 - 90,000; Full time

Job summary

LA International is seeking a data engineer to develop, test, deploy, and maintain data pipelines and ETL/ELT processes across cloud and on‑premise platforms, ensuring reliability and scalability.

You will build real-time streaming solutions with Apache Kafka, containerised services with Kubernetes, and automation scripts in Python and Java, collaborating with Solution Architects, Data Architects, Business Analysts, and stakeholders in Agile ceremonies, while documenting operational procedures

Responsibilities

• Develop, test, deploy, and maintain data pipelines and ETL/ELT processes across cloud and on-premise platforms, ensuring reliability and scalability.
• Build and support real-time data streaming solutions using Apache Kafka.
• Design and implement containerised applications and data services using Kubernetes.
• Develop software components and automation scripts using Python and Java.
• Integrate data from multiple internal and external sources into cloud and on-premise platforms.
• Monitor, troubleshoot, and optimise data processing workflows.
• Implement data quality checks, validation rules, and data governance controls.
• Support CI/CD pipelines and DevOps practices to enable automated deployments.
• Collaborate with Solution Architects, Data Architects, Business Analysts, and stakeholders to understand requirements.
• Produce technical documentation and maintain operational procedures.
• Participate in Agile ceremonies, including sprint planning, stand-ups, reviews, and retrospectives.
• Support incident investigation and resolution activities within agreed SLAs.

Job description

LA International is seeking a data engineer to develop, test, deploy, and maintain data pipelines and ETL/ELT processes across cloud and on‑premise platforms, ensuring reliability and scalability.

You will build real-time streaming solutions with Apache Kafka, containerised services with Kubernetes, and automation scripts in Python and Java, collaborating with Solution Architects, Data Architects, Business Analysts, and stakeholders in Agile ceremonies, while documenting operational procedures