Debezium and Kafka Connect Application Developer

N Consulting Limited; Eindhoven; Hybrid; EUR 70,000 - 100,000; Full time

Job summary

N Consulting Limited is hiring a Debezium and Kafka Connect Application Developer in Eindhoven, NL. The role focuses on building robust CDC pipelines using Debezium to stream data to a data lakehouse within AWS, Dockerized and managed on ECS/ECR.

You'll own end-to-end pipeline lifecycles, implement IaC with Terraform, and maintain CI/CD pipelines with GitHub Actions. Hybrid/onsite work with a strong emphasis on reliability and data integrity.

Qualifications

• Hands-on Kafka Connect/ Debezium experience in production environments.
• Strong Docker experience and container lifecycle management.
• Proven AWS ECS/ECR deployment and maintenance capabilities.

Responsibilities

• Develop and maintain CDC ingestion pipelines with Kafka Connect.
• Containerize and deploy Debezium and Kafka Connect workloads on AWS.
• Manage data ingestion to S3 with Parquet and Iceberg formats.
• Implement Terraform provisioning and CI/CD pipelines with GitHub Actions.
• Monitor health and reliability of the data platform; troubleshoot issues.
• Optional: support Airflow-based workflow orchestration.

Skills

Kafka Connect

Docker

AWS ECS

Terraform

GitHub Actions

Python

Bash

Airflow

Tools

AWS ECR

Parquet/ Iceberg

CI/CD pipelines

Job description

Role: Debezium and Kafka Connect Application Developer

Location: DAf, Eindhoven, Netherlands

Type: 1 yr contractMode: Fully Onsite

Role Overview

We are seeking an experienced Developer/Technical Application Manager to design, build, and maintain robust data ingestion pipelines. The core of this role involves leveraging Debezium for Change Data Capture (CDC) to stream data into our datalakehouse.You will be responsible for the end-to-end lifecycle of these pipelines, operating within an AWS environment using Docker containers on ECS and ECR. This is a hybrid role requiring strong development capabilities alongside operational management (Application Management) to ensure high availability and data integrity of our data ingestion pipelines.

Key Responsibilities

• Pipeline Development: Take ownership of the CDC ingestion framework utilizing Kafka connectors (Debezium, Iceberg sink, S3 sink).
• ContainerizedInfrastructure Management: Deploy and manage Debezium and Kafka Connect workers using Docker containers orchestrating on AWS ECS (Elastic Container Service) and ECR.Data Lake Integration: Manage data ingestion into AWS S3, utilizing Parquet and Apache Iceberg formats.
• Infrastructure as Code: Use Terraform to provision and manage AWS resources supporting the data platform.
• CI/CD: Build and maintain deployment pipelines using GitHub and GitHub Actions.
• Operational Excellence: Monitor pipeline health, troubleshoot connectivity issues, and ensure the reliability of the Kafka ecosystem.
• Optional: Support and optimize workflow orchestration using Airflow where applicable.

Candidate Profile

Must-Have Experience (Non-Negotiable)

• Apache Kafka & Kafka Connect: Multiple years of hands-on experience configuring, deploying, and managing Kafka Connect clusters in a production environment.
• Containerization: Extensive experience with Docker is required. You must be comfortable building images and managing container lifecycles.
• AWS Compute: Proven experience running containers on AWS ECS and managing images via AWS ECR.

Key Technical Requirements

• Infrastructure as Code: Strong proficiency with Terraform for provisioning AWS resources.
• CI/CD: Experience creating and maintaining workflows in GitHub and GitHub Actions.
• Optional: Familiarity with Apache Airflow for scheduling and monitoring is a plus but not mandatory.

Highly Desirable (Strong Plus)

• Debezium: While Kafka Connect experience is a must, specific experience configuring Debezium connectors for various databases (SQL Server, PostgreSQL, etc.) is a massive advantage.
• Open Table Formats: Strong knowledge of Apache Iceberg (or Hudi/Delta Lake) and storing data as Parquet on AWS S3 is significantly preferred.

Baseline Skills (Assumed)

• Programming/Scripting: Proficiency in Python and Bash scripting.
• Cloud Fundamentals: General knowledge of AWS core services (IAM, VPC, CloudWatch).
• Optional: Basic understanding of Object-Oriented (OO) principles and software design patterns.

Tech Stack Summary

• Streaming: Apache Kafka, Kafka Connect, Debezium
• Compute/Containerization: AWS ECS, AWS ECR, Docker
• Storage/Format: AWS S3, Apache Iceberg, Parquet
• DevOps: Terraform, GitHub Actions
• Languages: Python, Bash
• Optional Orchestration: Apache Airflow