Remote Data Platform Engineer - Azure, Kafka & CDC

Visa Hunt; Poland; On-site; PLN 180,000 - 240,000; Full time

Benefits offered by this job

Flexible work

International projects

Private healthcare

Multisport card

Language classes

Job summary

Software Mind develops solutions that make an impact for companies around the globe. We seek a Data/Platform Engineer to build and operate CDC pipelines, mainly from PostgreSQL into Azure, using Kafka Connect and Debezium.

You’ll tune Kubernetes workloads, automate onboarding with Python, and integrate with Event Hubs, ADLS, ADF, and Databricks. Ideal candidates have hands-on streaming experience, strong SQL/PostgreSQL skills, and proficiency with Azure data services.

Qualifications

• Solid commercial experience as a data or platform engineer, with hands-on work on streaming or CDC pipelines rather than batch reporting alone.
• Practical Kafka knowledge - topics, partitions, offsets, consumer groups and delivery semantics - including at least one Kafka Connect deployment you ran yourself
• Strong SQL and PostgreSQL skills, with working knowledge of WAL, logical replication, replication slots and replication lag
• Working understanding of CDC concepts: initial snapshots, inserts, updates and deletes, event ordering, at-least-once delivery, and schema evolution
• Confident Python for automation and tooling - orchestration, monitoring, recovery scripts, and automated tests
• Hands-on experience with Azure data services, for example Event Hubs, ADLS or Azure PostgreSQL
• Comfortable working with Kubernetes as a user: deploying workloads, handling configuration and secrets, reading logs, debugging failing pods
• Ability to debug a running pipeline from metrics and logs - telling throughput problems from backpressure, retries or a genuine connector failure

Responsibilities

• Build and operate change-data-capture pipelines from PostgreSQL into Azure, using Kafka Connect and Debezium as the core of the platform
• Configure, deploy and scale connectors end to end - connector setup, task management, offsets, schema history, and snapshot strategy
• Run these pipelines as stateful workloads on Kubernetes (AKS), covering configuration, secrets, networking and resource tuning
• Monitor and troubleshoot the platform in production: connector failures, task rebalances, restarts, throughput and backpressure, message-size limits, retries and recovery
• Automate the platform in Python - configuration-driven onboarding of new data sources, pipeline orchestration, monitoring and alerting, recovery workflows, and automated testing
• Integrate CDC streams with the wider Azure data stack: Event Hubs, ADLS, Azure PostgreSQL, ADF and Databricks
• Manage platform infrastructure as code, so environments are reproducible and changes are reviewable
• Apply data protection requirements to sensitive data flowing through the pipelines - masking, hashing, access control and retention

Skills

Kafka

PostgreSQL

Python

Azure

Kubernetes

Data pipelines

CDC concepts

Tools

Kafka Connect

Debezium

Terraform

Bicep

Databricks

ADF

Job description

Software Mind develops solutions that make an impact for companies around the globe. We seek a Data/Platform Engineer to build and operate CDC pipelines, mainly from PostgreSQL into Azure, using Kafka Connect and Debezium.

You’ll tune Kubernetes workloads, automate onboarding with Python, and integrate with Event Hubs, ADLS, ADF, and Databricks. Ideal candidates have hands-on streaming experience, strong SQL/PostgreSQL skills, and proficiency with Azure data services.