[C3F] Data Platform Engineer

Software Mind; Warszawa; On-site; PLN 180,000 - 280,000; Full time

Job summary

Software Mind is seeking an experienced data/platform engineer to build and operate CDC pipelines from PostgreSQL into Azure. You will configure connectors, deploy at scale on AKS, and monitor production reliability. You will automate onboarding of new data sources and orchestrate pipelines with Python.

Responsibilities include integrating CDC streams with the Azure data stack, managing infrastructure as code, and enforcing data protection and operational best practices across environments.

Responsibilities

• Build and operate change-data-capture pipelines from PostgreSQL into Azure.
• Configure, deploy and scale connectors end to end.
• Run pipelines as stateful workloads on Kubernetes (AKS).
• Monitor and troubleshoot production platform issues.
• Automate platform onboarding, testing and monitoring with Python.
• Integrate CDC streams with Azure data tools and services.
• Manage infrastructure as code to ensure reproducibility.

Skills

Data engineering

CDC pipelines

PostgreSQL

Azure data stack

Kafka Connect

Debezium

Kubernetes

Python

Infrastructure as code

Tools

Kafka Connect

Debezium

Azure

AKS

Python

Job description

Project – the aim you’ll have

Our customer provides innovative solutions and insights that enable our clients to manage risk and hire the best talent. Their advanced global technology platform supports fully scalable, configurable screening programs that meet the unique needs of over 33,000 clients worldwide. Headquartered in Atlanta, GA, they have an internationally distributed workforce spanning 19 countries with about 5,500 employees. Our partner perform over 93 million screens annually in over 200 countries and territories.

Position – how you’ll contribute

• Build and operate change-data-capture pipelines from PostgreSQL into Azure, using Kafka Connect and Debezium as the core of the platform.
• Configure, deploy and scale connectors end to end - connector setup, task management, offsets, schema history, and snapshot strategy.
• Run these pipelines as stateful workloads on Kubernetes (AKS), covering configuration, secrets, networking and resource tuning.
• Monitor and troubleshoot the platform in production: connector failures, task rebalances, restarts, throughput and backpressure, message-size limits, retries and recovery.
• Automate the platform in Python - configuration-driven onboarding of new data sources, pipeline orchestration, monitoring and alerting, recovery workflows, and automated testing.
• Integrate CDC streams with the wider Azure data stack: Event Hubs, ADLS, Azure PostgreSQL, ADF and Databricks.
• Manage platform infrastructure as code, so environments are reproducible and changes are reviewable.
• Apply data protection requirements to sensitive data flowing through the pipelines - masking, hashing, access control and retention.