Lead Data Software Engineer with Databricks with Apache Kafka, Apache Spark, Kubernetes

EPAM Systems; Deutschland; Vor Ort; EUR 90.000 - 130.000; Vollzeit

Zusammenfassung

EPAM Systems is seeking a Lead Data Software Engineer to drive streaming and batch data pipelines using Apache Spark on Kubernetes. You will lead the data generator framework and oversee deployments, testing, and iterative improvements across environments.

The role requires hands-on leadership, strong Python skills, and deep knowledge of Spark, Kafka, and cloud-agnostic migrations. Excellent communication and collaboration with the team are essential.

Qualifikationen

• 5+ years of experience with Apache Spark, Confluent/Apache Kafka, and Kubernetes.
• Proficiency in Python.
• Familiarity with the data engineering domain and cloud-agnostic platform migrations.
• Capability to dive deeper into new technologies during active implementation phases.
• Ability to work independently with syncs with a team lead.
• Strong communication skills and proactiveness.
• Hands-on team player and technical leader.
• Proficiency in English at a B2+ level.

Aufgaben

• Implement streaming and batch Spark pipelines on Kubernetes.
• Lead the implementation of objects within the data generator framework.
• Oversee deployment and testing across local and development environments.
• Drive improvement of the current, actively changing solution.
• Conduct bug investigation and resolution.
• Perform unit testing to ensure solution quality.
• Participate in refinement, planning, and demo sessions.
• Guide and mentor team members throughout the implementation phase.

Kenntnisse

Python

Apache Spark

Apache Kafka

Kubernetes

English (B2+)

Tools

Kafka Connect

KSQL

Kafka Streams

Ansible

Argo CD

Jobbeschreibung

Lead Data Software Engineer

Responsibilities

• Implement streaming and batch Spark pipelines running on Kubernetes
• Lead the implementation of objects within the data generator framework
• Oversee deployment and testing across local and development environments
• Drive improvement of the current, actively changing solution
• Conduct bug investigation and resolution
• Perform unit testing to ensure solution quality
• Participate in refinement, planning, and demo sessions
• Guide and mentor team members throughout the implementation phase

Requirements

• 5+ years of experience with Apache Spark, Confluent/Apache Kafka, and Kubernetes
• Proficiency in Python
• Familiarity with the data engineering domain and cloud-agnostic platform migrations
• Capability to dive deeper into new technologies in a short time during active implementation phases
• Ability to work independently with syncs with a team lead
• Strong communication skills and proactiveness
• Showcase of being a hands-on team player and technical leader
• Proficiency in English at a B2+ level
• Nice to have Knowledge of Java
• Familiarity with Kafka Connect, KSQL, and Kafka Streams
• Skills in Ansible and Argo CD
• Understanding of TDD