Lead Data Software Engineer with Databricks, Apache Kafka, Apache Spark, Kubernetes

EPAM Systems; Deutschland; Vor Ort; EUR 90.000 - 120.000; Vollzeit

Zusammenfassung

EPAM Systems is seeking a Lead Data Software Engineer to drive the migration of an existing data analytics platform to a cloud-agnostic, on-prem deployment. You will implement streaming and batch Spark pipelines on Kubernetes, develop the data generator framework, and mentor the team while coordinating with the team lead.

The role requires strong Python and Spark expertise, Kafka experience, and hands-on collaboration across local and development environments.

Qualifikationen

• 5+ years of experience with Apache Spark, Databricks, and Kubernetes (K8S).
• Proficiency in Python.
• Experience with Confluent/Apache Kafka.
• Capability to quickly adapt and dive into the active implementation phase of a project.
• Ability to work independently while coordinating with a team lead.
• Strong communication skills and a proactive, hands-on, team-player attitude.
• English proficiency at B2 level or higher.
• Nice to have Knowledge of Java.
• Familiarity with Kafka Connect, KSQL, and Kafka Streams.
• Background in Ansible and Argo CD.
• Understanding of TDD.

Aufgaben

• Implement streaming and batch Spark pipelines running on K8S.
• Develop objects of the data generator framework.
• Deploy and test solutions on local and development environments.
• Enhance and refine the current actively evolving solution.
• Investigate and resolve bugs.
• Conduct unit testing to ensure code quality.
• Participate in refinement, planning, and demo sessions.
• Guide and mentor team members while syncing with a team lead.

Kenntnisse

Apache Spark

Databricks

Kubernetes (K8S)

Python

Confluent Kafka

Java

Kafka Connect/KSQL/Kafka Streams

Ansible

Argo CD

TDD

Ausbildung

EPAM Data Engineering course or equivalent

Tools

Kafka Connect

KSQL

Kafka Streams

Argo CD

Ansible

Jobbeschreibung

We are seeking a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as DataFactory, Databricks, EventHub, and Cassandra, to a cloud-agnostic platform capable of on-premises deployment.

Responsibilities

• Implement streaming and batch Spark pipelines running on K8S
• Develop objects of the data generator framework
• Deploy and test solutions on local and development environments
• Enhance and refine the current actively evolving solution
• Investigate and resolve bugs
• Conduct unit testing to ensure code quality
• Participate in refinement, planning, and demo sessions
• Guide and mentor team members while syncing with a team lead

Requirements

• 5+ years of experience with Apache Spark, Databricks, and Kubernetes (K8S)
• Proficiency in Python
• Experience with Confluent/Apache Kafka
• Completed EPAM Data Engineering course or equivalent qualification
• Capability to quickly adapt and dive into the active implementation phase of a project
• Ability to work independently while coordinating with a team lead
• Strong communication skills and a proactive, hands-on, team-player attitude
• English proficiency at B2 level or higher
• Nice to have Knowledge of Java
• Familiarity with Kafka Connect, KSQL, and Kafka Streams
• Background in Ansible and Argo CD
• Understanding of TDD