Lead Data Engineer — Spark, Kafka, Kubernetes (Remote)

EPAM Systems; Georgia; On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Remote work from Georgia

Relocation options

Global assignments

Job summary

EPAM Systems in Georgia is seeking a Lead Data Software Engineer to drive migration of an analytics platform from Azure to a cloud-agnostic, on-premises capable solution. You will lead streaming and batch pipelines and mentor teammates.

You will architect Spark pipelines on Kubernetes, oversee deployments, and collaborate with teams to deliver high-quality data solutions. Proficiency in Python and English (B2+) is required, with strong communication.

Qualifications

• 5+ years of experience with Spark, Kafka, and Kubernetes.
• Proficiency in Python.
• Familiarity with data engineering migrations to cloud-agnostic platforms.
• Ability to work independently with guidance from a team lead.

Responsibilities

• Implement streaming and batch Spark pipelines on Kubernetes.
• Lead the data generator framework components.
• Oversee deployment and testing across environments.
• Drive improvement of the existing solution.
• Conduct bug investigation and resolution.
• Perform unit testing to ensure quality.
• Participate in planning and demos.
• Mentor team members through implementation.

Skills

Apache Spark

Confluent/Apache Kafka

Kubernetes

Python

English (B2+)

Tools

Kafka Connect

KSQL

Kafka Streams

Ansible

Argo CD

Job description

EPAM Systems in Georgia is seeking a Lead Data Software Engineer to drive migration of an analytics platform from Azure to a cloud-agnostic, on-premises capable solution. You will lead streaming and batch pipelines and mentor teammates.

You will architect Spark pipelines on Kubernetes, oversee deployments, and collaborate with teams to deliver high-quality data solutions. Proficiency in Python and English (B2+) is required, with strong communication.