Lead Data Platform Engineer: Spark, Kafka & Kubernetes

EPAM Systems, Inc.; Turkey; On-site; TRY 3,848,000 - 5,772,000; Full time

Benefits offered by this job

Extra leave days

Referral bonuses

Private health insurance

Well-being events

Recreation areas and kitchens

Tea, coffee and snacks

IT Equipment

HUP program

Job summary

EPAM Systems, Inc. seeks a Lead Data Software Engineer to migrate an existing data analytics platform from Azure to a cloud-agnostic, on-premises-capable solution. You will design streaming and batch Spark pipelines on Kubernetes and drive end-to-end implementation.

You will mentor teammates, conduct unit tests, engage in refinement sessions, and collaborate with product and operations while delivering robust data engineering solutions.

Qualifications

• 5+ years of experience with Apache Spark, Databricks, and Kubernetes.
• Proficiency in Python.
• Experience with Confluent/Apache Kafka.
• English proficiency at B2 level or higher.
• Ability to work independently while coordinating with a team lead.
• Strong communication and hands-on, proactive attitude.

Responsibilities

• Develop streaming and batch Spark pipelines on Kubernetes.
• Develop objects of the data generator framework.
• Deploy and test solutions on local and development environments.
• Enhance and refine the current actively evolving solution.
• Investigate and resolve bugs.
• Conduct unit testing to ensure code quality.
• Participate in refinement, planning, and demo sessions.
• Guide and mentor team members while syncing with the team lead.

Skills

Apache Spark

Databricks

Kubernetes

Python

English proficiency

Communication skills

Independent work

Education

EPAM Data Engineering course

Tools

Kafka Connect

KSQL

Kafka Streams

Argo CD

Ansible

Job description

EPAM Systems, Inc. seeks a Lead Data Software Engineer to migrate an existing data analytics platform from Azure to a cloud-agnostic, on-premises-capable solution. You will design streaming and batch Spark pipelines on Kubernetes and drive end-to-end implementation.

You will mentor teammates, conduct unit tests, engage in refinement sessions, and collaborate with product and operations while delivering robust data engineering solutions.