Lead Data Software Engineer with Databricks, Apache Kafka, Apache Spark, Kubernetes

EPAM Systems; Turkey; On-site; TRY 5,348,000 - 8,751,000; Full time

Benefits offered by this job

Private health insurance

Learning & development

Mentoring programs

English courses

Job summary

EPAM Systems in Turkey is seeking a Lead Data Software Engineer to drive the migration of an existing data analytics platform to a cloud-agnostic solution ready for on-premises deployment. You will implement streaming and batch Spark pipelines on Kubernetes, develop the data generator framework, and mentor the team while collaborating with the team lead.

The role requires 5+ years with Spark/Databricks/Kubernetes, Python proficiency, Kafka experience, and EPC Data Engineering coursework

Qualifications

• 5+ years of experience with Apache Spark, Databricks, and Kubernetes (K8S).
• Proficiency in Python.
• Experience with Confluent/Apache Kafka.
• Completed EPAM Data Engineering course or equivalent qualification.

Responsibilities

• Implement streaming and batch Spark pipelines running on Kubernetes.
• Develop objects of the data generator framework.
• Deploy and test solutions on local and development environments.
• Enhance and refine the current actively evolving solution.
• Investigate and resolve bugs.
• Conduct unit testing to ensure code quality.
• Participate in refinement, planning, and demo sessions.
• Guide and mentor team members while syncing with the team lead.

Skills

Apache Spark

Databricks

Kubernetes

Python

English (B2+)

Education

EPAM Data Engineering course

Tools

Confluent Kafka

Kafka Connect / KSQL / Kafka Streams

Job description

We are seeking a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as DataFactory, Databricks, EventHub, and Cassandra, to a cloud-agnostic platform capable of on-premises deployment.

Responsibilities

• Implement streaming and batch Spark pipelines running on K8S
• Develop objects of the data generator framework
• Deploy and test solutions on local and development environments
• Enhance and refine the current actively evolving solution
• Investigate and resolve bugs
• Conduct unit testing to ensure code quality
• Participate in refinement, planning, and demo sessions
• Guide and mentor team members while syncing with the team lead

Requirements

• 5+ years of experience with Apache Spark, Databricks, and Kubernetes (K8S)
• Proficiency in Python
• Experience with Confluent/Apache Kafka
• Completed EPAM Data Engineering course or equivalent qualification
• Capability to quickly adapt and dive into the active implementation phase of a project
• Ability to work independently while coordinating with a team lead
• Strong communication skills and a proactive, hands-on, team-player attitude
• English proficiency at B2 level or higher

Nice to have

• Knowledge of Java
• Familiarity with Kafka Connect, KSQL, and Kafka Streams
• Background in Ansible and Argo CD
• Understanding of TDD

We offer

• CONTINUOUS UPSKILLING, LEARNING & DEVELOPMENT
  • Diversity of tasks and projects
  • Assessment center for objective review of competency level
  • Personal development plan
  • Mentoring programs and leadership development
  • Certification and professional development support
  • Access to learning platforms including more than 2,500 internal courses
  • English courses taught by certified teachers
• CORPORATE BENEFITS
  • Extra leave days
  • Referral bonuses
• COMPENSATION PACKAGE
  • Competitive compensation paid in USD
  • Regular salary and performance reviews
• MEDICAL & HEALTHCARE
  • Private health insurance
  • Well-being events
• WORKING ENVIRONMENT
  • Recreation areas and kitchens
  • Tea, coffee and snacks
  • Sports equipment and game consoles
  • IT Equipment
  • Microsoft’s Software Assurance Home Use Program (HUP)

Please note that our Talent Attraction Team reviews applications and CVs submitted in English.

EPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI-Native enterprises, driving measurable value from innovation and digital investments.