Lead Data Software Engineer with Databricks, Apache Kafka, Apache Spark, Kubernetes

EPAM Systems, Inc.; Turkey; On-site; TRY 3,848,000 - 5,772,000; Full time

Benefits offered by this job

Extra leave days

Referral bonuses

Private health insurance

Well-being events

Recreation areas and kitchens

Tea, coffee and snacks

IT Equipment

HUP program

Job summary

EPAM Systems, Inc. seeks a Lead Data Software Engineer to migrate an existing data analytics platform from Azure to a cloud-agnostic, on-premises-capable solution. You will design streaming and batch Spark pipelines on Kubernetes and drive end-to-end implementation.

You will mentor teammates, conduct unit tests, engage in refinement sessions, and collaborate with product and operations while delivering robust data engineering solutions.

Qualifications

• 5+ years of experience with Apache Spark, Databricks, and Kubernetes.
• Proficiency in Python.
• Experience with Confluent/Apache Kafka.
• English proficiency at B2 level or higher.
• Ability to work independently while coordinating with a team lead.
• Strong communication and hands-on, proactive attitude.

Responsibilities

• Develop streaming and batch Spark pipelines on Kubernetes.
• Develop objects of the data generator framework.
• Deploy and test solutions on local and development environments.
• Enhance and refine the current actively evolving solution.
• Investigate and resolve bugs.
• Conduct unit testing to ensure code quality.
• Participate in refinement, planning, and demo sessions.
• Guide and mentor team members while syncing with the team lead.

Skills

Apache Spark

Databricks

Kubernetes

Python

English proficiency

Communication skills

Independent work

Education

EPAM Data Engineering course

Tools

Kafka Connect

KSQL

Kafka Streams

Argo CD

Ansible

Job description

We are seeking a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as DataFactory, Databricks, EventHub, and Cassandra, to a cloud-agnostic platform capable of on-premises deployment.ResponsibilitiesImplement streaming and batch Spark pipelines running on K8SDevelop objects of the data generator frameworkDeploy and test solutions on local and development environmentsEnhance and refine the current actively evolving solutionInvestigate and resolve bugsConduct unit testing to ensure code qualityParticipate in refinement, planning, and demo sessionsGuide and mentor team members while syncing with the team leadRequirements5+ years of experience with Apache Spark, Databricks, and Kubernetes (K8S)Proficiency in PythonExperience with Confluent/Apache KafkaCompleted EPAM Data Engineering course or equivalent qualificationCapability to quickly adapt and dive into the active implementation phase of a projectAbility to work independently while coordinating with a team leadStrong communication skills and a proactive, hands-on, team-player attitudeEnglish proficiency at B2 level or higherNice to haveKnowledge of JavaFamiliarity with Kafka Connect, KSQL, and Kafka StreamsBackground in Ansible and Argo CDUnderstanding of TDDWe offerCONTINUOUS UPSKILLING, LEARNING & DEVELOPMENTDiversity of tasks and projectsAssessment center for objective review of competency levelPersonal development planMentoring programs and leadership developmentCertification and professional development supportAccess to learning platforms including more than 2,500 internal coursesEnglish courses taught by certified teachersCORPORATE BENEFITSExtra leave daysReferral bonusesCOMPENSATION PACKAGECompetitive compensation paid in USDRegular salary and performance reviewsMEDICAL & HEALTHCAREPrivate health insuranceWell-being eventsWORKING ENVIRONMENTRecreation areas and kitchensTea, coffee and snacksSports equipment and game consolesIT EquipmentMicrosoft’s Software Assurance Home Use Program (HUP)Please note that our Talent Attraction Team reviews applications and CVs submitted in English.EPAM is a global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. We deliver globally and engage locally, making the future real for clients, partners, and employees. We are proud to be recognised by Forbes, Glassdoor, Newsweek, Time Magazine, Great Place to Work and kununu as a Most Loved Workplace around the world.