Senior Data Software Engineer with Apache Airflow, Databricks

EPAM Systems; Deutschland; Vor Ort; EUR 70.000 - 110.000; Vollzeit

Zusammenfassung

EPAM Systems is seeking a data engineer to design, build and maintain data pipelines in a flexible, cloud-based environment. You will migrate existing pipelines to Databricks, support ML model workflows and ensure data quality across systems.

The role requires strong Python, Spark and PySpark skills, hands-on Databricks and AWS experience, plus solid documentation in English. Collaboration with Bitbucket, Jira, Confluence and Teams is essential.

Qualifikationen

• 3+ years of experience in data engineering with advanced proficiency in Python, Spark and PySpark.
• Hands-on experience with Databricks and AWS.
• Background in ML model development, training and deployment.
• Skills in data pipeline development, maintenance and ETL processes.
• Understanding of database design, optimization and data quality assurance.
• Fluent in English with strong documentation skills.

Aufgaben

• Develop new data pipelines and maintain existing ones.
• Migrate pipelines from BDP to Databricks.
• Maintain ML ETL pipelines and update ML code as needed.
• Monitor daily pipelines and resolve issues.
• Support ML model development, training and deployment.
• Perform data integration and transformation through ETL.

Kenntnisse

Python

Spark

PySpark

ML model development

Data pipelines

ETL processes

Database design

Data quality

Documentation

English

Tools

Databricks

AWS

Bitbucket

Jira

Confluence

Teams

pandas

Jobbeschreibung

Responsibilities

• Develop new data pipelines and maintain existing ones
• Migrate existing data pipelines from BDP to Databricks
• Maintain ML ETL pipelines and apply necessary changes to ML code
• Monitor daily pipelines and resolve issues as they arise
• Support ML model development, training, and deployment
• Perform data integration and transformation through ETL processes
• Contribute to database design, optimization, and data quality assurance
• Document processes and workflows in clear and unambiguous language
• Collaborate with the team using BitBucket, Jira, Confluence, and Teams

Requirements

• 3+ years of experience in data engineering with advanced Proficiency in Python, Spark, and PySpark
• Hands-on experience with Databricks and AWS
• Background in ML model development, training, and deployment
• Skills in data pipeline development, maintenance, and ETL processes
• Understanding of database design, optimization, and data quality assurance
• Fluent in English with strong documentation skills

Nice to have

• Familiarity with pandas
• Knowledge of Databricks with MLFlow or AWS Sagemaker