Senior Data Software Engineer with Databricks and Azure

EPAM Systems; Deutschland; Vor Ort; EUR 70.000 - 90.000; Vollzeit

Zusammenfassung

EPAM Systems is seeking a Data Engineer to support a team building end-to-end data pipelines for ML modeling teams. You will develop and optimize transformations in Databricks using PySpark, handling data in Delta Lake formats stored in ADLS Gen2 and SAP HANA Data Lake.

You will implement data quality checks and work closely with stakeholders to understand data and model inputs. The role requires strong Python/PySpark skills, Delta Lake expertise, and familiarity with Git, GitHub, and GitHub

Qualifikationen

• 3+ years of experience in data engineering with proficiency in Python, PySpark, and Databricks.
• Hands-on experience with Delta Lake and Azure data lake technologies.
• Familiarity with version control tools such as GitHub and Git.
• Experience with CI/CD frameworks such as GitHub Actions.
• Knowledge of data lake technologies and large-scale dataset performance tuning.
• Proficiency in English (B2+) for effective communication with the customer’s team.

Aufgaben

• Support a team of data engineers to build pipelines used by MLOps and ML Engineers on ML modeling teams.
• Develop, optimize, and maintain data transformation pipelines in Databricks using PySpark.
• Work with data stored in ADLS Gen2 and SAP HANA Data Lake, primarily in Delta/Parquet format.
• Implement and maintain data quality checks, including schema validation, deduplication, enrichment, and tagging.
• Communicate with stakeholders to understand business processes and model input data.
• Tune performance for large-scale datasets to ensure efficient processing.

Kenntnisse

Python

PySpark

Databricks

Tools

Databricks

Delta Lake

Azure Data Lake

GitHub

Git

GitHub Actions

Jobbeschreibung

Responsibilities

• Support a team of data engineers to build pipelines used by MLOps and ML Engineers on ML modeling teams
• Develop, optimize, and maintain data transformation pipelines in Databricks using PySpark
• Work with data stored in ADLS Gen2 and SAP HANA Data Lake, primarily in Delta/Parquet format
• Implement and maintain data quality checks, including schema validation, deduplication, enrichment, and tagging
• Communicate with stakeholders to understand business processes and model input data
• Tune performance for large-scale datasets to ensure efficient processing

Requirements

• 3+ years of experience in data engineering with proficiency in Python, PySpark, and Databricks
• Hands-on experience with Delta Lake and Azure data lake technologies
• Familiarity with software version control tools such as GitHub and Git
• Experience with CI/CD frameworks such as GitHub Actions
• Knowledge of data lake technologies and large-scale dataset performance tuning
• Proficiency in English (B2+) for effective communication with the customer's team

Nice to have

• Familiarity with at least one other programming/scripting language, such as Java, SQL, or Scala