[HIRING WEEK] Senior Data Engineer (Databricks)

EPAM Systems Inc; Malaysia; On-site; MYR 250,000 - 320,000; Full time

Job summary

EPAM Systems Inc. seeks an experienced Senior Data Integration Engineer specializing in Databricks to lead the design of enterprise data products and platforms. You will own end-to-end data ingestion and transformation pipelines on Azure Databricks and mentor junior engineers.

Responsibilities include implementing Databricks on Azure or AWS, collaborating with analytics teams, and creating technical documentation. Strong English skills and a Bachelor's degree are required.

Qualifications

• At least 5 years of experience in Data Integration.
• Strong experience with Databricks and ETL/ELT.
• Proficient in SQL and Python; data modeling fundamentals.
• Experience with Delta Lake concepts.
• Familiar with Azure and AWS cloud platforms.
• Experience developing data pipelines on Databricks with multiple sources.
• Knowledge of data ingestion techniques and best practices.
• Orchestration experience (Databricks Workflows; ADF/Airflow a plus).

Responsibilities

• Develop, maintain and support ETL/ELT solutions.
• Implement and manage Databricks on Azure or AWS Cloud platforms.
• Own end-to-end delivery of data ingestion + transformation pipelines on Azure Databricks.
• Mentoring junior engineers.
• Work collaboratively with the team on Microsoft Power BI.
• Create technical documentation related to the project.

Skills

Databricks

ETL/ELT

SQL

Python

Data modeling

Delta Lake

Cloud platforms

Data pipelines

Orchestration

English communication

Education

Bachelor's degree

Tools

Databricks Workflows

Azure Data Factory

Airflow

Job description

We are looking for an experienced Senior Data Integration Engineer specializing in Databricks to join our team. As a key member of the team, you will lead the design of enterprise digital products and platforms that deliver value to users and achieve business goals. You will collaborate with various stakeholders and contribute to creating engaging and delightful solutions by leveraging your design skills and methods. Come and join us to promote the role of design and achieve positive design outcomes. * Terms and Conditions apply.

Responsibilities

• Develop, maintain and support ETL/ELT solutions
• Implement and manage Databricks on Microsoft Azure or AWS Cloud platforms
• Use SQL, PySpark or Scala to transform and analyze large datasets
• Own end-to-end delivery of data ingestion + transformation pipelines on Azure Databricks
• Works independently with minimal supervision
• Mentoring junior engineers
• Work collaboratively with the team on Microsoft Power BI
• Ensure the adherence to best practices in development, testing and deployment
• Create technical documentation related to the project

Requirements

• At least 5 years of experience and strong experience in Data Integration, proficient in Databricks and ETL/ELT Solutions
• Strong SQL/Python, data modeling fundamentals (dimensional / Lakehouse modeling), incremental processing, data transformation and analysis
• Experience with Delta Lake concepts (tables, partitions, schema evolution)
• Familiarity with cloud platforms (Azure, AWS)
• Have hands-on experience developing data pipelines on Databricks, integrating data from multiple sources, and working closely with analytics, data science and business teams to deliver reliable, high-quality data products
• Knowledge of data ingestion techniques and data integration best practices
• Orchestration experience (Databricks Workflows; ADF/Airflow is a plus)
• Excellent verbal and written communication skills in English
• Nice to have Bachelor's degree, ideally in Computer Science, Information Technology, or a related discipline