Senior Databricks Data Integration Engineer (Azure/AWS)

EPAM Systems; Malaysia; On-site; MYR 180,000 - 260,000; Full time

Job summary

EPAM Systems in Malaysia seeks a Senior Data Integration Engineer specializing in Databricks to join our team. You will develop, maintain and support ETL/ELT solutions, implement and manage Databricks on Azure or AWS, and use SQL and PySpark to transform and analyze large datasets.

You will own end-to-end delivery of data ingestion and transformation pipelines on Azure Databricks, work with minimal supervision, mentor junior engineers and collaborate with analytics teams to deliver reliable data

Qualifications

• 5+ years of experience in Data Integration or related field.
• Proficient in Databricks and ETL/ELT solutions.
• Strong SQL and Python skills.
• Knowledge of data modeling fundamentals (dimensional / Lakehouse).
• Experience with Delta Lake concepts (tables, partitions, schema evolution).
• Familiarity with Azure and AWS cloud platforms.
• Hands-on experience developing data pipelines on Databricks with multiple sources.
• Experience collaborating with analytics, data science and business teams to deliver data products.
• Knowledge of data ingestion techniques and data integration best practices.
• Orchestration experience (Databricks Workflows; ADF/Airflow is a plus).
• Excellent verbal and written communication skills in English.

Responsibilities

• Develop, maintain and support ETL/ELT solutions.
• Implement and manage Databricks on Microsoft Azure or AWS Cloud platforms.
• Use SQL, PySpark or Scala to transform and analyze large datasets.
• Own end-to-end delivery of data ingestion and transformation pipelines on Azure Databricks.
• Work independently with minimal supervision.
• Mentor junior engineers.
• Collaborate with the team on Microsoft PowerBI.
• Ensure adherence to best practices in development, testing and deployment.
• Create technical documentation related to the project.

Skills

Databricks

SQL

Python

ETL/ELT

Delta Lake

Data Modeling

Bachelor's degree

English communication

Education

Bachelor's degree

Tools

Databricks Workflows

ADF/Airflow

Job description

EPAM Systems in Malaysia seeks a Senior Data Integration Engineer specializing in Databricks to join our team. You will develop, maintain and support ETL/ELT solutions, implement and manage Databricks on Azure or AWS, and use SQL and PySpark to transform and analyze large datasets.

You will own end-to-end delivery of data ingestion and transformation pipelines on Azure Databricks, work with minimal supervision, mentor junior engineers and collaborate with analytics teams to deliver reliable data