Data Engineer (Senior Level-Databricks and Pyspark)

Unison Group; Kuala Lumpur; On-site; MYR 180,000 - 260,000; Full time

Benefits offered by this job

Bonus

Job summary

Unison Group in Malaysia is seeking a Senior Data Engineer to design, build, and optimize data pipelines using Databricks, PySpark, and Delta Lake within Azure/AWS environments. You will implement Lakehouse/Medallion architectures, integrate data from APIs, SFTP, and cloud storage, and lead data engineering initiatives with a focus on data quality, governance, and performance.

This full-time role includes a one-month bonus and opportunities to drive enterprise data platforms across cloud

Qualifications

• 5–8+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks, PySpark, Delta Lake, SQL, Python.
• Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents (S3, Lambda).
• Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage.
• Knowledge of ETL/ELT concepts, Lakehouse/Meddalion architecture, and distributed processing.
• Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines.
• Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).

Responsibilities

• Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python.
• Implement Delta Lake and Medallion architecture for scalable enterprise data platforms.
• Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
• Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
• Optimize Spark jobs for performance, reliability, and cost efficiency.
• Implement data validation, quality checks, and monitoring with automated alerts and retries.
• Design secure and governed datasets using Unity Catalog and cloud security best practices.
• Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
• Provide technical leadership and guidance to junior team members.

Skills

Databricks

PySpark

Delta Lake

SQL

Python

Azure

AWS

CI/CD

Git

Azure DevOps

YAML pipelines

Tools

Azure Data Lake

ADF

Azure Functions

S3

Lambda

SharePoint/Graph API

Job description

• Role : Full Time Perm role + one month Bonus.

We are looking for a Senior Data Engineer with strong expertise in Databricks, PySpark, Delta Lake, and cloud-based data pipelines. implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical leadership and hands-on architecture experience.

Key Responsibilities

• Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python.
• Implement Delta Lake and Medallion architecture for scalable enterprise data platforms.
• Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
• Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
• Optimize Spark jobs for performance, reliability, and cost efficiency.
• Implement data validation, quality checks, and monitoring with automated alerts and retries.
• Design secure and governed datasets using Unity Catalog and cloud security best practices.
• Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
• Provide technical leadership and guidance to junior team members.

Required Skills

• 5–8+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks, PySpark, Delta Lake, SQL, Python.
• Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents (S3, Lambda).
• Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage.
• Knowledge of ETL/ELT concepts, Lakehouse/Meddalion architecture, and distributed processing.
• Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines.
• Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).

Good to Have

• Exposure to Oil & Gas or trading analytics (SPARTA, KPLER, IIR, OPEC).
• Knowledge of Power BI or data visualization concepts.
• Familiarity with Terraform, Scala, or PostgreSQL.
• Experience with SharePoint development or .NET (optional)