Senior Data Engineer - Databricks, PySpark & Delta Lake

Unison Group; Kuala Lumpur; Sur place; MYR 120 000 - 180 000; Plein temps

Avantages offerts par ce poste

Bonus

Résumé du poste

Unison Group is seeking a Senior Data Engineer in Malaysia/Singapore to design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python. The role focuses on Delta Lake, Medallion architecture, and secure, governed datasets for reporting and analytics.

The candidate will lead data integration from APIs, SFTP, SharePoint/Graph API, AWS, and Azure sources, and will automate workflows with Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines,

Qualifications

• 5–8+ years of experience in Data Engineering.
• Hands-on experience with Databricks, PySpark, Delta Lake, SQL, Python.
• Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents.
• Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage.
• Knowledge of ETL/ELT concepts, Lakehouse/Meddalion architecture.
• Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines.
• Ability to optimize Spark workloads.

Responsabilités

• Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python.
• Implement Delta Lake and Medallion architecture for scalable enterprise data platforms.
• Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
• Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
• Optimize Spark jobs for performance, reliability, and cost efficiency.
• Implement data validation, quality checks, and monitoring with automated alerts and retries.
• Design secure and governed datasets using Unity Catalog and cloud security best practices.
• Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
• Provide technical leadership and guidance to junior team members.

Connaissances

Data Engineering

Databricks

PySpark

Delta Lake

SQL

Python

Azure DevOps CI/CD

Git

YAML pipelines

Outils

Azure Data Lake

ADF

Azure Functions

AWS (S3, Lambda)

Description du poste

Unison Group is seeking a Senior Data Engineer in Malaysia/Singapore to design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python. The role focuses on Delta Lake, Medallion architecture, and secure, governed datasets for reporting and analytics.

The candidate will lead data integration from APIs, SFTP, SharePoint/Graph API, AWS, and Azure sources, and will automate workflows with Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines,