Azure Data Engineer

INTELLECT MINDS PTE. LTD.; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

INTELLECT MINDS PTE. LTD. is seeking an experienced Azure Data Engineer to design, build and optimize large‑scale data pipelines leveraging Azure Data Factory and Databricks.

The candidate will implement PySpark and SQL solutions, ensure data quality, and support data migration efforts. The role focuses on data validation, performance tuning, and CI/CD practices with Azure DevOps and Git, while collaborating with stakeholders and analysts to deliver reliable data platforms in a fast‑paced

Qualifications

• Bachelor's or Master's degree in CS/IT or related field.
• Microsoft Certified: Azure Data Engineer Associate (Preferred).
• Databricks Certified Data Engineer Associate (Preferred).

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks.
• Develop ETL/ELT solutions using PySpark and SQL.
• Perform source‑to‑target data validation, reconciliation, and data quality checks.
• Build and optimize data ingestion pipelines into Azure Data Lake Storage (ADLS).
• Develop automated data validation scripts using Python/PySpark.
• Validate incremental and full‑load data processing.
• Perform schema validation, duplicate checks, null checks, and aggregation testing.
• Monitor and optimize pipeline performance to meet SLA requirements.
• Troubleshoot and resolve ETL and data pipeline issues.
• Collaborate with business stakeholders, data analysts, and development teams.
• Maintain CI/CD practices using Azure DevOps and Git.

Skills

Azure Databricks

Azure Data Factory

Azure Data Lake Storage

PySpark

Python

Advanced SQL

ETL Development

Data Pipeline Development

Data Validation

Source‑to‑Target Reconciliation

Data Migration

Performance Tuning

Azure DevOps

Git

Education

Bachelor's or Master's degree in Computer Science, Information Technology, or a related field

Microsoft Certified: Azure Data Engineer Associate (Preferred)

Databricks Certified Data Engineer Associate (Preferred)

Job description

We are looking for an experienced Azure Data Engineer with strong expertise in Azure cloud data platforms, Databricks, PySpark, and Azure Data Factory. The ideal candidate should have hands‑on experience in building, testing, validating, and optimizing large‑scale data pipelines while ensuring data quality, performance, and reliability.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks.
• Develop ETL/ELT solutions using PySpark and SQL.
• Perform source‑to‑target data validation, reconciliation, and data quality checks.
• Build and optimize data ingestion pipelines into Azure Data Lake Storage (ADLS).
• Develop automated data validation scripts using Python/PySpark.
• Validate incremental and full‑load data processing.
• Perform schema validation, duplicate checks, null checks, and aggregation testing.
• Monitor and optimize pipeline performance to meet SLA requirements.
• Troubleshoot and resolve ETL and data pipeline issues.
• Collaborate with business stakeholders, data analysts, and development teams.
• Maintain CI/CD practices using Azure DevOps and Git.

Required Skills

• Azure Databricks
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS Gen2)
• PySpark
• Python
• Advanced SQL
• ETL Development
• Data Pipeline Development
• Data Validation & Data Quality
• Source‑to‑Target Reconciliation
• Data Migration
• Performance Tuning
• Azure DevOps
• Git

Preferred Skills

• Delta Lake
• SSIS
• SQL Server
• Azure Synapse Analytics
• RBAC & Data Security
• Test Automation using Python/PySpark

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• Microsoft Certified: Azure Data Engineer Associate (Preferred)
• Databricks Certified Data Engineer Associate (Preferred)

Nice to Have

• Experience with Azure cloud migration projects.
• Experience working in Banking, Logistics, Retail, or Financial Services domains.
• Strong analytical, troubleshooting, and communication skills.