Senior Data Engineer

INTELLECT MINDS PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

INTELLECT MINDS PTE. LTD. is seeking a Senior Data Engineer to design and deliver scalable data platforms across Azure and AWS.

You will build high-performance data pipelines, data lakes, and real-time ingestion frameworks using PySpark and Databricks. The role requires hands-on experience with Azure and/or AWS, PySpark, Databricks, and strong knowledge of data lakes, ETL/ELT, and real-time processing. You will collaborate with data architects, engineers, and business stakeholders to ensure

Qualifications

• 5+ years of experience in Data Engineering.
• Hands-on experience with Azure and/or AWS data platforms.
• Excellent PySpark and Databricks experience.
• Knowledge of Data Lakes, ETL/ELT, and real-time data ingestion.
• Experience with SQL and Python.
• Knowledge of Apache Spark and distributed data processing.
• Experience with cloud data services and data architecture.
• Good understanding of CI/CD, Git, and Agile methodologies.

Responsibilities

• Design and develop scalable data platforms on Azure and AWS.
• Build and optimize high-performance ETL/ELT pipelines using PySpark and Databricks.
• Develop and manage cloud-based data lakes and data warehouses.
• Design real-time and batch data ingestion frameworks.
• Implement data processing solutions using Apache Spark/PySpark.
• Optimize data pipelines for performance, scalability, and reliability.
• Work with cloud services such as Azure Data Factory, ADLS, Azure Synapse, AWS S3, Glue, and EMR.
• Implement data quality, security, monitoring, and governance standards.
• Collaborate with Data Architects, Engineers, and business stakeholders.

Skills

SQL

Python

CI/CD

Git

Agile

Tools

Azure

AWS

PySpark

Databricks

Apache Spark

Delta Lake

Terraform

Azure Data Factory

Azure Synapse

ADLS

S3

Glue

EMR

Job description

Job Summary

We are looking for a Senior Data Engineer with strong experience designing and delivering scalable data platforms across Microsoft Azure and AWS ecosystems. The candidate will be responsible for building high-performance data pipelines, data lakes, and real-time data ingestion frameworks using PySpark and Databricks.

Key Responsibilities

• Design and develop scalable data platforms on Azure and AWS.
• Build and optimize high-performance ETL/ELT pipelines using PySpark and Databricks.
• Develop and manage cloud-based data lakes and data warehouses.
• Design real-time and batch data ingestion frameworks.
• Implement data processing solutions using Apache Spark/PySpark.
• Optimize data pipelines for performance, scalability, and reliability.
• Work with cloud services such as Azure Data Factory, ADLS, Azure Synapse, AWS S3, Glue, and EMR.
• Implement data quality, security, monitoring, and governance standards.
• Collaborate with Data Architects, Engineers, and business stakeholders.

Required Skills

• 5+ years of experience in Data Engineering.
• Strong hands-on experience with Azure and/or AWS data platforms.
• Excellent PySpark and Databricks experience.
• Strong knowledge of Data Lakes, ETL/ELT, and real-time data ingestion.
• Experience with SQL and Python.
• Knowledge of Apache Spark and distributed data processing.
• Experience with cloud data services and data architecture.
• Good understanding of CI/CD, Git, and Agile methodologies.

Preferred Skills

• Azure Data Factory / Azure Synapse / ADLS
• AWS S3 / Glue / EMR
• Kafka or other real-time streaming technologies
• Delta Lake
• Terraform / Infrastructure as Code
• Data governance and security