Lead Data Engineer

ELLIOTT MOSS CONSULTING PTE. LTD.; Singapore; On-site; SGD 120,000 - 160,000; Full time

Job summary

A leading data consulting firm in Singapore is seeking a Lead Data Engineer specializing in AWS, Databricks, and Informatica IDMC. This role involves designing and maintaining robust data infrastructure, optimizing data pipelines, and ensuring high-quality data management. The ideal candidate will have a significant background in data engineering with 10+ years of experience and a strong proficiency in relevant tools and programming languages. This position is an excellent opportunity for those looking to drive data-driven decision-making.

Qualifications

• Minimum 10 years of data engineering experience, including AWS and Databricks.
• Proficiency in Python, Java, or Scala for building data pipelines.
• Experience with ETL/ELT processes and data integration.

Responsibilities

• Design and architect data storage solutions using AWS and Databricks.
• Develop and optimize data pipelines for ingestion and processing.
• Implement data security best practices and comply with regulations.

Skills

AWS

Databricks

Informatica IDMC

Data Engineering

Python

SQL

NoSQL

ETL

Education

Bachelor’s or master’s degree in computer science, Data Engineering, or related field

Tools

Databricks

Informatica IDMC

Docker

Kubernetes

Tableau

Job description

Job Description

We are looking for a Lead Data Engineer specializing in AWS, Databricks, and Informatica IDMC to design, build, and maintain a robust, integrated, and governed data infrastructure. You will enable data-driven decision-making while ensuring high-quality, secure, and compliant data management.

Key Responsibilities

• Design and architect data storage solutions (databases, data lakes, data warehouses) using AWS services (S3, RDS, Redshift, DynamoDB) and Databricks Delta Lake. Integrate Informatica IDMC for metadata management and data cataloging.
• Develop, manage, and optimize data pipelines for data ingestion, processing, and transformation using AWS (Glue, Data Pipeline, Lambda), Databricks, and Informatica IDMC.
• Integrate and transform data from internal and external sources while ensuring data consistency, quality, and governance.
• Build ETL processes to cleanse, enrich, and prepare data for analytics using Spark (Databricks) and Informatica IDMC.
• Monitor, optimize, and troubleshoot data processing and queries for performance, scalability, and cost efficiency.
• Implement data security best practices and comply with data privacy regulations.
• Automate routine workflows for ingestion, transformation, and monitoring using AWS, Databricks Jobs, and Informatica IDMC.
• Maintain clear documentation of infrastructure, pipelines, and configurations with metadata management.
• Collaborate with cross-functional teams (data scientists, analysts, software engineers) to deliver reliable data solutions.
• Stay updated on AWS, Databricks, Informatica IDMC, and data engineering best practices.

Requirements

• Bachelor’s or master’s degree in computer science, Data Engineering, or related field.
• Minimum 10 years of data engineering experience, including AWS, Databricks, and Informatica IDMC.
• Proficiency in Python, Java, or Scala for building data pipelines.
• Strong knowledge of SQL and NoSQL databases; data modeling and schema design.
• Experience with ETL/ELT processes, data integration, and performance optimization.
• Strong analytical, problem-solving, and communication skills.
• AWS, Databricks, and/or Informatica certifications are a plus.

Preferred Skills

• PySpark experience on Databricks
• Knowledge of data governance and cataloging tools, especially Informatica IDMC
• Familiarity with Tableau or other data visualization tools
• Experience with containerization (Docker) and orchestration (Kubernetes)
• Understanding of DevOps principles for CI/CD in data pipelines
• Experience with Git or other version control systems