Big Data Engineer Lead (SQL_Python_ERD_Terraform_IaC_AWS_CICD_PerformanceTuning)

MALTEM ASIA PTE. LTD.; Singapore; On-site; SGD 100,000 - 130,000; Full time

Job summary

A leading data services company in Singapore is seeking a Lead Big Data Engineer to enhance their data solutions for a Media Communications Client. The ideal candidate will lead architecture discussions, design data pipelines using SQL and Python, and employ AWS services for cloud-based platforms. Responsibilities include ensuring data integrity and implementing best practices in data management and protection.

Qualifications

• Proven experience in leading big data projects.
• Strong technical expertise in ETL/ELT methodologies.
• Familiarity with data privacy and protection frameworks.

Responsibilities

• Design and optimize data pipelines and workflows.
• Lead architecture discussions and system design.
• Collaborate with engineering teams to meet data needs.
• Deploy infrastructure using CI/CD and IoC tools.
• Ensure adherence to data protection initiatives.

Skills

Big data solutions design

Data governance

ETL/ELT workflows

SQL

Python

AWS services

CI/CD pipelines

Infrastructure-as-Code

Tools

Terraform

Job description

Maltem Singapore is currently seeking a Big Data Engineer Lead for our Media Communications Client.

Summary:

• We are seeking a highly skilled and motivated Lead Big Data Engineer to join our data team. The ideal candidate will play a key role in designing, developing, and maintaining scalable big data solutions while providing technical leadership.
• This role will also support strategic Data Governance initiatives, ensuring data integrity, privacy, and accessibility across the organization.

Key Responsibilities:

• Design, implement, and optimize robust data pipelines and ETL/ELT workflows using both:
• SQL
• Python
• Lead Architecture discussions, including the creation and review of Entity Relationship Diagrams (ERDs) and overall system design.
• Collaborate closely with Data Engineers, Analysts, and cross-functional engineering teams to meet evolving data needs.
• Deploy and manage infrastructure using:
• Terraform
• Infrastructure-as-Code (IaC) tools.
• Develop and maintain:
• CI/CD pipelines for deploying Data Applications and Services.
• Leverage strong experience in:
• AWS services (e.g., S3, Glue, Lambda, RDS, Lake Formation) to support scalable and secure cloud-based data platforms.
• Handle both batch and real-time data processing effectively.
• Apply best practices in data modeling and support data privacy and data protection initiatives.
• Implement and manage data encryption and hashing techniques to secure sensitive information.
• Ensure adherence to software engineering best practices including version control, automated testing, and deployment standards.
• Lead performance tuning and troubleshooting for data applications and platforms.