Lead Data Engineer

eTeam; Johor; On-site; MYR 60,000 - 90,000; Full time

Job summary

eTeam is seeking a data engineer to consolidate data from multiple enterprise systems, clean and reconcile it, and build robust data pipelines supporting OTIF calculations.

The role requires 5+ years in data engineering, with 2+ years in a technical lead position, strong Python/SQL, Spark/PySpark, and deep AWS experience across S3, Glue, Redshift, Athena, EMR, Lambda, Kinesis, and more. Extended 1-year contract with opportunity for renewal.

Qualifications

• 5+ years of data engineering experience with a leadership track record.
• Strong Python and SQL skills with hands-on coding.
• Extensive Spark/PySpark experience for large-scale processing.
• Deep AWS experience across multiple services and data platforms.
• Experience designing data lake/lakehouse architectures and ETL/ELT pipelines.
• Knowledge of CDC, batch and streaming architectures.
• Familiarity with relational and NoSQL databases and data warehousing concepts.

Responsibilities

• Consolidate data from multiple enterprise systems into reliable pipelines.
• Lead technical work and mentor teammates for data engineering tasks.
• Design and implement data lake/lakehouse solutions supporting OTIF calculations.
• Ensure data quality, lineage, and scalable architectures.

Skills

5+ years of data engineering

Technical lead

Python

SQL

Apache Spark

PySpark

Data modeling

Data warehouse concepts

ETL/ELT

CDC

Batch processing

Streaming architectures

Relational databases

NoSQL databases

Tools

AWS S3

AWS Glue

AWS Redshift

AWS Athena

AWS EMR

AWS Lambda

AWS Kinesis

AWS MSK

AWS Step Functions

AWS Lake Formation

Data Lake/Lakehouse

SQL databases

NoSQL databases

Job description

The consultant needs to be strong in bringing data together from multiple enterprise systems, cleaning/reconciling it, and creating a reliable data pipeline/model that supports OTIF calculations.

Core

• 5+ years of experience in Data Engineering, with 2+ years in a technical lead role.
• Strong expertise in Python and SQL.
• Strong experience with Apache Spark/PySpark.
• Deep AWS experience with services such as S3, Glue, Redshift, Athena, EMR, Lambda, Kinesis/MSK, Step Functions and Lake Formation.
• Experience designing and implementing Data Lake/Lakehouse architectures.
• Strong understanding of ETL/ELT, CDC, batch and streaming architectures.
• Experience with relational and NoSQL databases.
• Strong understanding of data modeling, dimensional modeling, and data warehouse concepts.

Employment type: 1 year contract - extendable