Data Engineering Lead: Spark on AWS, ML Ops Mentor

Infosight Consulting Limited; Hong Kong; On-site; HKD 900,000 - 1,300,000; Full time

Job summary

Infosight Consulting Limited is seeking a Data Engineering Lead to guide a team building an AWS-based data platform that powers ML workloads. You will mentor engineers, shape architecture, and ensure high-quality delivery while exposing the team to GenAI initiatives.

You will lead Spark-on-EMR pipelines, design streaming APIs, and enable data scientists to self-serve through governed data products. Strong leadership and communication are essential.

Qualifications

• 6+ years of data engineering experience with leadership experience.
• Strong technical leadership to mentor engineers, set standards and review designs.
• Hands-on Apache Spark on AWS EMR for pipelines and optimization.
• Experience designing and handling streaming APIs for real-time data ingestion.

Responsibilities

• Lead, coach and mentor a team of data engineers, growing their skills and delivery capability.
• Provide technical leadership, hands-on guidance, set engineering standards, review designs and code to drive quality.
• Contribute solution ideas and detailed design aligned with client architecture patterns and governance.
• Present and articulate technical solutions to client stakeholders and internal teams, explaining design choices and trade-offs.
• Guide the team in building and optimizing Spark on EMR pipelines for batch and streaming data processing.
• Design and guide streaming API endpoints for real-time data ingestion and API-based serving.
• Enable data scientists to self-serve via curated data products with governed access.
• Champion MLOps practices, supporting model deployment, monitoring and lifecycle management on the platform.
• Support lightweight front-end/service components using React and Node.js in a BFF structure.

Education

Bachelor's degree in Computer Science / Engineering / Data Science

Tools

AWS EMR

S3

Glue

Athena

Redshift

Lambda

Kafka

Job description

Infosight Consulting Limited is seeking a Data Engineering Lead to guide a team building an AWS-based data platform that powers ML workloads. You will mentor engineers, shape architecture, and ensure high-quality delivery while exposing the team to GenAI initiatives.

You will lead Spark-on-EMR pipelines, design streaming APIs, and enable data scientists to self-serve through governed data products. Strong leadership and communication are essential.