Data Engineering Lead

Infosight Consulting Limited; Hong Kong; On-site; HKD 900,000 - 1,300,000; Full time

Job summary

Infosight Consulting Limited is seeking a Data Engineering Lead to guide a team building an AWS-based data platform that powers ML workloads. You will mentor engineers, shape architecture, and ensure high-quality delivery while exposing the team to GenAI initiatives.

You will lead Spark-on-EMR pipelines, design streaming APIs, and enable data scientists to self-serve through governed data products. Strong leadership and communication are essential.

Qualifications

• 6+ years of data engineering experience with leadership experience.
• Strong technical leadership to mentor engineers, set standards and review designs.
• Hands-on Apache Spark on AWS EMR for pipelines and optimization.
• Experience designing and handling streaming APIs for real-time data ingestion.

Responsibilities

• Lead, coach and mentor a team of data engineers, growing their skills and delivery capability.
• Provide technical leadership, hands-on guidance, set engineering standards, review designs and code to drive quality.
• Contribute solution ideas and detailed design aligned with client architecture patterns and governance.
• Present and articulate technical solutions to client stakeholders and internal teams, explaining design choices and trade-offs.
• Guide the team in building and optimizing Spark on EMR pipelines for batch and streaming data processing.
• Design and guide streaming API endpoints for real-time data ingestion and API-based serving.
• Enable data scientists to self-serve via curated data products with governed access.
• Champion MLOps practices, supporting model deployment, monitoring and lifecycle management on the platform.
• Support lightweight front-end/service components using React and Node.js in a BFF structure.

Education

Bachelor's degree in Computer Science / Engineering / Data Science

Tools

AWS EMR

S3

Glue

Athena

Redshift

Lambda

Kafka

Job description

Support a transforming, AWS-based Data Product Platform that runs ML workloads and enables data scientists to self-serve. As the Data Engineering Lead, you will lead and coach a team of data engineers, provide technical leadership, and contribute ideas and solution design that align with the client's required architecture practice and governance. While the platform is owned by the client, you are trusted to shape engineering direction, uplift the team, and ensure quality delivery. This role also offers the opportunity to gain exposure to emerging AI and GenAI initiatives - a great way to grow into the next wave of data and AI work.

Responsibilities:

• Lead, coach and mentor a team of data engineers, growing their technical skills and delivery capability.
• Provide technical leadership and hands- on guidance, setting engineering standards, reviewing designs and code, and driving quality across the team.
• Contribute solution ideas and detailed design that align with the client's required architecture practice, patterns and governance.
• Present and articulate technical solutions to client stakeholders and internal teams, clearly explaining design decisions and trade-offs.
• Guide the team in building and optimising Spark on EMR pipelines for batch and streaming data processing.
• Design and guide the handling of streaming API endpoints - real-time data ingestion, consumption and serving through APIs.
• Enable data scientists to self-serve through curated data products and governed access, advising on the best delivery approach.
• Champion good engineering practice around MLOps - supporting model deployment, monitoring and lifecycle management for ML models on the platform.
• Support delivery of lightweight front-end and service components using React and Node.js in a BFF (Backend-for-Frontend) structure.
• Partner with client stakeholders, data scientists and platform teams to translate requirements into robust, secure and cost-effective solutions.
• Promote good practice in data quality, integrity, lineage and security across the data lifecycle.
• Opportunity to gain exposure to AI / GenAI initiatives and contribute ideas as they emerge.

Requirement:

• Bachelor's degree in Computer Science, Engineering, Data Science or a related technical field, or equivalent industrial experience.
• 6+ years of data engineering experience, with a proven track record leading and coaching engineering teams.
• Strong technical leadership skills - able to mentor engineers, set standards, and review designs and code.
• Ability to present and articulate technical solutions clearly to both technical and non-technical stakeholders.
• Hands- on expertise in Apache Spark on AWS EMR (key skill) - pipeline design, performance tuning and cost optimisation.
• Experience designing and handling streaming API endpoints - real-time data ingestion, consumption and API- based serving.
• Solid AWS data platform experience: S3, Glue, Lake Formation, Athena, Redshift, Step Functions and Lambda.
• Proficiency in Python; a strong secondary skill set beyond Python is preferred - React / Node.js in a BFF structure.
• Ability to contribute solution ideas and design within a client's required architecture practice, patterns and governance.
• Understanding of MLOps concepts and experience enabling data scientist self- service.
• Streaming and messaging experience (e.g. Kafka, Spark Streaming) and strong SQL / data modelling skills.
• Interest in or exposure to AI / GenAI initiatives is a plus (not the primary focus of the role).
• Excellent communication, stakeholder management and coaching skills.
• Fluent in Cantonese and English; Mandarin is a plus.
• Previous IT experience in a consulting firm or services vendor is preferred.

Researching careers? Find all the information and tips you need on career advice.