Data Engineer Lead

MR DIY International; Seri Kembangan; On-site; MYR 180,000 - 300,000; Full time

Job summary

MR DIY International is seeking a Data Engineer Lead to bridge business needs and technical implementation by designing scalable data infrastructure, automating data pipelines, and mentoring teams.

You will steer Architecture & Pipeline, Medallion architecture, data lakes, batch and real-time streams, and data source integrations, while governing data quality and cost. Collaborate with IT, marketing, sales and design to ensure platform success and compliance.

Qualifications

• Bachelor’s degree or higher in a related field.
• 8 years of experience in data engineering or related roles with leadership experience.
• Strong experience in data architecture, Medallion Architecture, Data Lakes/Lakehouses, and scalable data platforms.
• Hands-on experience in building batch and real-time data pipelines, ETL/ELT, and data integrations.

Responsibilities

• Bridge between business and technical implementation.
• Design scalable data infrastructure and automated pipelines.
• Mentor and develop data engineering teams.
• Oversee governance, data quality, security and performance.

Skills

Python

SQL

Databricks

Spark

Kafka

Medallion architecture

Data lakes/lakehouses

Data governance

Leadership

Stakeholder management

Education

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Data Science, or related field

Tools

Databricks

Kafka

Azure

AWS

GCP

Job description

A data engineer lead is responsible to bridge between business and technical implementation, raw data and production-ready environments by designing scalable infrastructure, automating data pipelines, and mentoring teams.

Core Responsibilities

• Architecture & Pipeline: Design and govern data model (Medallion architecture) and oversee the implementation (e.g. data lakes, build automated batch / real-time streaming pipelines, data source integrations)
• Collaborations: Partnering with Analysts
• Governance: Establish technical best practices and enforcing data quality

Stakeholder and team collaboration

• Act as a bridge between teams: Collaborate with various departments, such as IT, marketing, sales, and design, to align on goals and execution.
• Manage stakeholder relationships: Keep stakeholders informed and gather their feedback to ensure the platform meets their needs.
• Foster collaboration: Encourage a collaborative and performance-driven culture within the team and across the organization.
• Platform adoption: Ensure the platform is understood, used properly, and delivers benefits

Performance, operations and vendor management

• Monitor and analyze performance: Track key performance indicators (KPIs) and analyze data to identify areas for improvement and inform decisions.
• Oversee technical performance: Ensure the platform is scalable, secure, and performs well by working closely with technical teams. Ensure platform decisions follow architectural, data, and cybersecurity standards
• Ensure compliance: Maintain compliance with brand, legal, and regulatory standards.
• Drive continuous improvement: Conduct experiments and implement updates to enhance user experience and platform capabilities.
• Governance: Approve changes, releases, and enhancements
• Manage key vendors delivering the platform or modules
• Ensure SLAs, performance, and contract obligations are met
• FinOps / Usage Monitoring / License Renewal:
• Own the platform budget — build vs run costs
• Evaluate financial impact of upgrades, licences, and new investments
• Ensure cost efficiency and scalability
• Audit support: Assisting with internal or external audits by providing required documentation, evidence, and responses to ensure compliance with standards and regulations
• Incident management: responding to and resolving unplanned disruptions or service issues to restore normal operations as quickly as possible.
• Demand management: forecasting, prioritizing, and controlling incoming requests for services or resources to ensure capacity aligns with business needs
• Problem management: structured approach to identifying the root causes of recurring incidents and implementing long-term fixes to prevent them from happening again

Job Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Data Science, or related field.
• 8 years of experience in data engineering, data platform development, or related technology roles, with proven technical leadership experience.
• Strong experience in data architecture, data modelling, Medallion Architecture, Data Lakes/Lakehouses, and scalable data platforms.
• Hands‑on experience in building batch and real‑time data pipelines, ETL/ELT, and data integrations.
• Proficiency in Python, SQL and relevant data engineering/cloud technologies such as Databricks, Spark, Kafka, Azure/AWS/GCP.
• Strong knowledge of data governance, data quality, security, performance optimisation, and cloud cost management.
• Proven stakeholder management skills, with the ability to translate business requirements into scalable technical solutions.
• Experience in vendor management, platform operations, incident/problem management, and audit/compliance.
• Strong leadership skills with experience in mentoring and developing data engineering teams.
• Excellent communication, analytical, problem-solving, and project management skills.