Data Engineer Lead

MR DIY International; Seri Kembangan; On-site; MYR 180,000 - 260,000; Full time

Job summary

MR DIY International is seeking a Data Engineering Lead to bridge business needs with technical implementation. You will design scalable data platforms, automate pipelines, and mentor engineers.

You will collaborate with IT, marketing, and design teams, enforce data quality, manage vendors, and drive platform adoption across the organization.

Qualifications

• Bachelor’s degree in CS/IT/Data Science or related field.
• 8+ years in data engineering or related technology roles with leadership.
• Strong experience in data architecture, Medallion architecture, data lakes and scalable platforms.
• Hands-on with batch/real-time pipelines, ETL/ELT, data integrations.
• Proficient in Python, SQL and cloud data tech (Databricks, Spark, Kafka, Azure/AWS/GCP).
• Solid data governance, security, performance, and cloud cost management.
• Proven stakeholder management and vendor management skills.
• Excellent communication, problem solving and project management.

Responsibilities

• Architecture & pipeline: design data models and govern implementation of data lakes and pipelines.
• Collaborate with Analysts and other teams to align goals and execution.
• Governance: enforce data quality and technical best practices across platforms.
• Leadership: mentor junior engineers and guide team development.
• Oversee platform performance, security and scalability with cross‑functional teams.
• Vendor management and audits: ensure SLAs, compliance and cost control.
• Incident, demand and problem management to ensure reliable operations.

Skills

Data pipeline design

Medallion architecture

Python

SQL

Databricks

Spark

Kafka

Cloud platforms

Data governance

Leadership

Stakeholder management

Education

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Data Science, or related field

Tools

Databricks

Spark

Kafka

Azure

AWS

GCP

Job description

A data engineer lead is responsible to bridge between business and technical implementation, raw data and production-ready environments by designing scalable infrastructure, automating data pipelines, and mentoring teams.

Core Responsibilities

• Architecture & Pipeline: Design and govern data model (Medallion architecture) and oversee the implementation (e.g. data lakes, build automated batch / real-time streaming pipelines, data source integrations)
• Collaborations: Partnering with Analysts
• Governance: Establish technical best practices and enforcing data quality
• Leadership & Team Management: Mentoring junior engineers

Stakeholder and team collaboration

• Act as a bridge between teams: Collaborate with various departments, such as IT, marketing, sales, and design, to align on goals and execution.
• Manage stakeholder relationships: Keep stakeholders informed and gather their feedback to ensure the platform meets their needs.
• Foster collaboration: Encourage a collaborative and performance-driven culture within the team and across the organization.
• Platform adoption: Ensure the platform is understood, used properly, and delivers benefits

Performance, operations and vendor management

• Monitor and analyze performance: Track key performance indicators (KPIs) and analyze data to identify areas for improvement and inform decisions.
• Oversee technical performance: Ensure the platform is scalable, secure, and performs well by working closely with technical teams. Ensure platform decisions follow architectural, data, and cybersecurity standards
• Ensure compliance: Maintain compliance with brand, legal, and regulatory standards.
• Drive continuous improvement: Conduct experiments and implement updates to enhance user experience and platform capabilities.
• Governance: Approve changes, releases, and enhancements
• Vendor Management
• Manage key vendors delivering the platform or modules
• Ensure SLAs, performance, and contract obligations are met
• FinOps / Usage Monitoring / License Renewal:
• Own the platform budget — build vs run costs
• Evaluate financial impact of upgrades, licences, and new investments
• Ensure cost efficiency and scalability
• Audit support: Assisting with internal or external audits by providing required documentation, evidence, and responses to ensure compliance with standards and regulations
• Incident management: responding to and resolving unplanned disruptions or service issues to restore normal operations as quickly as possible.
• Demand management: forecasting, prioritizing, and controlling incoming requests for services or resources to ensure capacity aligns with business needs
• Problem management: structured approach to identifying the root causes of recurring incidents and implementing long-term fixes to prevent them from happening again

Job Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Data Science, or related field.
• 8 years of experience in data engineering, data platform development, or related technology roles, with proven technical leadership experience.
• Strong experience in data architecture, data modelling, Medallion Architecture, Data Lakes/Lakehouses, and scalable data platforms.
• Hands-on experience in building batch and real-time data pipelines, ETL/ELT, and data integrations.
• Proficiency in Python, SQL and relevant data engineering/cloud technologies such as Databricks, Spark, Kafka, Azure/AWS/GCP.
• Strong knowledge of data governance, data quality, security, performance optimisation, and cloud cost management.
• Proven stakeholder management skills, with the ability to translate business requirements into scalable technical solutions.
• Experience in vendor management, platform operations, incident/problem management, and audit/compliance.
• Strong leadership skills with experience in mentoring and developing data engineering teams.
• Excellent communication, analytical, problem-solving, and project management skills.