Data Engineer Lead

MR DIY International; Selangor; On-site; MYR 180,000 - 320,000; Full time

Job summary

MR DIY International in Malaysia seeks a Senior Data Engineering Lead to design and govern data models and pipelines using a Medallion architecture. You will partner with IT, marketing, sales, and design to align goals, manage stakeholders, and drive platform adoption while ensuring performance, security, and cost efficiency.

Strong leadership and communication are essential to mentor teams and deliver scalable data solutions.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Data Science, or related field.
• 8 years of experience in data engineering, data platform development, or related technology roles, with proven technical leadership experience.
• Strong experience in data architecture, data modelling, Medallion Architecture, Data Lakes/Lakehouses, and scalable data platforms.
• Hands-on experience in building batch and real-time data pipelines, ETL/ELT, and data integrations.
• Proficiency in Python, SQL and relevant data engineering/cloud technologies such as Databricks, Spark, Kafka, Azure/AWS/GCP.
• Strong knowledge of data governance, data quality, security, performance optimisation, and cloud cost management.
• Proven stakeholder management skills, with the ability to translate business requirements into scalable technical solutions.
• Experience in vendor management, platform operations, incident/problem management, and audit/compliance.
• Strong leadership skills with experience in mentoring and developing data engineering teams.
• Excellent communication, analytical, problem-solving, and project management skills.

Responsibilities

• Architecture & Pipeline: Design and govern data model (Medallion) and oversee implementation of data lakes and pipelines.
• Collaborations: Partner with Analysts and cross-functional teams to align on goals.
• Governance: Establish best practices and enforce data quality.
• Bridge between teams: Collaborate with IT, marketing, sales, and design to align goals.
• Platform adoption: Ensure the platform is understood and delivers benefits.
• Performance & vendor management: Monitor KPIs, ensure scalability, security, and compliance.
• FinOps / Usage Monitoring: Manage platform budget and cost efficiency.
• Audit support & incident management: Provide documentation and quick incident resolution.
• Demand & problem management: Forecast requests and identify root causes for fixes.

Skills

Data engineering

Data architecture

Medallion architecture

Data modelling

Batch & real-time pipelines

Python

SQL

Databricks

Spark

Kafka

Cloud platforms

Data governance

Data quality

Security

Cost management

Stakeholder management

Leadership

Mentoring

Communication

Project management

Education

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Data Science, or related field

Tools

Databricks

Spark

Kafka

Azure

AWS

GCP

Job description

Core Responsibilities

• Architecture & Pipeline: Design and govern data model (Medallion architecture) and oversee the implementation (e.g. data lakes, build automated batch / real-time streaming pipelines, data source integrations)
• Collaborations: Partnering with Analysts
• Governance: Establish technical best practices and enforcing data quality

Stakeholder and team collaboration

• Act as a bridge between teams: Collaborate with various departments, such as IT, marketing, sales, and design, to align on goals and execution.
• Manage stakeholder relationships: Keep stakeholders informed and gather their feedback to ensure the platform meets their needs.
• Foster collaboration: Encourage a collaborative and performance-driven culture within the team and across the organization.
• Platform adoption: Ensure the platform is understood, used properly, and delivers benefits

Performance, operations and vendor management

• Monitor and analyze performance: Track key performance indicators (KPIs) and analyze data to identify areas for improvement and inform decisions.
• Oversee technical performance: Ensure the platform is scalable, secure, and performs well by working closely with technical teams. Ensure platform decisions follow architectural, data, and cybersecurity standards
• Ensure compliance: Maintain compliance with brand, legal, and regulatory standards.
• Drive continuous improvement: Conduct experiments and implement updates to enhance user experience and platform capabilities.
• Governance: Approve changes, releases, and enhancements
• Manage key vendors delivering the platform or modules
• Ensure SLAs, performance, and contract obligations are met
• FinOps / Usage Monitoring / License Renewal:
• Own the platform budget — build vs run costs
• Evaluate financial impact of upgrades, licences, and new investments
• Ensure cost efficiency and scalability
• Audit support: Assisting with internal or external audits by providing required documentation, evidence, and responses to ensure compliance with standards and regulations
• Incident management: responding to and resolving unplanned disruptions or service issues to restore normal operations as quickly as possible.
• Demand management: forecasting, prioritizing, and controlling incoming requests for services or resources to ensure capacity aligns with business needs
• Problem management: structured approach to identifying the root causes of recurring incidents and implementing long-term fixes to prevent them from happening again

Job Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Data Science, or related field.
• 8 years of experience in data engineering, data platform development, or related technology roles, with proven technical leadership experience.
• Strong experience in data architecture, data modelling, Medallion Architecture, Data Lakes/Lakehouses, and scalable data platforms.
• Hands-on experience in building batch and real-time data pipelines, ETL/ELT, and data integrations.
• Proficiency in Python, SQL and relevant data engineering/cloud technologies such as Databricks, Spark, Kafka, Azure/AWS/GCP.
• Strong knowledge of data governance, data quality, security, performance optimisation, and cloud cost management.
• Proven stakeholder management skills, with the ability to translate business requirements into scalable technical solutions.
• Experience in vendor management, platform operations, incident/problem management, and audit/compliance.
• Strong leadership skills with experience in mentoring and developing data engineering teams.
• Excellent communication, analytical, problem-solving, and project management skills.