Lead / Senior data engineer: Python / AWS / Databricks

Capcon; Kuala Lumpur; Hybrid; MYR 120,000 - 180,000; Full time

Job summary

A dynamic tech company in Kuala Lumpur is seeking a Lead Data Engineer to design and enhance their cloud data architecture. You’ll leverage your expertise in Databricks, Python, SQL, and AWS to streamline data processes and mentor junior engineers. Join a diverse team aiming to drive innovation in the aviation sector by optimizing data utilization for better decision-making.

Qualifications

• 8+ years of experience in data engineering with leadership skills.
• Proven track record in building data solutions, not just maintenance.
• Strong numerical intuition and comfort with data.

Responsibilities

• Design and improve cloud-based data architecture for scalability and security.
• Develop and manage reliable data pipelines, automation and CI/CD workflows.
• Lead technical discussions and mentor junior engineers.

Skills

Databricks

Python

SQL

AWS

Communication

Problem Solving

Job description

Why are they awesome?

• Data driven technology spin off from a leading aviation strategy, logistics and air cargo consultancy
• Building data-driven SaaS tools that help airlines make informed commercial decisions from; routes, maintenance, demand planning, operations and forecast models

Products / Projects

• 2 core SaaS products that are mature - actively used by customers and continuously evolving.
• 1 successful data product is already in place that provides market insights from the supply side. A new data product targeting the demand side is currently in its initial phase, with significant development and scaling planned for H2 2025 and into 2026

Current challenges

• Strategizing and planning efficient large-volume data extraction from various sources
• Optimizing and enhancing the existing data infrastructure, architecture and workflows
• Setting and driving technical standards across the data engineering function

Overall responsibilities

• Take ownership of designing and continuously improving the cloud-based data architecture (including AWS, EKS, and Databricks) to support scalability, performance, and security needs
• Develop and upkeep of reliable data pipelines, automation and CI/CD workflows
• Lead technical discussions - identify and dissect complex problems and effectively communicate solutions across tech and non-tech stakeholders
• Coach & mentor junior engineers by leading with examples

Team Org & Culture

• Reporting to the Head of Data, based in Netherlands (you will still work KL hours)
• Entrepreneurial and ambitious team from Malaysia and Europe, which include ex management consultants
• The engineering hub in KL (15 staff) supports software development, data engineering and analytics - engineers who are results-driven, proactive and ready to wear multiple hats

Must Have

• Databricks, Python, SQL and AWS - or any cloud platforms like GCP / Azure etc.
• Experience in building rather than only maintaining
• Excellent communication and problem-solving skills

Nice to have

• Strong numerical intuition and comfort working with data
• Experience in the cargo or logistics industry

Grade / Level

• Lead Data Engineer (8+ years and above, must have lead experience)
• Senior Data Engineer (5+ years and above)

Work Arrangement

• Hybrid with 3 days work from office
• Based in Bangsar South / KL Gateway (Close to MRT / LRT)