Lead / Senior data engineer: Python / AWS / Databricks

Capcon Asia; Kuala Lumpur; Hybrid; MYR 100,000 - 140,000; Full time

Job summary

An innovative technology consulting firm in Kuala Lumpur is seeking a Senior Data Engineer with expertise in Databricks, Python, SQL, and AWS. You will design and enhance cloud-based data architectures, develop data pipelines, and lead a proactive team. Ideal candidates have strong communication skills and experience in a data-driven environment. This role offers a hybrid work arrangement, supporting both on-site and remote collaboration.

Qualifications

• 8+ years of experience for Lead Data Engineer, with leadership skills.
• 5+ years of experience for Senior Data Engineer.
• Experience building data pipelines rather than just maintaining them.

Responsibilities

• Design and improve cloud-based data architecture to support scalability.
• Develop and maintain reliable data pipelines and automation workflows.
• Lead technical discussions with stakeholders.
• Coach and mentor junior engineers.

Skills

Databricks

Python

SQL

AWS

Communication skills

Problem-solving skills

Job description

About the job lead / Senior data engineer: Python / AWS / Databricks

Why are they awesome?

• Data driven technology spin off from a leading aviation strategy, logistics and air cargo consultancy
• Building data-driven SaaS tools that help airlines make informed commercial decisions from; routes, maintenance, demand planning, operations and forecast models

Products / Projects

• 2 core SaaS products that are mature - actively used by customers and continuously evolving.
• 1 successful data product is already in place that provides market insights from the supply side. A new data product targeting the demand side is currently in its initial phase, with significant development and scaling planned for H2 2025 and into 2026
• Strategizing and planning efficient large-volume data extraction from various sources
• Optimizing and enhancing the existing data infrastructure, architecture and workflows
• Setting and driving technical standards across the data engineering function

Overall responsibilities

• Take ownership of designing and continuously improving the cloud-based data architecture (including AWS, EKS, and Databricks) to support scalability, performance, and security needs
• Develop and upkeep of reliable data pipelines, automation and CI/CD workflows
• Lead technical discussions - identify and dissect complex problems and effectively communicate solutions across tech and non-tech stakeholders
• Coach & mentor junior engineers by leading with examples

Team Org & Culture

• Reporting to the Head of Data, based in Netherlands (you will still work KL hours)
• Entrepreneurial and ambitious team from Malaysia and Europe, which include ex management consultants
• The engineering hub in KL (15 staff) supports software development, data engineering and analytics - engineers who are results-driven, proactive and ready to wear multiple hats

Must Have

• Databricks, Python, SQL and AWS - or any cloud platforms like GCP / Azure etc.
• Experience in building rather than only maintaining
• Excellent communication and problem-solving skills

Nice to have

• Strong numerical intuition and comfort working with data
• Experience in the cargo or logistics industry
• Lead Data Engineer (8+ years and above, must have lead experience)
• Senior Data Engineer (5+ years and above)

Work Arrangement

• Hybrid with 3 days work from office
• Based in Bangsar South / KL Gateway (Close to MRT / LRT)