Databricks Technical Lead

Avanade Inc.; Kuala Lumpur; On-site; MYR 200,000 - 250,000; Full time

Job summary

A global consulting firm in Kuala Lumpur is seeking an experienced data professional to lead the design of data architectures and develop high-quality ETL/ELT solutions using PySpark and Scala. You will drive performance optimization and ensure adherence to best practices across the team. This is an opportunity to make significant impact by helping clients navigate complex data challenges with efficiency and excellence.

Qualifications

• Proficiency in PySpark and Scala for ETL/ELT development.
• Experience in utilizing Databricks features effectively.
• Understanding of performance optimization for Spark jobs.

Responsibilities

• Define and implement robust data architectures.
• Write high-quality production-grade code.
• Lead performance tuning and optimization efforts.
• Enforce technical standards and testing frameworks.

Skills

Data architectures

PySpark

Scala

SQL

Performance tuning

CI/CD pipelines

DataOps

Tools

Databricks

Job description

Summary

Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly do what matters.

Key Responsibilities

• Lead the Design: Define and implement robust data architectures utilizing the Databricks ecosystem, including Delta Lake, Unity Catalog, Delta Live Tables (DLT), and Databricks Workflows.
• Hands‑on Development: Serve as the most senior developer, writing high-quality, production‑grade code in PySpark/Scala and SQL for complex batch and streaming ETL/ELT pipelines.
• Performance Optimization: Lead performance tuning and optimization efforts for large‑scale Spark jobs, ensuring efficient cluster utilization and cost management.
• Standards & Best Practices: Define and enforce technical standards, code quality, testing frameworks (unit, integration), and DataOps/CI/CD pipelines for the engineering team.

Equal Employment Opportunity Statement

All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

Job candidates will not be obligated to disclose sealed or expunged records of conviction or arrest as part of the hiring process.

Avanade is committed to providing veteran employment opportunities to our service men and women.

Please read Avanade's Hiring and Privacy Statement for more information on how we process your data during the Recruiting and Hiring process.