Databricks Technical Lead

Avanade Inc.; Kuala Lumpur; On-site; MYR 80,000 - 120,000; Full time

Job summary

A leading technology consulting firm is seeking experienced data professionals to lead data architecture design and hands-on development. You'll be responsible for writing production-grade code for complex ETL/ELT pipelines using the Databricks ecosystem. The position emphasizes performance optimization and enforcing technical standards within the team. Ideal candidates should have expertise in PySpark/Scala and strong analytical skills. Join us to help clients drive data-driven decisions in a collaborative environment.

Responsibilities

• Define and implement robust data architectures utilizing the Databricks ecosystem.
• Write high-quality production-grade code in PySpark/Scala and SQL for ETL/ELT pipelines.
• Lead performance tuning and optimization for large-scale Spark jobs.
• Define and enforce technical standards and best practices for the engineering team.

Job description

Summary

Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly do what matters.

Key Responsibilities

• Lead the Design: Define and implement robust data architectures utilizing the Databricks ecosystem, including Delta Lake, Unity Catalog, Delta Live Tables (DLT), and Databricks Workflows.
• Hands‑on Development: Serve as the most senior developer, writing high‑quality, production‑grade code in PySpark/Scala and SQL for complex batch and streaming ETL/ELT pipelines.
• Performance Optimization: Lead performance tuning and optimization efforts for large‑scale Spark jobs, ensuring efficient cluster utilization and cost management.
• Standards & Best Practices: Define and enforce technical standards, code quality, testing frameworks (unit, integration), and DataOps/CI/CD pipelines for the engineering team.

Equal Employment Opportunity Statement

All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

Job candidates will not be obligated to disclose sealed or expunged records of conviction or arrest as part of the hiring process.

Avanade is committed to providing veteran employment opportunities to our service men and women.

Please read Avanade's Hiring and Privacy Statement for more information on how we process your data during the Recruiting and Hiring process.