Databricks Architect

Avanade Inc.; Kuala Lumpur; On-site; MYR 200,000 - 250,000; Full time

Job summary

A global technology consultancy in Kuala Lumpur is looking for a Data & AI professional to lead architectural initiatives and optimize performance within the enterprise data platform. The ideal candidate will define strategies using Databricks, engage with stakeholders, and provide mentorship to engineers, while driving innovation and ensuring data governance.

Responsibilities

• Define and own the end-to-end technical architecture for the enterprise data platform.
• Design and implement Lakehouse architecture using Delta Lake and Databricks.
• Architect and enforce data governance and security policies.
• Provide technical leadership and mentorship to a team of engineers.
• Lead major performance tuning and cost optimization initiatives.
• Integrate Databricks with core cloud services and other enterprise systems.
• Engage stakeholders to translate complex requirements into technical designs.
• Define the DevOps/DataOps strategy for the Databricks environment.
• Evaluate new technologies to drive platform improvement.

Job description

Summary

Our talented Data & AI Practice is made up of globally recognized experts - and there’s room for more analytical and ambitious data professionals. If you’re passionate about helping clients make better data-driven decisions to tackle their most complex business issues, let’s talk. Take your skills to a new level and launch a career where you can truly do what matters.

Key Responsibilities

• Architectural Leadership: Define and own the end-to-end technical architecture and roadmap for the enterprise data platform on Databricks, including data ingestion, transformation, storage, and consumption layers.
• Lakehouse Design: Design and champion the implementation of the Lakehouse architecture utilizing Delta Lake, Databricks Unity Catalog, and Databricks SQL Warehouse to support all data, analytics, and AI/ML initiatives.
• Data Governance & Security: Architect and enforce enterprise-level data governance, security, and access control policies using Unity Catalog (e.g., fine-grained access, lineage tracking, auditing).
• Technical Guidance & Mentorship: Provide technical leadership, guidance, and mentorship to a team of Databricks Engineers. Conduct architectural reviews, code audits, and ensure adherence to best practices and standards.
• Performance and Cost Optimization: Define strategies for, and lead, major performance tuning and cost optimization initiatives across all Databricks workloads, clusters, and Delta Lake storage.
• Cloud Integration: Lead the integration of Databricks with core cloud services (AWS, Azure, or GCP) and other enterprise systems (e.g., data catalogs, BI tools, ML platforms).
• Stakeholder Collaboration: Engage with executive stakeholders, data scientists, and business leaders to understand complex requirements and translate them into robust, scalable technical designs and solution blueprints.
• DataOps and Automation Strategy: Define the DevOps/DataOps strategy for the Databricks environment, including continuous integration/continuous delivery (CI/CD) pipelines and Infrastructure-as-Code (IaC) using tools like Terraform or Databricks Asset Bundles.
• Innovation: Evaluate new Databricks features, open-source technologies (e.g., MLflow), and industry trends to drive continuous platform improvement and competitive advantage.

Equal Employment Opportunity Statement

All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

Job candidates will not be obligated to disclose sealed or expunged records of conviction or arrest as part of the hiring process.

Avanade is committed to providing veteran employment opportunities to our service men and women.

Please read Avanade's Hiring and Privacy Statement for more information on how we process your data during the Recruiting and Hiring process.