Azure Databricks Architect

Tredence; Bengaluru; On-site; INR 4,000,000 - 7,500,000; Full time

Job summary

Tredence is seeking an experienced Azure Databricks Architect to lead enterprise data and AI initiatives on a Lakehouse platform. You will design end-to-end Databricks architectures and guide data engineering, ML teams, and governance practices across cloud environments.

Ideal candidates have 8–10 years in data engineering/architecture, strong Spark/Delta Lake expertise, and hands-on Databricks experience, with leadership capabilities and strong communication skills.

Qualifications

• 8-10 years of overall data engineering/architecture experience, with at least 4.5 years hands-on with Databricks.
• Deep expertise in Apache Spark, Delta Lake, and the Databricks Lakehouse architecture.
• Strong experience with Unity Catalog, workspace administration, and cluster/job management.
• Hands-on experience with at least one hyperscaler (Azure, AWS, or GCP) underlying the Databricks deployment.
• Proficiency in Python and/or Scala, and strong SQL skills.
• Experience designing batch and streaming data pipelines at enterprise scale.
• Solid understanding of data governance, lineage, and security practices in a Lakehouse environment.
• Strong communication skills and experience leading technical teams.

Responsibilities

• Design end-to-end Lakehouse architectures on Databricks, including ingestion, transformation, storage, and consumption layers.
• Define best practices for Delta Lake, Unity Catalog, and workspace governance across environments.
• Architect scalable data pipelines using Spark, Delta Live Tables, and Databricks Workflows.
• Establish security, access control, and data governance standards within Databricks and the underlying cloud platform (Azure/AWS/GCP).
• Guide performance tuning and cost optimization of Spark jobs and cluster configurations.
• Collaborate with data engineering, data science, and ML teams to enable a unified data and AI platform.
• Define CI/CD and DevOps practices for Databricks notebooks, jobs, and infrastructure (Repos, Terraform).
• Provide architectural oversight and mentorship to data engineering teams.
• Partner with stakeholders to align the Databricks platform roadmap with business and analytics needs.

Skills

Databricks

Apache Spark

Delta Lake

Unity Catalog

Workspace administration

Security governance

Python/Scala

SQL

Batch & streaming

Cloud (Azure/AWS/GCP)

CI/CD

Tools

Airflow

Azure Data Factory

Terraform

Job description

Role: Azure Databricks Architect

Exp Level: 8-10 Years

Work Location: Bangalore

Notice Period: Immediate Joiners

We are seeking a Databricks Architect to design and lead our enterprise data and AI platform built on Databricks. You will define the architecture for large-scale data engineering, analytics, and machine learning workloads on the Lakehouse platform.

Key Responsibilities

• Design end-to-end Lakehouse architectures on Databricks, including ingestion, transformation, storage, and consumption layers.
• Define best practices for Delta Lake, Unity Catalog, and workspace governance across environments.
• Architect scalable data pipelines using Spark, Delta Live Tables, and Databricks Workflows.
• Establish security, access control, and data governance standards within Databricks and the underlying cloud platform (Azure/AWS/GCP).
• Guide performance tuning and cost optimization of Spark jobs and cluster configurations.
• Collaborate with data engineering, data science, and ML teams to enable a unified data and AI platform.
• Define CI/CD and DevOps practices for Databricks notebooks, jobs, and infrastructure (Repos, Terraform).
• Provide architectural oversight and mentorship to data engineering teams.
• Partner with stakeholders to align the Databricks platform roadmap with business and analytics needs.

Required Skills & Qualifications

• 8-10 years of overall data engineering/architecture experience, with at least 4.5 years hands-on with Databricks.
• Deep expertise in Apache Spark, Delta Lake, and the Databricks Lakehouse architecture.
• Strong experience with Unity Catalog, workspace administration, and cluster/job management.
• Hands-on experience with at least one hyperscaler (Azure, AWS, or GCP) underlying the Databricks deployment.
• Proficiency in Python and/or Scala, and strong SQL skills.
• Experience designing batch and streaming data pipelines at enterprise scale.
• Solid understanding of data governance, lineage, and security practices in a Lakehouse environment.
• Strong communication skills and experience leading technical teams.

Preferred Qualifications (Good to Have)

• Databricks Certified Data Engineer Professional or Databricks Certified Solutions Architect.
• Experience with MLflow and enabling ML/AI workloads on Databricks.
• Exposure to orchestration tools such as Airflow or Azure Data Factory.
• Experience migrating legacy data warehouses to a Lakehouse architecture.