Senior Architect_Databricks

Tredence; Kolkata District, Pune District, Chennai; Hybrid; INR 3,500,000 - 7,500,000; Full time

Job summary

Tredence Inc. is seeking a senior data engineering/architecture leader to design end-to-end Databricks Lakehouse solutions. You will define governance, optimize Spark performance, and guide DevOps for notebooks, workflows, and Terraform deployments.

You will collaborate with data engineering, science, and ML teams to scale a unified data platform across cloud environments, ensuring security and cost efficiency.

Qualifications

• 10-14 years of overall data engineering/architecture experience, with at least 45 years hands-on with Databricks.
• Deep expertise in Apache Spark, Delta Lake, and the Databricks Lakehouse architecture.
• Strong experience with Unity Catalog, workspace administration, and cluster/job management.
• Hands-on experience with at least one hyperscaler (Azure, AWS, or GCP) underlying the Databricks deployment.
• Proficiency in Python and/or Scala, and strong SQL skills.
• Experience designing batch and streaming data pipelines at enterprise scale.
• Solid understanding of data governance, lineage, and security practices in a Lakehouse environment.
• Strong communication skills and experience leading technical teams.

Responsibilities

• Design end-to-end Lakehouse architectures on Databricks, including ingestion, transformation, storage, and consumption layers.
• Define best practices for Delta Lake, Unity Catalog, and workspace governance across environments.
• Architect scalable data pipelines using Spark, Delta Live Tables, and Databricks Workflows.
• Establish security, access control, and data governance standards within Databricks and the underlying cloud platform (Azure/AWS/GCP).
• Guide performance tuning and cost optimization of Spark jobs and cluster configurations.
• Collaborate with data engineering, data science, and ML teams to enable a unified data and AI platform.
• Define CI/CD and DevOps practices for Databricks notebooks, jobs, and infrastructure (Repos, Terraform).
• Provide architectural oversight and mentorship to data engineering teams.
• Partner with stakeholders to align the Databricks platform roadmap with business and analytics needs.

Skills

Databricks

Apache Spark

Delta Lake

Unity Catalog

Workspace administration

Cluster management

Python

Scala

SQL

Data governance

Security practices

Team leadership

Education

Bachelor's degree in CS/Engineering

Master's degree preferred

Tools

Terraform

Airflow

Azure Data Factory

Repos (Git)

Job description

Role & responsibilities

• Design end-to-end Lakehouse architectures on Databricks, including ingestion, transformation, storage, and consumption layers.


• Define best practices for Delta Lake, Unity Catalog, and workspace governance across environments.


• Architect scalable data pipelines using Spark, Delta Live Tables, and Databricks Workflows.


• Establish security, access control, and data governance standards within Databricks and the underlying cloud platform (Azure/AWS/GCP).


• Guide performance tuning and cost optimization of Spark jobs and cluster configurations.


• Collaborate with data engineering, data science, and ML teams to enable a unified data and AI platform.


• Define CI/CD and DevOps practices for Databricks notebooks, jobs, and infrastructure (Repos, Terraform).


• Provide architectural oversight and mentorship to data engineering teams.


• Partner with stakeholders to align the Databricks platform roadmap with business and analytics needs.

Required Skills & Qualifications

• 10-14 years of overall data engineering/architecture experience, with at least 45 years hands-on with Databricks.


• Deep expertise in Apache Spark, Delta Lake, and the Databricks Lakehouse architecture.


• Strong experience with Unity Catalog, workspace administration, and cluster/job management.


• Hands-on experience with at least one hyperscaler (Azure, AWS, or GCP) underlying the Databricks deployment.


• Proficiency in Python and/or Scala, and strong SQL skills.


• Experience designing batch and streaming data pipelines at enterprise scale.


• Solid understanding of data governance, lineage, and security practices in a Lakehouse environment.


• Strong communication skills and experience leading technical teams.

Preferred Qualifications (Good to Have)

• Databricks Certified Data Engineer Professional or Databricks Certified Solutions Architect.


• Experience with MLflow and enabling ML/AI workloads on Databricks.


• Exposure to orchestration tools such as Airflow or Azure Data Factory.


• Experience migrating legacy data warehouses to a Lakehouse architecture.

Education

Bachelor's degree in Computer Science, Engineering, or a related field (Master's preferred).