Databricks Data Engineering Architect

Celebal Technologies; Dadri, Jaipur, Bengaluru; On-site; INR 2,500,000 - 5,200,000; Full time

Job summary

Celebal Technologies in India seeks a Senior Databricks Architect with 9+ years of data engineering and 4+ years of hands-on Databricks experience to own end-to-end Databricks architecture, governance, and Lakehouse implementations across AWS. You will lead enterprise data pipelines, RBAC, cost controls, and data governance frameworks while collaborating with Security, Cloud, Data Science, and BI teams.

The ideal candidate demonstrates strong communication, leadership, and problem-solving

Qualifications

• 9+ years of Data Engineering/Data Architecture experience with 4+ years of hands-on Databricks experience.
• Strong experience with Databricks, Delta Lake, Spark, SQL, Python/PySpark, and Lakehouse architecture.
• Experience designing enterprise batch, streaming, and reusable data pipelines.
• Strong knowledge of Unity Catalog, Data Governance, Data Security, RBAC, lineage, auditing, data quality, and access controls.
• Excellent communication, architecture, problem-solving, and technical leadership skills.

Responsibilities

• Define and drive the Databricks Data Engineering architecture and strategy, aligned with enterprise standards.
• Take end-to-end ownership of Databricks solutions from architecture through deployment and optimization.
• Define and enforce policy-based governance and cost controls across workloads, including AI Gateway.
• Establish and enforce RBAC and least-privilege security across Unity Catalog and environments.
• Design reusable data ingestion and transformation frameworks using Databricks, Spark, Delta Lake, SQL, Python/PySpark, and Lakeflow.
• Architect scalable Lakehouse solutions on Databricks and AWS for batch, streaming, near-real-time, analytics, and AI use cases.
• Design and support Agentic AI and Databricks Apps development, including deployment and monitoring.
• Establish Data Engineering Governance and Security using Unity Catalog, including data access controls and lineage.
• Define engineering/governance standards across Dev, QA, and Production, including Git workflows.
• Design data integration patterns for SAP/ERP, databases, APIs, files, streaming, and cloud services.
• Define data models and serving patterns for BI, analytics, Data Science, AI, applications, and downstream consumers.
• Set standards for compute, performance optimization, monitoring, reliability, and cost management.
• Proactively resolve architecture, security, governance, performance, and production issues.
• Collaborate with Security, Cloud, Infrastructure, Data Governance, Data Science, BI, and business teams to deliver enterprise solutions.
• Provide technical leadership, architecture reviews, prototyping, and mentoring.
• Demonstrate ownership mindset by driving initiatives and ensuring successful delivery.

Job description

Role & responsibilities

• Define and drive the Databricks Data Engineering architecture and strategy, aligned with enterprise Data & Analytics standards.
• Take end-to-end ownership of Databricks solutions from architecture and design through implementation, production deployment, support, and optimization.
• Define and enforce policy-based governance and cost controls across Databricks workloads, including Apps, Agents, compute, AI/ML resources, and AI Gateway.
• Establish and enforce RBAC and least-privilege security across Unity Catalog, Databricks Apps, Agents, AI Gateway, service principals, and Dev/QA/Production environments.
• Design reusable data ingestion and transformation frameworks using Databricks, Spark, Delta Lake, SQL, Python/PySpark, and Lakeflow.
• Architect scalable Lakehouse solutions on Databricks and AWS supporting batch, streaming, near-real-time, analytics, and AI use cases.
• Design and support Agentic AI and Databricks Apps development, including architecture, deployment, integration, security, monitoring, and production readiness.
• Establish and enforce Data Engineering Governance and Security using Unity Catalog, including RBAC, least privilege, service principals, data access controls, lineage, auditing, and data quality.
• Define engineering and governance standards across Dev, QA, and Production, including Git , Branches etc.
• Design data integration patterns for SAP/ERP, databases, APIs, files, streaming platforms, and cloud services.
• Design and evaluate data models and serving patterns for BI, analytics, Data Science, AI, applications, and downstream consumers.
• Establish standards for Databricks compute, performance optimization, monitoring, reliability, and cost management.
• Proactively identify and resolve architecture, security, governance, performance, and production issues.
• Work closely with Security, Cloud, Infrastructure, Data Governance, Data Science, BI, application, and business teams to deliver enterprise solutions.
• Provide technical leadership, architecture reviews, hands‑on prototyping, engineering standards, and mentoring.
• Demonstrate a strong ownership mindset by independently driving initiatives, managing dependencies and risks, and ensuring successful delivery.

Preferred candidate profile

• 9+ years of Data Engineering/Data Architecture experience with 4+ years of hands‑on Databricks experience.
• Strong experience with Databricks, Delta Lake, Spark, SQL, Python/PySpark, and Lakehouse architecture.
• Strong knowledge of Unity Catalog, Data Governance, Data Security, RBAC, lineage, auditing, data quality, and access controls.
• Experience designing enterprise batch, streaming, and reusable data pipelines.
• 1+ years of Experience with Databricks AI/BI, Genie, Apps, and other emerging services .
• Strong AWS experience, including S3, Lambda, Glue, CloudWatch, RDS, and Redshift.
• Strong experience in Git, Dev/QA/Production deployment patterns
• Strong understanding of data modeling, performance optimization, scalability, reliability, security, and cost management.
• Proven ability to own complex technical initiatives, solve production issues, and work effectively across multiple technical and business teams.
• Excellent communication, architecture, problem‑solving, and technical leadership skills.
• Bachelors degree in Computer Science, Information Technology, Data Engineering, or a related field.

Nice to Have

• Databricks and/or AWS certifications.
• Experience with dbt, Sigma, Tableau, or Power BI.
• Experience with enterprise FinOps, cost/usage monitoring, and platform governance.
• SAP integration or functional experience.