Databricks Data Engineering Architect

Blue Cloud Softech Solutions Limited; Bengaluru; On-site; INR 3,500,000 - 7,000,000; Full time

Job summary

Blue Cloud Softech Solutions Limited in Bengaluru seeks an experienced Data Architect/Engineer to own Databricks architecture and Lakehouse solutions across AWS.

You will drive end-to-end delivery, enforce governance and security, design scalable data pipelines, and mentor teams. Strong AWS, Databricks, and Lakehouse expertise are required; excellent communication is essential for cross-functional collaboration.

Qualifications

• 9+ years of Data Engineering/Data Architecture experience with 4+ years hands-on Databricks.
• Expertise in Databricks, Delta Lake, Spark, SQL, Python/PySpark, and Lakehouse architecture.
• Strong knowledge of Unity Catalog, Data Governance, Data Security, RBAC, lineage, auditing, data quality, and access controls.
• Experience designing enterprise batch, streaming, and reusable data pipelines.
• 1+ years with Databricks AI/BI, Genie, Apps and other emerging services.
• Strong AWS experience, including S3, Lambda, Glue, CloudWatch, RDS, and Redshift.
• Understanding of data modeling, performance optimization, scalability, reliability, security, and cost management.
• Proven ability to own complex technical initiatives and collaborate across teams.
• Excellent communication, architecture, problem-solving, and technical leadership.
• Bachelor’s degree in Computer Science, Information Technology, Data Engineering, or a related field.

Responsibilities

• Define and drive the Databricks Data Engineering architecture and strategy.
• Take end-to-end ownership of Databricks solutions from architecture to production.
• Define and enforce policy-based governance and cost controls across Databricks workloads.
• Establish RBAC and least-privilege security across Unity Catalog and related environments.
• Design reusable data ingestion and transformation frameworks using Databricks, Spark, Delta Lake, SQL, Python/PySpark, and Lakeflow.
• Architect scalable Lakehouse solutions on Databricks and AWS for batch, streaming, analytics, and AI use cases.
• Design and support Agentic AI and Databricks Apps development, deployment, and monitoring.
• Establish Data Engineering Governance and Security with Unity Catalog (RBAC, data access, lineage, auditing, quality).
• Define engineering and governance standards across Dev, QA, and Production.
• Collaborate with Security, Cloud, Infrastructure, Data Governance, Data Science, BI, applications, and business teams.

Skills

Databricks

Data Architecture

Lakehouse

Spark

Python/PySpark

SQL

Data Governance

RBAC & Unity Catalog

AWS

Leadership

Education

Bachelor’s degree in Computer Science, IT, Data Engineering

Tools

Delta Lake

Lakeflow

Databricks AI/BI

dbt

Tableau

Power BI

Job description

• Define and drive the Databricks Data Engineering architecture and strategy, aligned with enterprise Data & Analytics standards.
• Take end-to-end ownership of Databricks solutions from architecture and design through implementation, production deployment, support, and optimization.
• Define and enforce policy-based governance and cost controls across Databricks workloads, including Apps, Agents, compute, AI/ML resources, and AI Gateway.
• Establish and enforce RBAC and least-privilege security across Unity Catalog, Databricks Apps, Agents, AI Gateway, service principals, and Dev/QA/Production environments.
• Design reusable data ingestion and transformation frameworks using Databricks, Spark, Delta Lake, SQL, Python/PySpark, and Lakeflow.
• Architect scalable Lakehouse solutions on Databricks and AWS supporting batch, streaming, near-real-time, analytics, and AI use cases.
• Design and support Agentic AI and Databricks Apps development, including architecture, deployment, integration, security, monitoring, and production readiness.
• Establish and enforce Data Engineering Governance and Security using Unity Catalog, including RBAC, least privilege, service principals, data access controls, lineage, auditing, and data quality.
• Define engineering and governance standards across Dev, QA, and Production, including Git , Branches etc.
• Design data integration patterns for SAP/ERP, databases, APIs, files, streaming platforms, and cloud services.
• Design and evaluate data models and serving patterns for BI, analytics, Data Science, AI, applications, and downstream consumers.
• Establish standards for Databricks compute, performance optimization, monitoring, reliability, and cost management.
• Proactively identify and resolve architecture, security, governance, performance, and production issues.
• Work closely with Security, Cloud, Infrastructure, Data Governance, Data Science, BI, application, and business teams to deliver enterprise solutions.
• Provide technical leadership, architecture reviews, hands‑on prototyping, engineering standards, and mentoring.
• Demonstrate a strong ownership mindset by independently driving initiatives, managing dependencies and risks, and ensuring successful delivery.

You Must Have

• 9+ years of Data Engineering/Data Architecture experience with 4+ years of hands-on Databricks experience.
• Strong experience with Databricks, Delta Lake, Spark, SQL, Python/PySpark, and Lakehouse architecture.
• Strong knowledge of Unity Catalog, Data Governance, Data Security, RBAC, lineage, auditing, data quality, and access controls.
• Experience designing enterprise batch, streaming, and reusable data pipelines.
• 1+ years of Experience with Databricks AI/BI, Genie, Apps, and other emerging services .
• Strong AWS experience, including S3, Lambda, Glue, CloudWatch, RDS, and Redshift.
• Strong understanding of data modeling, performance optimization, scalability, reliability, security, and cost management.
• Proven ability to own complex technical initiatives, solve production issues, and work effectively across multiple technical and business teams.
• Excellent communication, architecture, problem-solving, and technical leadership skills.
• Bachelor’s degree in Computer Science, Information Technology, Data Engineering, or a related field.

Nice to Have

• Databricks and/or AWS certifications.
• Experience with dbt, Sigma, Tableau, or Power BI.
• Experience with enterprise FinOps, cost/usage monitoring, and platform governance.
• SAP integration or functional experience.