Databricks Data Engineering Architect

Blue Cloud Softech Solutions Limited; India; On-site; INR 3,500,000 - 5,500,000; Full time

Job summary

Blue Cloud Softech Solutions Limited is seeking a Databricks Data Engineering Architect to define and drive the Databricks data engineering architecture and strategy, aligned with enterprise standards. You will own end-to-end Databricks solutions from design through production deployment and optimization.

The role requires 9+ years in data engineering/architecture, deep Databricks and AWS experience, governance, security, and mentoring; strong communication and leadership to deliver

Qualifications

• 9+ years of Data Engineering/Data Architecture experience with 4+ years of hands-on Databricks experience.
• Strong experience with Databricks, Delta Lake, Spark, SQL, Python/PySpark, and Lakehouse architecture.
• Strong knowledge of Unity Catalog, Data Governance, Data Security, RBAC, lineage, auditing, data quality, and access controls.
• Experience designing enterprise batch, streaming, and reusable data pipelines.
• 1+ years of Experience with Databricks AI/BI, Genie, Apps, and other emerging services .
• Strong AWS experience, including S3, Lambda, Glue, CloudWatch, RDS, and Redshift.
• Strong experience in Git, Dev/QA/Production deployment patterns
• Strong understanding of data modeling, performance optimization, scalability, reliability, security, and cost management.
• Proven ability to own complex technical initiatives, solve production issues, and work effectively across multiple technical and business teams.
• Excellent communication, architecture, problem-solving, and technical leadership skills.
• Bachelor s degree in Computer Science, Information Technology, Data Engineering, or a related field.

Responsibilities

• Define and drive the Databricks Data Engineering architecture and strategy, aligned with enterprise Data & Analytics standards.
• Take end-to-end ownership of Databricks solutions from architecture and design through implementation, production deployment, support, and optimization.
• Define and enforce policy-based governance and cost controls across Databricks workloads, including Apps, Agents, compute, AI/ML resources, and AI Gateway.
• Establish and enforce RBAC and least-privilege security across Unity Catalog, Databricks Apps, Agents, AI Gateway, service principals, and Dev/QA/Production environments.
• Design reusable data ingestion and transformation frameworks using Databricks, Spark, Delta Lake, SQL, Python/PySpark, and Lake flow.
• Architect scalable Lakehouse solutions on Databricks and AWS supporting batch, streaming, near-real-time, analytics, and AI use cases.
• Design and support Agentic AI and Databricks Apps development, including architecture, deployment, integration, security, monitoring, and production readiness.
• Establish and enforce Data Engineering Governance and Security using Unity Catalog, including RBAC, least privilege, service principals, data access controls, lineage, auditing, and data quality.
• Define engineering and governance standards across Dev, QA, and Production, including Git , Branches etc.
• Design data integration patterns for SAP/ERP, databases, APIs, files, streaming platforms, and cloud services.
• Design and evaluate data models and serving patterns for BI, analytics, Data Science, AI, applications, and downstream consumers.
• Establish standards for Databricks compute, performance optimization, monitoring, reliability, and cost management.
• Proactively identify and resolve architecture, security, governance, performance, and production issues.
• Work closely with Security, Cloud, Infrastructure, Data Governance, Data Science, BI, application, and business teams to deliver enterprise solutions.
• Provide technical leadership, architecture reviews, hands‑on prototyping, engineering standards, and mentoring.
• Demonstrate a strong ownership mindset by independently driving initiatives, managing dependencies and risks, and ensuring successful delivery.

Skills

Data engineering

Data architecture

AWS

Leadership

Communication

Education

Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field

Tools

Databricks

Delta Lake

Spark

SQL

Python/PySpark

Git

Unity Catalog

Lakehouse

dbt

AWS (S3, Lambda, Glue, CloudWatch, RDS, Redshift)

Job description

Databricks Data Engineering Architect Primary Responsibilities

• Define and drive the Databricks Data Engineering architecture and strategy, aligned with enterprise Data & Analytics standards.
• Take end-to-end ownership of Databricks solutions from architecture and design through implementation, production deployment, support, and optimization.
• Define and enforce policy-based governance and cost controls across Databricks workloads, including Apps, Agents, compute, AI/ML resources, and AI Gateway.
• Establish and enforce RBAC and least-privilege security across Unity Catalog, Databricks Apps, Agents, AI Gateway, service principals, and Dev/QA/Production environments.
• Design reusable data ingestion and transformation frameworks using Databricks, Spark, Delta Lake, SQL, Python/PySpark, and Lake flow.
• Architect scalable Lakehouse solutions on Databricks and AWS supporting batch, streaming, near-real-time, analytics, and AI use cases.
• Design and support Agentic AI and Databricks Apps development, including architecture, deployment, integration, security, monitoring, and production readiness.
• Establish and enforce Data Engineering Governance and Security using Unity Catalog, including RBAC, least privilege, service principals, data access controls, lineage, auditing, and data quality.
• Define engineering and governance standards across Dev, QA, and Production, including Git , Branches etc.
• Design data integration patterns for SAP/ERP, databases, APIs, files, streaming platforms, and cloud services.
• Design and evaluate data models and serving patterns for BI, analytics, Data Science, AI, applications, and downstream consumers.
• Establish standards for Databricks compute, performance optimization, monitoring, reliability, and cost management.
• Proactively identify and resolve architecture, security, governance, performance, and production issues.
• Work closely with Security, Cloud, Infrastructure, Data Governance, Data Science, BI, application, and business teams to deliver enterprise solutions.
• Provide technical leadership, architecture reviews, hands‑on prototyping, engineering standards, and mentoring.
• Demonstrate a strong ownership mindset by independently driving initiatives, managing dependencies and risks, and ensuring successful delivery.

You Must Have

• 9+ years of Data Engineering/Data Architecture experience with 4+ years of hands‑on Databricks experience.
• Strong experience with Databricks, Delta Lake, Spark, SQL, Python/PySpark, and Lakehouse architecture.
• Strong knowledge of Unity Catalog, Data Governance, Data Security, RBAC, lineage, auditing, data quality, and access controls.
• Experience designing enterprise batch, streaming, and reusable data pipelines.
• 1+ years of Experience with Databricks AI/BI, Genie, Apps, and other emerging services .
• Strong AWS experience, including S3, Lambda, Glue, CloudWatch, RDS, and Redshift.
• Strong experience in Git, Dev/QA/Production deployment patterns
• Strong understanding of data modeling, performance optimization, scalability, reliability, security, and cost management.
• Proven ability to own complex technical initiatives, solve production issues, and work effectively across multiple technical and business teams.
• Excellent communication, architecture, problem-solving, and technical leadership skills.
• Bachelor s degree in Computer Science, Information Technology, Data Engineering, or a related field.

Nice to Have

• Databricks and/or AWS certifications.
• Experience with dbt, Sigma, Tableau, or Power BI.
• Experience with enterprise FinOps, cost/usage monitoring, and platform governance.
• SAP integration or functional experience.