AWS Databricks Architect

Tata Consultancy Services; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Tata Consultancy Services is seeking an experienced AWS Databricks Architect with 12+ years of IT experience to own Databricks Lakehouse architecture on AWS. You will drive data engineering, data science, and GenAI solutioning while providing technical leadership across engagements and client interactions.

The role focuses on end-to-end design, governance, cost optimization, and DevOps integration, working with Delta Lake, Unity Catalog, and LangChain/LlamaIndex.

Qualifications

• Experience designing and delivering enterprise-scale Data & AI platforms on AWS using Databricks.
• Own end-to-end Databricks Lakehouse architecture on AWS across ingestion, processing, storage, and governance.
• Lead data engineering, data science, and GenAI solutioning for enterprise engagements.

Responsibilities

• Own end-to-end architecture, design, and delivery of Databricks Lakehouse platforms on AWS across ingestion, processing, storage, consumption, and governance layers.
• Define reference architectures and blueprints using Delta Lake, Unity Catalog, and the Medallion (Bronze/Silver/Gold) framework.
• Lead design of scalable batch and streaming ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Structured Streaming.
• Architect ML and Data Science workloads using Databricks ML, MLflow, and feature stores; drive MLOps best practices.
• Design and deliver GenAI solutions including RAG pipelines, vector search, prompt engineering, and LLM integrations using LangChain / LlamaIndex.
• Drive Spark performance tuning, cluster sizing, cost optimization, and FinOps for large workloads.
• Define data governance, security, lineage, monitoring, and CI/CD standards using Unity Catalog, IAM, Terraform, and DevOps toolchains.
• Engage with clients in pre-sales, solution discussions, RFP/RFI responses, PoCs, and workshops.
• Provide technical leadership, mentor engineers and leads, review solutions, and enforce architecture best practices.
• Collaborate with AWS partner teams, product engineering, and business stakeholders to align architecture with business outcomes.

Skills

Databricks on AWS

Delta Lake & Unity Catalog

PySpark

Spark SQL

LLM integration

MLOps

GenAI/LLM

Terraform

CI/CD

Security & RBAC

Tools

Databricks

Delta Lake

Unity Catalog

Spark

Python

LangChain

LlamaIndex

Terraform

GitHub Actions

Jenkins/Azure DevOps

Job description

The TCS AWS Business Unit is a dedicated, full-stack group within Tata Consultancy Services launched to help global enterprises accelerate cloud migration, application modernization, and AI-driven digital transformation using Amazon Web Services

Role: AWS Databricks Architect

We are hiring an experienced AWS Databricks Architect with 12+ years of overall IT experience and strong hands-on background in designing and delivering enterprise-scale Data & AI platforms on AWS using Databricks.

The ideal candidate will own the end-to-end Databricks Lakehouse architecture on AWS, drive Data Engineering, Data Science, and Generative AI solutioning, provide technical leadership across multiple engagements, and engage with clients and business stakeholders as an SME.

Responsibility:

• Own end-to-end architecture, design, and delivery of Databricks Lakehouse platforms on AWS across ingestion, processing, storage, consumption, and governance layers.


• Define reference architectures and blueprints using Delta Lake, Unity Catalog, and the Medallion (Bronze/Silver/Gold) framework.


• Lead design of scalable batch and streaming ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Structured Streaming.


• Architect ML and Data Science workloads using Databricks ML, MLflow, and feature stores; drive MLOps best practices.


• Design and deliver GenAI solutions including RAG pipelines, vector search, prompt engineering, and LLM integrations using LangChain / LlamaIndex.


• Drive Spark performance tuning, cluster sizing, cost optimization, and FinOps for large workloads.


• Define data governance, security, lineage, monitoring, and CI/CD standards using Unity Catalog, IAM, Terraform, and DevOps toolchains.


• Engage with clients in pre-sales, solution discussions, RFP/RFI responses, PoCs, and workshops.


• Provide technical leadership, mentor engineers and leads, review solutions, and enforce architecture best practices.


• Collaborate with AWS partner teams, product engineering, and business stakeholders to align architecture with business outcomes.

Must Have Skills:

• Databricks on AWS – workspace design, cluster policies, jobs, workflows, and administration.


• Delta Lake, Unity Catalog, Databricks SQL, and Lakehouse architecture patterns.


• PySpark, Spark SQL, Python, advanced SQL, and Spark performance tuning at scale.


• AWS Services: S3, Glue, IAM, Lambda, Athena, Redshift, Kinesis, CloudWatch, VPC, KMS.


• Batch and streaming pipeline design (Structured Streaming, Kinesis, Kafka).


• MLflow, model lifecycle management, and MLOps on Databricks.


• Generative AI / LLM integration – RAG architecture, prompt engineering, embeddings.


• LangChain / LlamaIndex and vector databases (e.g., Pinecone, OpenSearch, FAISS).


• Infrastructure as Code – Terraform; CI/CD using Git, GitHub Actions / Jenkins / Azure DevOps.


• Data governance, security, RBAC, encryption, and audit logging.

Good to have:

• MLOps / LLMOps practices at enterprise scale.


• Experience leading large enterprise modernization and cloud migration programs.


• Exposure to multi-cloud (AWS) Databricks deployments.

Location: Pan India

Experience: 12 to 20 years

Certification:

• Databricks Certified Data Engineer / ML / Architect and AWS Solutions Architect certifications.