Azure Data engineer + AI - 21 Aug (Friday)- Bengaluru- Video Interview

Tata Consultancy Services; Bengaluru; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Tata Consultancy Services in Bengaluru is seeking a Data Engineering professional to design robust ETL/ELT pipelines on Azure, integrating AI/LLM data flows and governance across data lakehouses.

The role emphasizes medallion data architectures, real-time streaming, and collaboration with data scientists and analysts to operationalize ML models and AI agents.

Qualifications

• Hands-on experience with Microsoft Azure, including ADF, Azure Databricks, Synapse, ADLS Gen2, and Microsoft Fabric.
• Proficient in Python, PySpark, Spark SQL, and SQL.
• Knowledge of vector embeddings, semantic search, prompt grounding, and Azure OpenAI endpoints.

Responsibilities

• Data Pipeline & Architecture: Design and build robust ETL/ELT data pipelines using Azure Data Factory, Azure Databricks, and Microsoft Fabric to process batch and real-time streaming data.
• AI & LLM Data Integration: Prepare, clean, and structure enterprise data lakes/lakehouses to feed Azure OpenAI, vector databases, and Retrieval-Augmented Generation (RAG) architectures.
• Data Modeling & Governance: Implement medallion architectures (Bronze, Silver, Gold), ensure data quality, and manage data governance/lineage using tools like Microsoft Purview or Unity Catalog.
• Collaboration: Partner with data scientists, AI engineers, and business analysts to operationalize machine learning models and AI agents.

Skills

Python

PySpark

Spark SQL

SQL

Vector embeddings

Semantic search

Prompt grounding

Azure OpenAI endpoints

Tools

Azure Data Factory

Azure Databricks

Microsoft Fabric

Purview

Unity Catalog

Job description

Key Responsibilities

• Data Pipeline & Architecture: Design and build robust ETL/ELT data pipelines using Azure Data Factory, Azure Databricks, and Microsoft Fabric to process batch and real-time streaming data. [1, 2]
• AI & LLM Data Integration: Prepare, clean, and structure enterprise data lakes/lakehouses to feed Azure OpenAI, vector databases, and Retrieval-Augmented Generation (RAG) architectures. [1]
• Data Modeling & Governance: Implement medallion architectures (Bronze, Silver, Gold), ensure data quality, and manage data governance/lineage using tools like Microsoft Purview or Unity Catalog. [1, 2]
• Collaboration: Partner with data scientists, AI engineers, and business analysts to operationalize machine learning models and AI agents. [1]

Required Skills & Qualifications

• Cloud Platforms: Hands-on experience with Microsoft Azure (ADF, Azure Databricks, Azure Synapse Analytics, ADLS Gen2, and Microsoft Fabric).
• Programming & Querying: Advanced proficiency in Python, PySpark, Spark SQL, and complex SQL.
• AI & GenAI Familiarity: Knowledge of vector embeddings, semantic search, prompt grounding, and integrating Azure AI / OpenAI endpoints.