Azure Data Engineer + AI

Tata Consultancy Services; Bengaluru; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Tata Consultancy Services in Bengaluru is seeking a data engineer to design and build robust ETL/ELT data pipelines using Azure Data Factory, Azure Databricks, and Microsoft Fabric for batch and real-time processing.

You will structure enterprise data lakes/lakehouses to feed Azure OpenAI, vector databases, and RAG architectures, while implementing medallion architectures and governance. Collaboration with data scientists and analysts is required.

Qualifications

• Hands-on experience with Azure data services and cloud platforms.
• Advanced Python and SQL skills for data integration and modeling.
• Knowledge of GenAI concepts, vector embeddings, and semantic search.

Responsibilities

• Design and build robust ETL/ELT data pipelines using Azure tools.
• Prepare data lakes/lakehouses for AI workloads and RAG architectures.
• Implement medallion data architectures and ensure data governance.
• Collaborate with data scientists and analysts to operationalize ML models.

Skills

Python

PySpark

Spark SQL

SQL

Tools

Azure Data Factory

Azure Databricks

Azure Synapse Analytics

ADLS Gen2

Microsoft Fabric

Job description

Job description

Key Responsibilities

• Data Pipeline & Architecture: Design and build robust ETL/ELT data pipelines using Azure Data Factory, Azure Databricks, and Microsoft Fabric to process batch and real-time streaming data. [1, 2]
• AI & LLM Data Integration: Prepare, clean, and structure enterprise data lakes/lakehouses to feed Azure OpenAI, vector databases, and Retrieval-Augmented Generation (RAG) architectures. [1]
• Data Modeling & Governance: Implement medallion architectures (Bronze, Silver, Gold), ensure data quality, and manage data governance/lineage using tools like Microsoft Purview or Unity Catalog. [1, 2]
• Collaboration: Partner with data scientists, AI engineers, and business analysts to operationalize machine learning models and AI agents. [1]

Required Skills & Qualifications

• Cloud Platforms: Hands-on experience with Microsoft Azure (ADF, Azure Databricks, Azure Synapse Analytics, ADLS Gen2, and Microsoft Fabric).
• Programming & Querying: Advanced proficiency in Python, PySpark, Spark SQL, and complex SQL.
• AI & GenAI Familiarity: Knowledge of vector embeddings, semantic search, prompt grounding, and integrating Azure AI / OpenAI endpoints.