Azure Data engineer + AI - 11 Sept (Friday) - Bengaluru- Video - 8+Yrs

Tata Consultancy Services; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Bengaluru is seeking a Senior Data Engineer to design and build robust ETL/ELT data pipelines using Azure Data Factory, Databricks, and Microsoft Fabric.

You will structure enterprise data lakes/lakehouses for feed into Azure OpenAI and RAG architectures, implement Bronze-Silver-Gold medallion models, and collaborate with data scientists to operationalize AI solutions.

Qualifications

• Experience designing ETL/ELT data pipelines in Microsoft Azure.
• Strong proficiency in Python and Spark-based querying.
• Experience with data governance and data quality in large-scale environments.

Responsibilities

• Design and build ETL/ELT pipelines using Azure Data Factory, Azure Databricks, and Microsoft Fabric.
• Structure data lakes/lakehouses to support Azure OpenAI, vector databases, and RAG architectures.
• Implement Bronze/Silver/Gold medallion architectures and govern data using Purview or Unity Catalog.
• Collaborate with data scientists, AI engineers, and analysts to operationalize ML/AI solutions.

Skills

Azure

Python

PySpark

Spark SQL

SQL

Azure OpenAI

Job description

Key Responsibilities

• Data Pipeline & Architecture: Design and build robust ETL/ELT data pipelines using Azure Data Factory, Azure Databricks, and Microsoft Fabric to process batch and real-time streaming data.
• AI & LLM Data Integration: Prepare, clean, and structure enterprise data lakes/lakehouses to feed Azure OpenAI, vector databases, and Retrieval-Augmented Generation (RAG) architectures.
• Data Modeling & Governance: Implement medallion architectures (Bronze, Silver, Gold), ensure data quality, and manage data governance/lineage using tools like Microsoft Purview or Unity Catalog.
• Collaboration: Partner with data scientists, AI engineers, and business analysts to operationalize machine learning models and AI agents.

Required Skills & Qualifications

• Cloud Platforms: Hands-on experience with Microsoft Azure (ADF, Azure Databricks, Azure Synapse Analytics, ADLS Gen2, and Microsoft Fabric).
• Programming & Querying: Advanced proficiency in Python, PySpark, Spark SQL, and complex SQL.
• AI & GenAI Familiarity: Knowledge of vector embeddings, semantic search, prompt grounding, and integrating Azure AI / OpenAI endpoints.