Azure Data engineer + AI - 12th Aug (Wednesday) - Bengaluru- Virtual

Tata Consultancy Services; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Bengaluru, India, is seeking an experienced data engineering professional to design and implement robust data pipelines using Azure Data Factory, Azure Databricks, and Microsoft Fabric. The role involves preparing data lakes for AI workloads and ensuring governance with Purview and Unity Catalog.

The ideal candidate will have strong Python/PySpark skills, expertise in Spark SQL, and experience with Azure Synapse Analytics.

Qualifications

• Experience designing end-to-end data pipelines using Azure services.
• Strong programming skills in Python, PySpark, Spark SQL and SQL.
• Knowledge of AI/GenAI integration and data governance concepts.

Responsibilities

• Design and build ETL/ELT data pipelines using Azure Data Factory, Azure Databricks, and Microsoft Fabric.
• Prepare and structure data lakes/lakehouses for Azure OpenAI, vector databases, and RAG architectures.
• Implement medallion architectures and ensure data quality and governance with Purview or Unity Catalog.
• Collaborate with data scientists, AI engineers, and analysts to operationalize AI models.

Skills

Advanced data pipeline

Collaboration

Problem solving

Education

Bachelor's degree in a related field

Tools

Azure Data Factory

Azure Databricks

Spark SQL

Python

PySpark

Azure Synapse Analytics

Microsoft Fabric

Microsoft Purview

Unity Catalog

Job description

Key Responsibilities

• Data Pipeline & Architecture: Design and build robust ETL/ELT data pipelines using Azure Data Factory, Azure Databricks, and Microsoft Fabric to process batch and real-time streaming data. [1, 2]
• AI & LLM Data Integration: Prepare, clean, and structure enterprise data lakes/lakehouses to feed Azure OpenAI, vector databases, and Retrieval-Augmented Generation (RAG) architectures. [1]
• Data Modeling & Governance: Implement medallion architectures (Bronze, Silver, Gold), ensure data quality, and manage data governance/lineage using tools like Microsoft Purview or Unity Catalog. [1, 2]
• Collaboration: Partner with data scientists, AI engineers, and business analysts to operationalize machine learning models and AI agents. [1]

Required Skills & Qualifications

• Cloud Platforms: Hands-on experience with Microsoft Azure (ADF, Azure Databricks, Azure Synapse Analytics, ADLS Gen2, and Microsoft Fabric).
• Programming & Querying: Advanced proficiency in Python, PySpark, Spark SQL, and complex SQL.
• AI & GenAI Familiarity: Knowledge of vector embeddings, semantic search, prompt grounding, and integrating Azure AI / OpenAI endpoints.