Azure Data Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 4,000,000 - 6,500,000; Full time

Job summary

TATA Consultancy Services is seeking an Azure Databricks Data Engineer to design and build scalable batch and streaming data pipelines. You will work with Azure Data Lake, Databricks, Kafka, and ADF to deliver robust ETL/ELT solutions, ensuring data quality and governance across diverse data platforms.

The role requires 6+ years in data engineering, strong SQL and PySpark/Scala skills, and experience with both relational and NoSQL data stores.

Qualifications

• 6+ years of experience in data engineering with Azure Data Lake, Databricks, and ADF.
• Hands-on expertise in SQL and PySpark or Scala for building complex ETL/ELT solutions.
• Proven track record with batch & real-time ingestion, Kafka, and large-scale big data processing.
• Experience with relational and NoSQL engines: SQL Server, Oracle, Azure SQL, and Cosmos DB.
• Experience with Informatica PowerCenter, Informatica IDMC, or equivalent cloud data integration tools.

Responsibilities

• Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
• Develop highly scalable ETL/ELT solutions using SQL, PySpark, or Scala.
• Implement and support robust Data Lake / Lakehouse architectures.
• Integrate and transform structured and unstructured data from legacy and modern platforms.
• Apply rigorous data quality checks and support data governance basics including metadata, data lineage, and access controls.
• Optimize Spark jobs systematically for performance tuning, execution reliability, and cost efficiency.
• Collaborate dynamically with cross-functional business teams and mentor junior engineers within the ecosystem.

Skills

Azure Data Lake

Databricks

Azure Data Factory

SQL

PySpark

Scala

Kafka

SQL Server

Oracle

Azure SQL

Cosmos DB

Informatica PowerCenter

Informatica IDMC

Education

Bachelor’s degree in a relevant field

Tools

Informatica PowerCenter

Informatica IDMC

Job description

TATA Consultancy Services is hiring for Azure Databricks Data Engineer

Desired experience range: 6+ years

Desired Competencies (Technical/Behavioral Competency)

Must-Have

• 6+ years of experience in data engineering with strong expertise in Azure Data Lake, Databricks, and ADF.
• Hands-on expertise in SQL and PySpark or Scala for building complex ETL/ELT solutions.
• Proven track record with batch & real-time ingestion, Kafka, and large-scale big data processing.
• Experience with relational and NoSQL engines: SQL Server, Oracle, Azure SQL, and Cosmos DB.
• Experience with Informatica PowerCenter, Informatica IDMC, or equivalent cloud data integration tools.

Good-to-Have

• Insurance domain knowledge (Policy, Claims, Billing, Underwriting, Actuarial, Regulatory data).
• Experience with Azure DevOps (AzDO) and GitHub-based CI/CD pipelines.
• Practical usage of Copilot for data engineering and development productivity.
• Exposure to Generative AI / GenAI use cases in data platforms and analytics.
• Familiarity with Maestro or similar enterprise schedulers, and data tools like Precisely or equivalent.

Responsibilities / Expectations from the Role

• Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
• Develop highly scalable ETL/ELT solutions using SQL, PySpark, or Scala.
• Implement and support robust Data Lake / Lakehouse architectures.
• Integrate and transform structured and unstructured data from legacy and modern platforms.
• Apply rigorous data quality checks and support data governance basics including metadata, data lineage, and access controls.
• Optimize Spark jobs systematically for performance tuning, execution reliability, and cost efficiency.
• Collaborate dynamically with cross-functional business teams and mentor junior engineers within the ecosystem.