Azure Databricks, ADF, Pyspark, Python, Terraform

Tata Consultancy Services; Chennai District; On-site; INR 900,000 - 1,400,000; Full time

Job summary

Tata Consultancy Services in Chennai is seeking a data engineer to design and build scalable data pipelines using Azure Databricks and ADF. You will optimize ETL/ELT processes with PySpark and Python, ingest data into ADLS, and implement IaC with Terraform.

Responsibilities include CI/CD, data governance, and collaboration with stakeholders to deliver reliable, secure data solutions for business needs.

Qualifications

• Experience with ETL/ELT development and data warehousing concepts.
• Batch and streaming data processing experience.
• Knowledge of data modeling concepts (Star/Snowflake) and data quality checks.

Responsibilities

• Design and develop scalable data pipelines using Azure Data Factory and Azure Databricks.
• Build and optimize large-scale ETL/ELT processes with PySpark and Python.
• Ingest data into Azure Data Lake and develop batch/real-time processing solutions.
• Implement IaC using Terraform and manage Azure resources (Data Lake Storage, Key Vault, Databricks, ADF).
• Develop CI/CD pipelines for deployment automation and monitor data pipelines.
• Collaborate with stakeholders to define data requirements and ensure governance & security.

Skills

Azure Databricks

Azure Data Factory (ADF)

PySpark

Python

Terraform

SQL

Azure Data Lake Storage (ADLS)

Azure DevOps

Tools

Delta Lake

Unity Catalog

Spark Optimization

Data Governance

Real-time Streaming (Kafka/Event Hub)

Synapse Analytics

Machine Learning Integration

Job description

JOB DESCRIPTION

Key Responsibilities

• Design and develop scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks.
• Build and optimize large-scale ETL/ELT processes using PySpark and Python.
• Ingest data from various structured and unstructured sources into Azure Data Lake.
• Develop batch and real-time data processing solutions.
• Create reusable data transformation frameworks and data quality checks.
• Implement Infrastructure as Code (IaC) using Terraform.
• Manage Azure resources including Data Lake Storage, Key Vault, Databricks, ADF, and Monitoring services.
• Optimize Spark jobs for performance, scalability, and cost efficiency.m
• Develop CI/CD pipelines for deployment automation.
• Collaborate with business stakeholders, architects, and development teams to define data requirements and solutions.
• Support production deployments and troubleshoot data pipeline issues.
• Follow data governance, security, and compliance standards.

Required Technical Skills

• 1) Azure Services: Azure Databricks, Azure Data Factory (ADF), Azure Data Lake Storage Gen2, Azure Key Vault, Azure Monitor, Azure DevOps
• 2) Data Engineering: Strong expertise in ETL/ELT development, Data Warehousing concepts, Data Modeling (Star Schema, Snowflake Schema). Batch and Streaming Data Processing
• 3) Programming: Python, PySpark, SQL
• 4) Infrastructure as Code: Terraform, Azure Resource Management Concepts
• 5) Databases: Azure SQL Database, SQL Server, PostgreSQL, Cosmos DB (Preferred)
• 6) DevOps: Azure DevOps, Git, CI/CD Pipeline Implementation

Mandatory Skills

• Azure Databricks
• Azure Data Factory (ADF)
• PySpark
• Python
• Terraform
• SQL
• Azure Data Lake Storage (ADLS)
• Azure DevOps

Preferred Skills

• Delta Lake
• Unity Catalog
• Spark Optimization
• Data Governance
• Real-time Streaming (Kafka/Event Hub)
• Synapse Analytics
• Machine Learning Integration