Azure Databricks Engineer

Tata Consultancy Services; Chennai District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services in Chennai seeks a Data Engineer to design, develop, and maintain scalable data pipelines for ingestion, transformation, and processing. The role emphasizes PySpark in Databricks, Azure Synapse/ADF, and data governance across platforms.

You will collaborate with data scientists and analysts in an Agile environment, build APIs for data exchange, and contribute to DevOps practices including CI/CD for deployment and testing.

Responsibilities

• Design, develop, and maintain scalable data pipelines for ingestion, transformation, and processing.
• Ensure data quality, validation, monitoring, and governance across platforms.
• Build and optimize batch and real-time data workflows.
• Implement Medallion Architecture (Bronze, Silver, Gold) for analytics.
• Develop and integrate APIs for data exchange and system connectivity.
• Collaborate with cross-functional teams in Agile environments to deliver data solutions.
• Follow Git-based development practices, including branching, reviews, and version control.
• Implement CI/CD pipelines to automate deployment, testing, and release processes.

Job description

• Design, develop, and maintain scalable data pipelines for data ingestion, transformation, and processing.
• Ensure high standards of data quality, validation, monitoring, and governance across data platforms.
• Build, orchestrate, and optimize batch and real-time data pipeline workflows.
• Implement and support Medallion Architecture (Bronze, Silver, Gold layers) for efficient data management and analytics.
• Develop and integrate APIs for data exchange and system connectivity.
• Collaborate with cross-functional teams in an Agile environment to deliver high-quality data solutions.
• Follow Git-based development practices, including branching strategies, code reviews, and version control.
• Implement and maintain CI/CD pipelines to automate deployment, testing, and release processes.

.

Key Responsibilities*

• Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL), Azure Synapse, Azure Data Factory, and other Azure data services.
• Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
• Optimize and tune data pipelines for performance and scalability.
• Write and optimize complex SQL queries for data extraction, transformation, and analysis.
• Use PySpark for large-scale data processing and analytics.
• Implement data partitioning, bucketing, z-ordering, liquid clustering and indexing strategies for efficient data retrieval.
• Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
• Work with APIs, streaming data, and batch processing to ensure seamless data integration.
• Implement data governance practices to ensure data quality, consistency, and security.
• Monitor and troubleshoot data pipelines to ensure data accuracy and availability.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Work closely with DevOps teams to deploy and monitor data pipelines in production environments.
• Document data pipelines, workflows, and processes for knowledge sharing and future reference.
• Maintain up-to-date documentation on data architecture and data models.