Azure Databricks Engineer - 26th Aug (Wed)- Video Interview

Tata Consultancy Services; Kolkata District, Pune District; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Tata Consultancy Services in India seeks an experienced data engineering professional to design and optimize ELT pipelines on Azure. The role emphasizes Data Lake architecture, ADLS, ADF, Databricks, and Synapse, with a strong focus on scalable metadata-driven workflows.

Candidates should have hands-on experience with Spark, SQL, Delta Lake, and CI/CD, along with version control. This position is based in Kolkata and offers cloud-first projects and growth opportunities.

Qualifications

• Good knowledge of Data Brick lake house and Azure Data Lake concepts.
• Knowledge of Data Bricks delta concept.
• Delta live tables (DLT) built-in experience.
• Strong hands-on experience in ELT pipeline development using Azure Data Factory and Databricks Autoloader.
• Notebook scripting and Azure Synapse Activity Copy, Data Flow Task.
• Strong knowledge of metadata-driven data pipelines and dynamic logic.
• In-depth knowledge of ADLS and Azure Serverless SQL Pool.
• Experience with Spark and SQL data transformation.
• Knowledge of code management with Git or similar tools.
• Strong debugging skills in ETL workflows and data pipelines.

Skills

Data Lake Architecture

Azure Data Lake

Azure Data Factory

Azure Databricks

Azure Synapse

SQL

PySpark

Python

Delta Lake

ELT Pipelines

Tools

Databricks Autoloader

Git

CI/CD

Azure Synapse Analytics

Job description

Required Skill Set: Data Lake architecture, Azure Services ADLS, ADF, Azure Databricks, Synapse

ADF, ADB, SQL, Pyspark, python

• Good Knowledge of Data Brick lake house and Azure Data Lake concept
• Knowledge of Data Bricks delta concept

Delta live tables (DLT)

Strong hands-on experience in ELT pipeline development using Azure Data factory and Databricks Autoloader, Notebook scripting and Azure Synapse Activity Copy, Data Flow Task

Strong knowledge of metadata-driven data pipeline, metadata management, dynamic logic

In-depth knowledge of data storage solutions, including Azure Data Lake Storage (ADLS), and Azure Serverless SQL Pool.

Experience with data transformation using Spark, and SQL technologies. Solid understanding of design patterns, and best practices of the cloud stack.

Experience with code management and version control using Git or similar tools.

Strong problem-solving and debugging skills in ETL workflows and data pipelines.

Strong understanding of Azure Data bricks and Azure Synapse internals features and capabilities.

Knowledge of Azure DevOps and continuous integration and deployment (CI/CD) process.

Knowledge of data quality and data profiling techniques, with experience in data validation and data cleansing.