Azure Data Engineer / Databricks

Tata Consultancy Services Limited; Hyderabad, Pune District, Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services Limited is seeking an Azure Data Engineer to design, develop, and deploy end-to-end data pipelines on Azure, integrating data from diverse sources for analytics and reporting.

You'll work with Azure Data Factory, Data Lake, Synapse, Databricks and related services, collaborating with teams to migrate data, implement security controls, and enable data access via Power BI and APIs.

Qualifications

• Experience designing and implementing Azure data pipelines (ETL) across multiple sources.
• Hands-on with Azure Data Factory, Data Lake, Synapse, Databricks.
• Proficient in Python and SQL; PySpark experience.
• Experience with data migrations to Azure from on-premises or other platforms.
• Knowledge of DevOps practices and CI/CD using Azure DevOps.
• Experience exposing data via Power BI and Azure API Apps.
• Understanding of on-premises and cloud data architecture.

Responsibilities

• Develop fully functional Azure data pipelines.
• Data migrations.
• Team player, responsive, organized.
• Strong analytical, problem-solving, client-facing, troubleshooting and multi-tasking skills.
• Data modeling, ETL processes and Data Warehousing.
• Data security, compliance standards and privacy policies.

Skills

Python

SQL

PySpark

ETL

Data modeling

Data wrangling

Tools

Azure Data Factory

Azure Data Lake

Azure Synapse Analytics

Azure Databricks

Azure IoT

Azure HDInsight + Spark

Azure Cosmos DB

Azure Stream Analytics

Azure SQL DB

Power BI

Azure DevOps

ARM templates

PowerShell

Job description

Must-Have

• Design, develop and implement data pipelines to perform (ETL) from multiple sources into Azure Services.
• Experience on Azure Data & Analytics PaaS Services: Azure Data Factory, Azure Data Lake, Azure Synapse Analytics, Azure Databricks, Azure IoT, Azure HDInsight + Spark, Azure Cosmos DB, Azure Stream Analytics and Azure SQL DB.
• Proficiency in programming languages (e.g., Python, SQL, PySpark etc.) and data manipulation tools.
• Manage Azure Data Storage Solutions including Azure Data Lake and different databases.
• Knowledge of DevOps tools.
• Manage and monitor the health of data integration processes, troubleshooting.
• Experience in Data Migrations (performing data migrations to Azure from on-premises or other platforms).
• Experience with Azure ARM templates, PowerShell and CI/CD using Azure DevOps.
• Experience with exposing data to end-users via Power BI and Azure API Apps.
• Understanding of on-premises and cloud-based data architecture and integration.
• Knowledge of DevOps practices and tools.

Good-to-Have

• Knowledge of integrating different systems with Python.
• Good documentation skills.

Responsibilities / Expectations from the Role

1. Develop fully functional Azure data pipelines.
2. Data migrations.
3. Team player, responsive, organized.
4. Strong analytical, problem-solving, client-facing, troubleshooting and multi-tasking skills.
5. Data modeling, ETL processes and Data Warehousing.
6. Data security, compliance standards and data privacy policies.