Azure Data Engineer

Tata Consultancy Services; New Delhi; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Tata Consultancy Services in India is seeking a data engineer to design and implement scalable data ingestion and transformation pipelines on Azure. You will work with Azure Data Factory, Data Lake and Databricks to deliver performant solutions across large datasets.

You will collaborate in agile delivery teams with DevOps practices, develop APIs, and contribute to data governance, cataloging, and lineage using Azure-native tools.

Qualifications

• Experience with Azure Data Factory and Azure Databricks for building data pipelines.
• Knowledge of designing and implementing scalable ingestion/transformation pipelines on Azure.
• Familiarity with CI/CD practices and Agile methodologies.

Responsibilities

• Implement highly performant, scalable data ingestion and transformation pipelines across Azure components and services.
• Strong experience using Logic Apps, Functions & Data Factory.
• Developing integrations for large, complex software using Microsoft Azure and related services.
• Designing and implementing a data warehouse on Azure using Azure HDInsight, Data Factory, ADLS, Databricks, SQL Server and related Azure services.
• Orchestrating event-based and streaming data ingestion using Azure PaaS services.
• Data ingestion, data engineering or data curation using native Azure tools or Azure Marketplace offerings.
• Data governance, cataloguing and lineage solutions using tools like Azure Data Catalog or Informatica Data Catalog.
• Developing physical data models in MongoDB, SQL DWH, etc.
• Developing APIs and contributing to DevOps driven delivery with Jira, Git, Confluence or similar.
• Participation in data migration activities including migration strategy and mapping.

Tools

Azure Data Factory

Azure Databricks

Python

Pyspark

Job description

Must have skills: Azure Data Factory, Azure Databricks

Good to have skills: Python, Pyspark

Responsibilities or Expectations from the Role:

• Implementing highly performant, scalable and re-usable data ingestion and transformation pipelines across Azure components and services
• Strong experience using Logic Apps, Functions & Data Factory
• Strong experience of development using Microsoft Azure as well as experience of developing integrations for large, complex pieces of software.
• Strong understanding of the technical side of CI / CD
• Strong understanding of Agile
• Designing and implementing a data warehouse on Azure using Azure HDInsight, Azure Data Factory, ADLS, Databricks, SQL Server, SQL DWH, Analytics Service, Event Hubs, KeyVault and other Azure services
• Designing, orchestrating and implementing highly performant, scalable and re-usable data ingestion and transformation pipelines across Azure components and services
• Designing and implementing Event based and streaming data ingestion and processing using Azure PaaS services
• Data ingestion, data engineering and/or data curation using native Azure services or tools available in Azure Marketplace
• Designing and implementing data governance, data cataloguing and data lineage solutions using tools such as Azure Data Catalog/Informatica Data Catalog
• Developing physical data models in MongoDB, SQL DWH , ...etc.
• Developing APIs
• Have worked in agile delivery teams with DevOps ways of working to continuously deliver iterative deployments with experience in using Jira, Git Repositories, Confluence, or similar
• Have had experience in data migration activities in the past including migration strategy and approach, source and target system discovery, analysis, mapping, development and reconciliation