Azure Data Engineer

Tata Consultancy Services; New Delhi, Chennai District, Bengaluru; Presencial; INR 1.800.000 - 3.200.000; Jornada completa

Descripción de la vacante

Tata Consultancy Services in India is seeking a data engineer to design and implement scalable data ingestion and transformation pipelines on Azure. You will leverage Azure Data Factory, Databricks and other Azure services to build data warehouses and governance solutions.

Responsibilities include implementing streaming and batch data processes, APIs development, and collaborating in agile DevOps teams with Jira, Git and Confluence.

Formación

• Must-have Azure Data Factory and Azure Databricks.
• Experience designing data pipelines on Azure.
• Python/PySpark knowledge is a plus.

Responsabilidades

• Implement scalable data ingestion and transformation pipelines across Azure components.
• Use Logic Apps, Functions, and Data Factory.
• Develop integrations for large, complex software on Azure.
• Apply CI/CD and DevOps practices in delivery.
• Collaborate in Agile delivery teams with Jira, Git, Confluence.
• Migrate data sources and reconcile target systems.
• Design data warehouses on Azure using HDInsight, ADLS, Databricks, SQL DWH and more.
• Develop APIs and data governance, cataloguing and lineage.

Conocimientos

Azure Data Factory

Azure Databricks

CI/CD

Agile

DevOps

Jira

Git

Confluence

MongoDB

SQL DWH

Event Hubs

Data Catalog

Azure HDInsight

Data Governance

Herramientas

Jira

Git Repositories

Confluence

Descripción del empleo

Role & responsibilities

• Implementing highly performant, scalable and re-usable data ingestion and transformation pipelines across Azure components and services


• Strong experience using Logic Apps, Functions & Data Factory


• Strong experience of development using Microsoft Azure as well as experience of developing integrations for large, complex pieces of software.


• Strong understanding of the technical side of CI / CD


• Strong understanding of Agile


• Designing and implementing a data warehouse on Azure using Azure HDInsight, Azure Data Factory, ADLS, Databricks, SQL Server, SQL DWH, Analytics Service, Event Hubs, KeyVault and other Azure services


• Designing, orchestrating and implementing highly performant, scalable and re-usable data ingestion and transformation pipelines across Azure components and services


• Designing and implementing Event based and streaming data ingestion and processing using Azure PaaS services


• Data ingestion, data engineering and/or data curation using native Azure services or tools available in Azure Marketplace


• Designing and implementing data governance, data cataloguing and data lineage solutions using tools such as Azure Data Catalog/Informatica Data Catalog


• Developing physical data models in MongoDB, SQL DWH , ...etc.


• Developing APIs


• Have worked in agile delivery teams with DevOps ways of working to continuously deliver iterative deployments with experience in using Jira, Git Repositories, Confluence, or similar


• Have had experience in data migration activities in the past including migration strategy and approach, source and target system discovery, analysis, mapping, development and reconciliation

Preferred candidate profile

Must-Have

• Azure Data Factory, Azure Databricks

Good-to-Have

• Python/Pyspark