Azure Data Engineer

Tata Consultancy Services; Dadri; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Tata Consultancy Services is seeking an experienced Azure Data Engineer to design and implement data ingestion, transformation, storage, and management solutions within the Azure ecosystem. The role focuses on building scalable ETL pipelines using Azure Data Factory, Databricks, and Synapse, while collaborating with data scientists and analysts to enable analytics workloads.

You will also ensure operational excellence with DevOps practices, monitoring, and automation across data pipelines in a

Qualifications

• Experience in data engineering on Azure for 6–10 years.
• Ingestion, transformation, storage and management of data in Azure.
• ETL pipelines and cloud data platforms experience.
• Collaboration with data scientists and analysts for analytics workloads.

Responsibilities

• Ingest data from databases, files, APIs and streaming platforms into Azure (ETL).
• Perform data transformation and processing using Azure Databricks or Synapse to support analytics.
• Select and manage storage solutions: Azure SQL Database, Cosmos DB, Data Lake and Blob Storage.
• Implement data warehousing with Azure Synapse Analytics for large datasets.
• Model data and work with analysts to enable interactive analytics.
• Automate data pipelines, monitor data processes and manage deployments using DevOps and monitoring tools.

Skills

Data ingestion

Data transformation

Data modeling

Analytical thinking

Tools

Azure Data Factory

Azure Databricks

Azure Synapse Analytics

Azure SQL Database

Azure Cosmos DB

Azure Data Lake Storage

Azure Blob Storage

Azure Analysis Services

Azure DevOps

Azure Monitor

Azure Automation

Apache Spark

Job description

Greetings from Tata Consultancy Services (TCS)!

We are looking for As an Azure Data Engineer, you will be responsible for various tasks related to data ingestion, transformation, storage, and management within the Azure ecosystem.

Experience Required - 6 to 10 Years

Responsibilities

Data Ingestion and Integration: Ingest data from various sources such as databases, files, APIs, and streaming platforms into Azure. For example, you could use Azure Data Factory to orchestrate the extraction, transformation, and loading (ETL) of data from an on-premises SQL Server database to Azure Blob Storage.

• Data Transformation and Processing: Perform data transformation and processing tasks to prepare the data for analysis and reporting. Azure Databricks or Azure Synapse Analytics can be used for large-scale data transformations using tools like Apache Spark.
• Data Storage and Management: Select and manage the appropriate storage solutions for your data. For instance, you could use Azure SQL Database or Azure Cosmos DB for structured data, Azure Data Lake Storage for big data and unstructured data, or Azure Blob Storage for file storage.
• Data Warehousing: Implementing data warehousing solutions is a common use case for Azure Data Engineers. Azure Synapse Analytics (formerly Azure SQL Data Warehouse) offers scalable and distributed analytics capabilities for handling large datasets.
• Data Modeling and Analysis: Collaborate with data scientists and analysts to design and implement data models for analytical workloads. Azure Analysis Services or Azure Databricks can be used for building and deploying models to support interactive data analysis.
• DevOps and Automation: Automating data pipelines, monitoring data processes, and managing deployments are important tasks for Azure Data Engineers. They may use tools like Azure DevOps, Azure Monitor, and Azure Automation to achieve continuous integration and delivery.