ADF, Databrick Professional

Infosys; Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Infosys in Bengaluru is seeking a data engineering professional to design and implement end-to-end data pipelines on Azure. You will develop ETL/ELT processes using ADF and build scalable data workflows in Azure Databricks using PySpark/Spark SQL.

The role requires integrating data from SQL Server, Azure SQL, ADLS, and various APIs, while ensuring data quality, governance and security across data lakes and warehouses.

Qualifications

• BSc, BCA, BTech, MCA, MTech or MSc in a relevant field.
• Strong knowledge of ETL, data warehousing concepts.
• Experience with Azure cloud services and data pipelines.
• Understanding of data governance and security practices.

Responsibilities

• Design and develop ETL/ELT pipelines using Azure Data Factory.
• Build data workflows with Azure Databricks (PySpark/Spark SQL).
• Integrate data from SQL Server, Azure SQL, ADLS, and APIs.
• Implement data transformation, cleansing and validation processes.
• Develop data lakes and Azure-based data warehouses.
• Monitor pipelines, troubleshoot failures and data quality issues.
• Translate business requirements into technical solutions with cross-functional teams.
• Create technical documentation and support deployment activities.

Education

BSc

BCA

BTech

MCA

MTech

MSc

Tools

Azure Data Factory (ADF)

Databricks

SQL Server

Azure SQL

ADLS

APIs

Job description

Service Line

Data Analytics Unit

Responsibilities

• Design and develop ETL/ELT pipelines using Azure Data Factory (ADF).
• Build and optimize data processing workflows using Azure Databricks (PySpark/Spark SQL).
• Integrate data from various sources such as SQL Server, Azure SQL, ADLS, APIs, and third-party applications.
• Implement data transformation, cleansing, and validation processes.
• Develop and maintain data lakes and data warehouse solutions on Azure.
• Monitor and troubleshoot pipeline failures, performance bottlenecks, and data quality issues.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Implement security, governance, and best practices for data management.
• Create technical documentation and support deployment activities.

Educational Requirements

• BSc
• BCA
• BTech
• MCA
• MTech
• MSc

Preferred Skills

• Technology->Cloud Integration->Azure Data Factory (ADF)
• Technology->Data Engineering->Databricks