Azure Databricks Consultant

Infosys; Bengaluru; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Infosys Bengaluru seeks a data engineer with 6–8 years of experience to design and maintain scalable data pipelines on Azure Databricks. You will build end-to-end ETL/ELT processes using PySpark, Spark SQL, and Delta Lake to support analytics across enterprise platforms.

You will work with ADLS Gen2, Azure Synapse, and Azure Data Factory, ensuring data governance and security while optimizing performance and reliability through CI/CD and production support activities.

Qualifications

• 6–8 years of hands-on experience in Data Engineering and Big Data technologies.
• Strong expertise with Azure Databricks, PySpark, and Spark SQL.
• Experience building enterprise-scale data pipelines and ETL/ELT processes on Azure.

Responsibilities

• Design, develop, and maintain scalable Azure Databricks solutions for enterprise data platforms.
• Build end-to-end ETL/ELT pipelines using PySpark, Spark SQL, and Azure Data Factory.
• Implement Delta Lake and Lakehouse architecture for high-volume data processing and ensure data governance.

Skills

Data engineering

PySpark

Spark SQL

Azure Databricks

Azure Data Factory

Delta Lake

ADLS Gen2

Tools

Azure Synapse Analytics

Azure Key Vault

Job description

6 to 8 years of experience in Data Engineering and Big Data technologies. Strong hands‑on experience with Azure Databricks and PySpark. Expertise in Apache Spark, Spark SQL, and distributed data processing. Experience with Azure Data Factory (ADF) and building enterprise‑scale data pipelines. Strong knowledge of Azure Data Lake Storage (ADLS Gen2).

Design, develop, and maintain scalable Azure Databricks solutions for enterprise data platforms. Build and optimize end‑to‑end ETL/ELT pipelines using PySpark, Spark SQL, and Azure Data Factory (ADF). Develop robust data ingestion frameworks from multiple structured and unstructured data sources. Implement and manage Delta Lake and Lakehouse Architecture for high‑volume data processing. Create and optimize data models, transformations, and workflows to support analytical and business requirements. Perform performance tuning and troubleshooting of Databricks workloads and Apache Spark jobs. Work with Azure services including ADLS Gen2, Azure Synapse Analytics, Azure Key Vault, and Azure Data Factory. Collaborate with business stakeholders, architects, and cross‑functional teams to deliver data solutions. Ensure adherence to data governance, security, and best practices. Participate in code reviews, CI/CD implementation, testing, deployment, and production support activities.