Azure Databricks Data Engineer

Infosys Limited; Bengaluru; On-site; INR 700,000 - 900,000; Full time

Job summary

Infosys Limited in Bengaluru seeks a Databricks Data Engineer with 2-3 years of experience to design and maintain scalable data pipelines. You will work with Databricks, Delta Lake, PySpark, and Spark SQL to ingest, transform, and optimize large datasets across cloud environments.

You will collaborate with analytics and BI teams to deliver reusable data assets, implement data quality and governance, and participate in architecture reviews and deployment standards.

Qualifications

• 2-3 years of experience in Data Engineering and Big Data technologies.
• Strong hands-on experience with Databricks.
• Proficiency in PySpark or Spark SQL for distributed data processing.
• Strong SQL skills with experience in query optimization and performance tuning.
• Experience in designing and developing ETL/ELT pipelines.
• Hands-on experience with Delta Lake, DataFrames, and Spark architecture.
• Knowledge of data warehousing concepts, dimensional modeling, and data lake architectures.
• Experience working with cloud platforms such as Azure, AWS, or GCP.
• Familiarity with Git-based source code management and DevOps practices.
• Strong analytical, debugging, and problem-solving skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Build and optimize ETL/ELT workflows for processing large-scale structured and unstructured datasets.
• Develop data ingestion frameworks to integrate data from multiple sources, including databases, APIs, applications, and cloud storage.
• Implement and manage Lakehouse architecture using Databricks and Delta Lake.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
• Collaborate with analytics, BI, and business teams to understand data requirements and deliver reusable data assets.
• Implement data quality frameworks, monitoring solutions, and governance controls across data pipelines.
• Troubleshoot production issues and provide timely resolution to ensure business continuity.
• Participate in architecture reviews, code reviews, and technical discussions to promote best practices.
• Create and maintain technical documentation, reusable components, and deployment standards.

Skills

Databricks

PySpark

Spark SQL

SQL

Delta Lake

ETL/ELT pipelines

Cloud platforms (Azure/AWS/GCP)

Git/DevOps practices

Education

Bachelor of Engineering

Tools

Databricks

Delta Lake

Git

CI/CD

Job description

Job ID/Reference Code INFSYS-EXTERNAL-254558

Work Experience 2 - 3 Years

Educational Requirements

Bachelor of Engineering

Service Line

Data & Analytics Unit

Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.Build and optimize ETL/ELT workflows for processing large-scale structured and unstructured datasets.Develop data ingestion frameworks to integrate data from multiple sources, including databases, APIs, applications, and cloud storage.Implement and manage Lakehouse architecture using Databricks and Delta Lake.Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.Collaborate with analytics, BI, and business teams to understand data requirements and deliver reusable data assets.Implement data quality frameworks, monitoring solutions, and governance controls across data pipelines.Troubleshoot production issues and provide timely resolution to ensure business continuity.Participate in architecture reviews, code reviews, and technical discussions to promote best practices.Create and maintain technical documentation, reusable components, and deployment standards.

Additional Responsibilities:

Infosys Data & Analytics practice helps organizations modernize their data landscape through cloud-native architectures, advanced analytics, AI, and intelligent data platforms. As a Databricks Data Engineer, you will work on enterprise-scale data modernization programs involving Lakehouse implementations, cloud migrations, real-time analytics, and AI-ready data ecosystems.

Technical and Professional Requirements:

2-3 years of experience in Data Engineering and Big Data technologies.Strong hands-on experience with Databricks.Proficiency in PySpark or Spark SQL for distributed data processing.Strong SQL skills with experience in query optimization and performance tuning.Experience in designing and developing ETL/ELT pipelines.Hands-on experience with Delta Lake, DataFrames, and Spark architecture.Knowledge of data warehousing concepts, dimensional modeling, and data lake architectures.Experience working with cloud platforms such as Azure, AWS, or GCP.Familiarity with Git-based source code management and DevOps practices.Strong analytical, debugging, and problem-solving skills.

Preferred Skills:

Technology->Data Engineering->Databricks