Databricks Engineer

Infosys; Bengaluru; On-site; INR 2,800,000 - 4,400,000; Full time

Job summary

Infosys in Bengaluru is seeking an experienced Data Engineer to design, build, and optimize data pipelines using Databricks and PySpark. You will develop ETL/ELT workflows, implement Delta Lake storage, and manage batch and streaming data ingestion.

The role requires strong Python/SQL skills and experience across Azure/AWS/GCP, with orchestration tools like Airflow or ADF. You will collaborate with data analysts and scientists, ensure data quality and governance, and support production systems

Qualifications

• 5-8 years of experience in Data Engineering and Big Data technologies.
• Strong hands-on experience with Databricks, PySpark, and Spark SQL.
• Experience with Delta Lake, Databricks Workflows, and Unity Catalog.
• Proficiency in Python and SQL.
• Experience with Azure/AWS/GCP cloud platforms.
• Knowledge of Azure Data Factory (ADF), Apache Airflow, or similar orchestration tools.
• Understanding of data warehousing concepts and ETL/ELT methodologies.
• Experience with Kafka/Event Hub for streaming data processing is preferred.
• Familiarity with Git, CI/CD pipelines, and DevOps practices.
• Strong problem-solving, analytical, and communication skills.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks.
• Build ETL/ELT workflows for large-scale data processing.
• Develop and optimize data transformation logic using PySpark and Spark SQL.
• Implement and manage Delta Lake for reliable and efficient data storage.
• Work with batch and real-time data ingestion frameworks.
• Monitor and optimize data pipelines for performance and scalability.
• Collaborate with data analysts, data scientists, and business stakeholders.
• Ensure data quality, security, and governance standards are followed.
• Troubleshoot production issues and provide ongoing support.

Skills

Data engineering

Big data technologies

Python

SQL

ETL/ELT

Data warehousing

Streaming data processing

PySpark

Spark SQL

Databricks

Delta Lake

Databricks Workflows

Unity Catalog

Apache Airflow

Azure Data Factory

Git

CI/CD pipelines

DevOps practices

Cloud platforms (Azure/AWS/GCP)

Data governance

Tools

Databricks

PySpark

Spark SQL

Delta Lake

Databricks Workflows

Unity Catalog

Python

SQL

Azure

AWS

GCP

Azure Data Factory

Apache Airflow

Kafka

Event Hub

Git

CI/CD

DevOps

Job description

• 5-8 years of experience in Data Engineering and Big Data technologies.
• Strong hands-on experience with Databricks, PySpark, and Spark SQL.
• Experience with Delta Lake, Databricks Workflows, and Unity Catalog.
• Proficiency in Python and SQL.
• Experience with Azure/AWS/GCP cloud platforms.
• Knowledge of Azure Data Factory (ADF), Apache Airflow, or similar orchestration tools.
• Understanding of data warehousing concepts and ETL/ELT methodologies.
• Experience with Kafka/Event Hub for streaming data processing is preferred.
• Familiarity with Git, CI/CD pipelines, and DevOps practices.
• Strong problem-solving, analytical, and communication skills.

• 5-8 years of experience in Data Engineering and Big Data technologies.
• Strong hands-on experience with Databricks, PySpark, and Spark SQL.
• Experience with Delta Lake, Databricks Workflows, and Unity Catalog.
• Proficiency in Python and SQL.
• Experience with Azure/AWS/GCP cloud platforms.
• Knowledge of Azure Data Factory (ADF), Apache Airflow, or similar orchestration tools.
• Understanding of data warehousing concepts and ETL/ELT methodologies.
• Experience with Kafka/Event Hub for streaming data processing is preferred.
• Familiarity with Git, CI/CD pipelines, and DevOps practices.
• Strong problem-solving, analytical, and communication skills.

Roles & Responsibilities

Design, develop, and maintain data pipelines using Databricks. Build ETL/ELT workflows for large-scale data processing. Develop and optimize data transformation logic using PySpark and Spark SQL. Implement and manage Delta Lake for reliable and efficient data storage. Work with batch and real-time data ingestion frameworks. Monitor and optimize data pipelines for performance and scalability. Collaborate with data analysts, data scientists, and business stakeholders. Ensure data quality, security, and governance standards are followed. Troubleshoot production issues and provide ongoing support.