Databricks

Infosys; Bengaluru; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Infosys in Bengaluru is seeking a Data Engineer to design and maintain Databricks-based data pipelines and ETL/ELT workflows. You will optimize Spark jobs, build reusable notebooks, and ensure data quality for analytics and reporting.

Collaboration with delivery teams and adherence to coding standards are essential. The role requires 2–3 years of hands-on Databricks experience, strong PySpark skills, and solid data modeling knowledge.

Qualifications

• Bachelor’s or Master’s degree in BTECH, MTECH, MCA, or MSC (or equivalent).
• 2–3 years of hands-on experience working with Databricks in data engineering or analytics engineering projects.
• Strong experience in PySpark for building transformations and distributed data processing.
• Solid understanding of data pipeline concepts, data modeling basics, and structured/semi-structured data handling.
• Ability to debug and troubleshoot Spark jobs and collaborate effectively within delivery teams.
• Experience with Spark optimization techniques and practical performance tuning in Databricks environments.
• Familiarity with Delta Lake concepts such as ACID tables, schema evolution, and incremental processing patterns.
• Exposure to orchestrating workflows and managing dependencies for end-to-end pipeline execution.
• Experience working in agile delivery models with strong ownership of tasks, timelines, and quality outcomes.
• Strong communication skills to translate requirements into implementable data solutions and clearly document outcomes.

Responsibilities

• Develop and maintain data pipelines and transformations using Databricks and PySpark.
• Implement scalable ETL/ELT workflows to ingest, cleanse, and curate data for downstream analytics and reporting.
• Optimize Spark jobs for performance and cost by tuning partitions, caching, joins, and cluster configurations.
• Build reusable notebooks and modular code to support consistent development and easier maintenance.
• Perform data validation, reconciliation, and quality checks to ensure accuracy and reliability of datasets.
• Collaborate with cross-functional teams to gather requirements, clarify data definitions, and deliver aligned solutions.
• Support deployments and production operations by troubleshooting failures, analyzing logs, and resolving incidents.
• Contribute to documentation, coding standards, and best practices for Databricks-based development.

Skills

ETL

PySpark

Databricks

Delta Lake

Spark SQL

Data Modeling

Workflow Orchestration

Performance Tuning

Education

Bachelor’s or Master’s degree in BTECH, MTECH, MCA, or MSC (or equivalent).

Job description

ETL, PYSPARK, DATABRICKS, Delta Lake, Spark SQL, Data Modeling, Workflow Orchestration, Performance Tuning

Key Responsibilities:

• Develop and maintain data pipelines and transformations using Databricks and PySpark.
• Implement scalable ETL/ELT workflows to ingest, cleanse, and curate data for downstream analytics and reporting.
• Optimize Spark jobs for performance and cost by tuning partitions, caching, joins, and cluster configurations.
• Build reusable notebooks and modular code to support consistent development and easier maintenance.
• Perform data validation, reconciliation, and quality checks to ensure accuracy and reliability of datasets.
• Collaborate with cross-functional teams to gather requirements, clarify data definitions, and deliver aligned solutions.
• Support deployments and production operations by troubleshooting failures, analyzing logs, and resolving incidents.
• Contribute to documentation, coding standards, and best practices for Databricks-based development. Minimum Qualifications:

Minimum Qualifications:

• Bachelor’s or Master’s degree in BTECH, MTECH, MCA, or MSC (or equivalent).
• 2–3 years of hands‑on experience working with Databricks in data engineering or analytics engineering projects.
• Strong experience in PySpark for building transformations and distributed data processing.
• Solid understanding of data pipeline concepts, data modeling basics, and structured/semi‑structured data handling.
• Ability to debug and troubleshoot Spark jobs and collaborate effectively within delivery teams.
• Experience with Spark optimization techniques and practical performance tuning in Databricks environments.
• Familiarity with Delta Lake concepts such as ACID tables, schema evolution, and incremental processing patterns.
• Exposure to orchestrating workflows and managing dependencies for end‑to‑end pipeline execution.
• Experience working in agile delivery models with strong ownership of tasks, timelines, and quality outcomes.
• Strong communication skills to translate requirements into implementable data solutions and clearly document outcomes.