Data Engineer - Spark and Scala

Infosys; Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Infosys in Bengaluru is seeking an experienced Data Engineer to design, develop, and maintain ETL pipelines using Spark-Scala on Databricks for batch and incremental processing.

You will implement data transformations, joins, aggregations, and validations to ensure accurate outputs, build reusable Spark components, and collaborate with stakeholders to deliver reliable data solutions. Strong knowledge of Spark with Scala and Databricks is required, along with ETL best practices.

Qualifications

• Bachelor’s/Master’s degree (BE/BTech/MSc/MCA/MTech or equivalent).
• 3–5 years of experience in data engineering or ETL development roles.
• Strong hands-on experience with Spark using Scala and working on Databricks.
• Solid understanding of ETL concepts, data transformations, and pipeline troubleshooting.
• Ability to write clean, testable code and collaborate effectively with technical and non-technical stakeholders.

Responsibilities

• Design, develop, and maintain ETL pipelines using Spark-Scala on Databricks for batch and incremental processing.
• Implement data transformations, joins, aggregations, and validations to ensure accurate and consistent outputs.
• Build reusable Spark components and follow best practices for modular, maintainable code.
• Tune Spark jobs (partitioning, caching, shuffle optimization) to improve performance and cost efficiency.
• Monitor job execution, troubleshoot failures, and provide timely production support with root-cause analysis.
• Collaborate with cross-functional teams to gather requirements and translate them into technical solutions.

Skills

Spark-Scala

Databricks

Education

Bachelor/Master degree

Tools

SQL

Delta Lake

Apache Airflow

Azure Data Lake Storage

Git

Job description

Primary skills

Spark-Scala, Databricks

Technology->Big Data - Data Processing->Spark Technology->Data Engineering->Databricks Technology->Functional Programming->Scala

Minimum Qualifications

• Bachelor’s/Master’s degree (BE/BTech/MSc/MCA/MTech or equivalent).
• 3–5 years of experience in data engineering or ETL development roles.
• Strong hands-on experience with Spark using Scala and working on Databricks.
• Solid understanding of ETL concepts, data transformations, and pipeline troubleshooting.
• Ability to write clean, testable code and collaborate effectively with technical and non-technical stakeholders.

Preferred Qualifications

• Experience building end-to-end pipelines on Databricks including notebooks, jobs/workflows, and cluster configuration basics.
• Strong SQL skills and experience integrating Spark pipelines with structured data sources and curated datasets.
• Familiarity with data quality frameworks, reconciliation strategies, and automated validation checks.
• Exposure to CI/CD practices for data engineering (version control, automated testing, release management).
• Proven ability to optimize distributed workloads and deliver measurable improvements in runtime and stability.

Good to have skills

SQL, Delta Lake, Apache Airflow, Azure Data Lake Storage (ADLS), Git

Key Responsibilities

• Design, develop, and maintain ETL pipelines using Spark-Scala on Databricks for batch and incremental processing.
• Implement data transformations, joins, aggregations, and validations to ensure accurate and consistent outputs.
• Build reusable Spark components and follow best practices for modular, maintainable code.

Performance, Reliability & Operations

• Tune Spark jobs (partitioning, caching, shuffle optimization) to improve performance and cost efficiency.
• Monitor job execution, troubleshoot failures, and provide timely production support with root-cause analysis.
• Implement logging, error handling, and data quality checks to improve pipeline reliability.

Collaboration & Delivery

• Work with cross-functional teams to gather requirements and translate them into technical solutions.
• Participate in code reviews, documentation, and knowledge sharing to uplift team standards.
• Support release cycles by validating outputs, ensuring backward compatibility, and maintaining deployment readiness.