Spark

Infosys; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Infosys in Bengaluru is seeking a data engineer to design, develop, and maintain scalable Spark-based data pipelines for batch and near-real-time workloads. You will collaborate with engineers, analysts, and stakeholders to translate requirements into reliable technical solutions.

The role emphasizes ownership, performance tuning, and robust fault handling, with opportunities to build reusable components and contribute to production readiness across projects.

Qualifications

• Bachelors degree (or equivalent) in Engineering/Technology/Computer Science or related field.
• 35 years of experience working on data engineering or big data processing initiatives.
• Hands-on experience building and maintaining Spark-based data processing solutions.
• Strong understanding of distributed computing concepts and data processing fundamentals.
• Ability to work independently on assigned modules and collaborate effectively within a team.

Responsibilities

• Design, develop, and maintain scalable data processing jobs using Spark for batch and/or near-real-time workloads.
• Analyze large datasets to identify trends, anomalies, and data quality issues; implement validation and reconciliation checks.
• Optimize Spark applications for performance by tuning partitions, caching strategies, memory usage, and execution plans.
• Collaborate with cross-functional teams to translate business requirements into technical solutions and well-defined deliverables.
• Implement robust error handling, logging, and monitoring to ensure reliability and easier troubleshooting.
• Participate in code reviews, follow engineering best practices, and contribute to reusable components and standards.
• Support deployments and production issues by performing root-cause analysis and implementing preventive fixes.

Skills

Spark

Distributed computing

Data analysis

Education

BTech/B.E.

MTech/MSc/MCA

Tools

Hadoop

Hive

Kafka

Airflow

Delta Lake

Job description

Job Description

Join a data-driven team where your work with distributed processing helps turn complex datasets into meaningful insights. In this role, youll collaborate closely with engineers, analysts, and stakeholders to build reliable, scalable data solutions using Spark, contributing to faster decision-making and better customer outcomes. Youll be encouraged to take ownership of deliverables, improve performance, and bring clarity to ambiguous problem statements through structured analysis and thoughtful implementation. If you enjoy solving large-scale data challenges, optimizing pipelines, and working in a collaborative environment that values learning and continuous improvement, this opportunity will help you grow your technical depth while making a visible impact across projects and teams.

Roles ResponsibilitiesKey Responsibilities

• Design, develop, and maintain scalable data processing jobs using Spark for batch and/or near-real-time workloads.
• Analyze large datasets to identify trends, anomalies, and data quality issues; implement validation and reconciliation checks.
• Optimize Spark applications for performance by tuning partitions, caching strategies, memory usage, and execution plans.
• Collaborate with cross-functional teams to translate business requirements into technical solutions and well-defined deliverables.
• Implement robust error handling, logging, and monitoring to ensure reliability and easier troubleshooting.
• Participate in code reviews, follow engineering best practices, and contribute to reusable components and standards.
• Support deployments and production issues by performing root-cause analysis and implementing preventive fixes.

Technical Requirement

Primary skills:Technology->Big Data - Data Processing->Spark

Additional ResponsibilityMinimum Qualifications

• Bachelors degree (or equivalent) in Engineering/Technology/Computer Science or related field (BTech/BE/MSc or equivalent).
• 35 years of experience working on data engineering or big data processing initiatives.
• Hands-on experience building and maintaining Spark-based data processing solutions.
• Strong understanding of distributed computing concepts and data processing fundamentals.
• Ability to work independently on assigned modules and collaborate effectively within a team.

Preferred Qualifications

• Masters degree (MTech/MCA or equivalent) in a relevant discipline.
• Proven experience delivering end-to-end Spark pipelines, including development, testing, and production support.
• Experience improving job performance and stability through Spark tuning and structured troubleshooting practices.
• Familiarity with building reusable frameworks/components to standardize Spark development across projects.
• Strong communication skills to explain technical trade-offs and align solutions with stakeholder expectations.

Good to have skills

Hadoop, Hive, Kafka, Airflow, Delta Lake

Educational Requirement

MCA,MSc,MTech,Bachelor of Engineering,BTech

Preferred Skills

Technology->Big Data - Data Processing->Spark

Service Line

Data Analytics Unit