Spark

Infosys; Bengaluru; On-site; INR 900,000 - 1,400,000; Full time

Job summary

Infosys in Bengaluru seeks a skilled Spark Data Engineer to design, develop, and maintain scalable data processing jobs for batch and near-real-time workloads.

You will analyze large datasets to identify trends, anomalies, and data quality issues; implement validation and reconciliation checks, and optimize Spark apps for performance. Collaborate with cross-functional teams to deliver robust data solutions and ensure reliability.

Qualifications

• Bachelors degree in Engineering/Technology/CS or related field.
• 35 years of experience in data engineering or big data processing.
• Hands-on Spark experience building data processing solutions.
• Strong understanding of distributed computing concepts.
• Ability to work independently and within a team.

Responsibilities

• Design, develop, and maintain scalable data processing jobs using Spark for batch and/or near-real-time workloads.
• Analyze large datasets to identify trends, anomalies, and data quality issues; implement validation and reconciliation checks.
• Optimize Spark applications for performance by tuning partitions, caching strategies, memory usage, and execution plans.
• Collaborate with cross-functional teams to translate business requirements into technical solutions and well-defined deliverables.
• Implement robust error handling, logging, and monitoring to ensure reliability and easier troubleshooting.
• Participate in code reviews, follow engineering best practices, and contribute to reusable components and standards.
• Support deployments and production issues by performing root-cause analysis and implementing preventive fixes.

Skills

Spark

Big Data

Data processing

Education

Bachelors degree in Engineering/Technology/CS

Masters degree in relevant discipline

Tools

Hadoop

Hive

Kafka

Airflow

Delta Lake

Job description

• Primary skills:Technology->Big Data - Data Processing->Spark
  

• Primary skills:Technology->Big Data - Data Processing->Spark
  

Key Responsibilities:

• Design, develop, and maintain scalable data processing jobs using Spark for batch and/or near-real-time workloads.
• Analyze large datasets to identify trends, anomalies, and data quality issues; implement validation and reconciliation checks.
• Optimize Spark applications for performance by tuning partitions, caching strategies, memory usage, and execution plans.
• Collaborate with cross-functional teams to translate business requirements into technical solutions and well-defined deliverables.
• Implement robust error handling, logging, and monitoring to ensure reliability and easier troubleshooting.
• Participate in code reviews, follow engineering best practices, and contribute to reusable components and standards.
• Support deployments and production issues by performing root-cause analysis and implementing preventive fixes.

Minimum Qualifications:

• Bachelor’s degree (or equivalent) in Engineering/Technology/Computer Science or related field (BTech/BE/MSc or equivalent).
• 3–5 years of experience working on data engineering or big data processing initiatives.
• Hands‑on experience building and maintaining Spark‑based data processing solutions.
• Strong understanding of distributed computing concepts and data processing fundamentals.
• Ability to work independently on assigned modules and collaborate effectively within a team.

Preferred Qualifications:

• Master’s degree (MTech/MCA or equivalent) in a relevant discipline.
• Proven experience delivering end-to‑end Spark pipelines, including development, testing, and production support.
• Experience improving job performance and stability through Spark tuning and structured troubleshooting practices.
• Familiarity with building reusable frameworks/components to standardize Spark development across projects.
• Strong communication skills to explain technical trade‑offs and align solutions with stakeholder expectations.

Good to have skills:

Hadoop, Hive, Kafka, Airflow, Delta Lake