Spark

Infosys; Bengaluru; Sur place; INR 1 200 000 - 1 800 000; Plein temps

Résumé du poste

Infosys in Bengaluru, India seeks a Spark-based Data Engineer to design, develop, and maintain scalable data processing jobs for batch and near-real-time workloads.

You will analyze large datasets for trends, anomalies and data quality issues, and optimize Spark applications by tuning partitions, caching, memory usage and execution plans. You will collaborate across teams to deliver robust, well-documented components and support production deployments.

Qualifications

• Bachelor’s degree (or equivalent) in Engineering/Technology/CS or related field.
• 3–5 years of experience working on data engineering or big data processing initiatives.
• Hands-on experience building and maintaining Spark-based data processing solutions.
• Strong understanding of distributed computing concepts and data processing fundamentals.

Responsabilités

• Design, develop, and maintain scalable data processing jobs using Spark for batch and/or near-real-time workloads.
• Analyze large datasets to identify trends, anomalies, and data quality issues; implement validation and reconciliation checks.
• Optimize Spark applications for performance by tuning partitions, caching strategies, memory usage, and execution plans.
• Collaborate with cross-functional teams to translate business requirements into technical solutions and well-defined deliverables.
• Implement robust error handling, logging, and monitoring to ensure reliability and easier troubleshooting.
• Participate in code reviews, follow engineering best practices, and contribute to reusable components and standards.
• Support deployments and production issues by performing root-cause analysis and implementing preventive fixes.

Connaissances

Spark

Big Data

Data Processing

Distributed computing

Formation

Bachelor’s degree in Engineering/Technology/CS

Outils

Hadoop

Hive

Kafka

Airflow

Delta Lake

Description du poste

• Primary skills:Technology->Big Data - Data Processing->Spark

• Primary skills:Technology->Big Data - Data Processing->Spark

Key Responsibilities

• Design, develop, and maintain scalable data processing jobs using Spark for batch and/or near-real-time workloads.
• Analyze large datasets to identify trends, anomalies, and data quality issues; implement validation and reconciliation checks.
• Optimize Spark applications for performance by tuning partitions, caching strategies, memory usage, and execution plans.
• Collaborate with cross-functional teams to translate business requirements into technical solutions and well-defined deliverables.
• Implement robust error handling, logging, and monitoring to ensure reliability and easier troubleshooting.
• Participate in code reviews, follow engineering best practices, and contribute to reusable components and standards.
• Support deployments and production issues by performing root-cause analysis and implementing preventive fixes.

Minimum Qualifications

• Bachelor’s degree (or equivalent) in Engineering/Technology/Computer Science or related field (BTech/BE/MSc or equivalent).
• 3–5 years of experience working on data engineering or big data processing initiatives.
• Hands-on experience building and maintaining Spark-based data processing solutions.
• Strong understanding of distributed computing concepts and data processing fundamentals.
• Ability to work independently on assigned modules and collaborate effectively within a team.

Preferred Qualifications

• Master’s degree (MTech/MCA or equivalent) in a relevant discipline.
• Proven experience delivering end-to-end Spark pipelines, including development, testing, and production support.
• Experience improving job performance and stability through Spark tuning and structured troubleshooting practices.
• Familiarity with building reusable frameworks/components to standardize Spark development across projects.
• Strong communication skills to explain technical trade-offs and align solutions with stakeholder expectations.
• Good to have skills: Hadoop, Hive, Kafka, Airflow, Delta Lake