Big Data Engineer - Hadoop & PySpark Lead

Infosys; Bengaluru; On-site; INR 1,500,000 - 2,600,000; Full time

Job summary

Infosys in Bengaluru is seeking a Big Data Engineer to design scalable Hadoop and PySpark solutions for batch and large-scale processing. You will architect end-to-end data pipelines, optimize performance on Hadoop clusters, and develop Hive data models to support analytics consumption.

The role requires 5-9 years of hands-on experience with Hadoop and PySpark, plus strong abilities in data quality, monitoring, and stakeholder collaboration.

Qualifications

• Experience designing reusable data pipelines and frameworks to boost team productivity.
• Strong expertise in optimizing Spark jobs (partitioning, caching, shuffles) and Hive performance.
• Experience implementing data quality checks, monitoring, and dashboards for production pipelines.
• Ability to drive stakeholder communication, manage trade-offs, and lead data engineering solutions.
• Demonstrated mentoring and leadership to deliver Big Data solutions at scale.

Responsibilities

• Lead the design and development of scalable Big Data solutions using Hadoop and PySpark for batch and large-scale processing.
• Architect and implement end-to-end data pipelines, ensuring reliability, performance, and efficient resources on Hadoop clusters.
• Develop and optimize Hive data models, queries, and partitioning strategies for analytics consumption.
• Drive technical planning, estimation, and delivery for data engineering initiatives with timelines and quality.
• Establish coding standards, review code, and enforce best practices for maintainability and production readiness.
• Troubleshoot production issues, perform root-cause analysis, and implement preventive measures to improve stability and throughput.
• Collaborate with product, analytics, and platform teams to translate requirements into scalable technical solutions.
• Mentor team members, guide technical decisions, and support skill development across Hadoop, PySpark, Big Data, and Hive.

Skills

Hadoop

PySpark

Spark SQL

YARN

HDFS

Oozie

Airflow

Education

BTECH/MTECH/MCA/MSC

Job description

Hadoop / PySpark Good to have skills: Spark SQL, YARN, HDFS, Oozie, Airflow

Preferred Qualifications

• Experience designing reusable frameworks and standardized pipeline patterns to improve team productivity and consistency.


• Strong expertise in optimizing Spark jobs (partitioning, caching, shuffles) and Hive performance (file formats, partitions, bucketing).


• Experience implementing data quality checks, monitoring, and operational dashboards for production pipelines.


• Ability to drive stakeholder communication, manage technical trade-offs, and lead solutioning for complex data use cases.


• Demonstrated mentoring and leadership experience, enabling teams to deliver high-quality Big Data solutions at scale.

Key Responsibilities

• Lead the design and development of scalable Big Data solutions using Hadoop and PySpark for batch and large-scale processing.


• Architect and implement end-to-end data pipelines, ensuring reliability, performance tuning, and efficient resource utilization on Hadoop clusters.


• Develop and optimize Hive data models, queries, and partitioning strategies to support analytics and downstream consumption.


• Drive technical planning, estimation, and delivery for data engineering initiatives, ensuring timelines and quality standards are met.


• Establish coding standards, review code, and enforce best practices for maintainability, testing, and production readiness.


• Troubleshoot production issues, perform root-cause analysis, and implement preventive measures to improve stability and throughput.


• Collaborate with product, analytics, and platform teams to translate requirements into scalable technical solutions.


• Mentor team members, guide technical decisions, and support skill development across Hadoop, PySpark, Big Data, and Hive.

Minimum Qualifications

• Education: BTECH, MTECH, MCA, MSC (or equivalent).


• 5-9 years of overall experience with strong hands-on expertise in Hadoop and PySpark for large-scale data processing.


• Proven experience building and maintaining Big Data pipelines and working with Hive for querying and data modeling.


• Strong understanding of distributed processing concepts, performance optimization, and data reliability practices.


• Experience leading technical execution through code reviews, design discussions, and delivery ownership.