Big Data Engineer (Hadoop & PySpark)

Infosys; Hyderabad, Pune District, Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Infosys in Hyderabad is seeking a Big Data Engineer (Hadoop & PySpark) with 2–5 years of experience to design, develop and maintain scalable data pipelines. The role focuses on PySpark workloads, ETL/ELT processes, and high-performance Spark jobs.

You will work with data architects, analysts and stakeholders to transform data into actionable insights, ensure data quality, security and governance, and optimize pipelines across batch and real-time processing.

Qualifications

• 2–5 years of experience in Big Data engineering.
• Experience with PySpark and Hadoop ecosystems is required.
• Strong SQL and data warehousing concepts.
• Experience designing scalable data pipelines.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark.
• Process and transform large datasets from multiple data sources.
• Build and optimize ETL/ELT workflows for data ingestion and processing.
• Develop Spark jobs for batch and real-time data processing.
• Write complex SQL queries for data extraction and reporting.
• Perform data cleansing, validation, and quality checks.
• Optimize Spark performance by tuning jobs, partitions, and cluster configurations.
• Integrate data from APIs, databases, cloud storage, and enterprise systems.
• Work with data architects, analysts, and business stakeholders to understand requirements.
• Troubleshoot production issues and implement performance improvements.
• Ensure data security, governance, and compliance standards are followed.
• Participate in code reviews and follow best development practices.
• Maintain technical documentation for data pipelines and processes.

Skills

PySpark

Spark

SQL

ETL/ELT

Data pipelines

Data processing

Performance tuning

Data security & governance

Tools

Hadoop

Job description

Position:Big Data Engineer (Hadoop & PySpark)

Experience: 2 to 5 years

Location: Hyderabad

Employment Type: Full-Time

Roles & Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark.
• Process and transform large datasets from multiple data sources.
• Build and optimize ETL/ELT workflows for data ingestion and processing.
• Develop Spark jobs for batch and real-time data processing.
• Write complex SQL queries for data extraction and reporting.
• Perform data cleansing, validation, and quality checks.
• Optimize Spark performance by tuning jobs, partitions, and cluster configurations.
• Integrate data from APIs, databases, cloud storage, and enterprise systems.
• Work with data architects, analysts, and business stakeholders to understand requirements.
• Troubleshoot production issues and implement performance improvements.
• Ensure data security, governance, and compliance standards are followed.
• Participate in code reviews and follow best development practices.
• Maintain technical documentation for data pipelines and processes.