Big Data Engineer

Infosys; Hyderabad, Pune District, Bengaluru; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Infosys is seeking a data engineer to design, develop, and maintain scalable batch and streaming pipelines using Spark and Scala. The role focuses on building reliable ingestion, leveraging Hive for data warehousing, and delivering well-modeled datasets for downstream use.

You will lead discussions, mentor engineers, and contribute to platform standards while implementing automated data quality checks and runbooks for reliability.

Qualifications

• Hands-on experience with Kafka for streaming ingestion and event-driven pipelines.
• Experience designing end-to-end data architectures for large-scale systems.
• Strong understanding of data partitioning, file formats, and big data processing patterns.
• Proven ability to lead technical discussions and mentor engineers.
• Experience implementing automated data quality checks and runbooks.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines with Spark and Scala.
• Optimize data workflows using Hive for querying and warehousing needs.
• Implement robust ingestion patterns for high-volume datasets with quality controls.
• Develop reusable Spark jobs and libraries to standardize practices.
• Tune Spark jobs for performance and runtime efficiency.
• Collaborate with stakeholders to deliver well-modeled datasets.
• Establish monitoring, alerting, and runbooks for pipeline reliability.
• Perform code reviews and contribute to platform standards.

Skills

Spark

Scala

Hive

Kafka

Data pipelines

Tools

Hadoop

HDFS

YARN

Airflow

HBase

Job description

Role & responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines using Spark and Scala.
• Build and optimize data processing workflows leveraging Hive for querying, transformations, and data warehousing needs.
• Implement reliable ingestion and integration patterns for high-volume datasets, ensuring data quality, consistency, and completeness.
• Develop reusable Spark jobs, libraries, and frameworks to standardize data engineering practices across teams.
• Tune Spark applications for performance (partitioning, caching, shuffles, memory management) and improve runtime efficiency.
• Work with stakeholders to understand data requirements and deliver well-modeled datasets for downstream consumption.
• Implement monitoring, alerting, and operational runbooks to ensure pipeline reliability and faster incident resolution.
• Perform code reviews, enforce engineering best practices, and contribute to continuous improvement of data platform standards.

Preferred Qualifications

• Hands‑on experience with Kafka for building streaming ingestion and event‑driven data pipelines.
• Experience designing end‑to‑end data architectures (ingestion, processing, storage, and serving layers) for large‑scale systems.
• Strong understanding of data partitioning strategies, file formats, and efficient processing patterns for big data workloads.
• Proven ability to lead technical discussions, mentor engineers, and drive best practices across delivery teams.
• Experience improving reliability through automated validations, data quality checks, and operational excellence practices.

Good to have skills

Hadoop, HDFS, YARN, Airflow, HBase

Location

PAN INDIA

EXP

5-15 Years