Big Data Engineer_STG

Infosys; Hyderabad, Pune District, Bengaluru; Remote; INR 2,000,000 - 4,000,000; Full time

Job summary

Infosys is seeking an experienced Data Engineer to design, build, and optimize large-scale batch and streaming data pipelines using Spark and Scala. You will leverage Hive for querying, transformations, and data warehousing, and collaborate with cross-functional teams across India.

The role emphasizes data quality, scalable architectures, and operational runbooks. You will mentor engineers, review code, and drive best practices across delivery teams while delivering well-modeled datasets for

Qualifications

• 8–15 years of hands-on data engineering experience.
• Proficient in building batch and streaming pipelines with Spark/Scala.
• Experience with Hive for data warehousing and querying.
• Strong data quality, lineage, and governance practices.
• Ability to lead technical discussions and mentor engineers.

Responsibilities

• Design, develop, and maintain scalable batch and streaming pipelines with Spark and Scala.
• Build data workflows using Hive for transformations and warehousing.
• Ingest high-volume data with reliable patterns and data quality checks.
• Develop reusable Spark jobs and libraries to standardize practices.
• Tune Spark apps for performance and runtime efficiency.
• Collaborate with stakeholders to deliver well-modeled datasets.
• Implement monitoring, alerting, and runbooks for reliability.
• Perform code reviews and promote engineering best practices.

Job description

Role & responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines using Spark and Scala.
• Build and optimize data processing workflows leveraging Hive for querying, transformations, and data warehousing needs.
• Implement reliable ingestion and integration patterns for high-volume datasets, ensuring data quality, consistency, and completeness.
• Develop reusable Spark jobs, libraries, and frameworks to standardize data engineering practices across teams.
• Tune Spark applications for performance (partitioning, caching, shuffles, memory management) and improve runtime efficiency.
• Work with stakeholders to understand data requirements and deliver well-modeled datasets for downstream consumption.
• Implement monitoring, alerting, and operational runbooks to ensure pipeline reliability and faster incident resolution.
• Perform code reviews, enforce engineering best practices, and contribute to continuous improvement of data platform standards.

Preferred Qualifications

• Hands‑on experience with Kafka for building streaming ingestion and event‑driven data pipelines.
• Experience designing end‑to‑end data architectures (ingestion, processing, storage, and serving layers) for large‑scale systems.
• Strong understanding of data partitioning strategies, file formats, and efficient processing patterns for big data workloads.
• Proven ability to lead technical discussions, mentor engineers, and drive best practices across delivery teams.
• Experience improving reliability through automated validations, data quality checks, and operational excellence practices.

Good to have skills

• Hadoop
• HDFS
• YARN
• Airflow
• HBase

Location: PAN INDIA

EXP:8-15 Years