Python, Spark Scala Developer

Infosys; Hyderabad; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

Infosys is looking for a Data Engineer in Hyderabad, India. The role involves designing and maintaining scalable data pipelines using Spark with Python and Scala. Candidates should have 5+ years of experience in data engineering and strong proficiency in Python.

The ideal candidate will collaborate with stakeholders to deliver well-documented solutions, conduct code reviews, and ensure performance through tuning Spark jobs. A Bachelor's degree in Engineering or related field is required, along with experience in big data technologies.

Qualifications

• 5+ years of experience in data engineering or backend development.
• Strong proficiency in Python and Spark for large-scale data processing.
• Solid experience with Scala for production-grade implementations.

Responsibilities

• Design and maintain scalable batch/stream data pipelines.
• Tune Spark jobs for performance and cost efficiency.
• Work with stakeholders to refine requirements.

Skills

Data Engineering

Python

Apache Spark

Scala

SQL

Education

Bachelor of Engineering

Bachelors degree in Computer Science or related field

Job description

Educational Requirements

Educational Requirements Bachelor of Engineering

Service Line

Service Line Data Analytics Unit

Responsibilities

• Data Engineering Development: Design, develop, and maintain scalable batch/stream data pipelines using Spark with Scala and Python.
• Implement efficient transformations, joins, aggregations, and data quality checks for large datasets.
• Build reusable frameworks/utilities to standardize pipeline patterns and reduce delivery time.
• Performance/Reliability: Tune Spark jobs (partitioning, caching, shuffles, memory/executor settings) to improve performance and cost efficiency.
• Troubleshoot production issues, perform root-cause analysis, and implement preventive fixes.
• Ensure reliability through robust logging, monitoring hooks, and failure-handling strategies.
• Collaboration / Delivery: Work with stakeholders to refine requirements and deliver well-documented, production-ready solutions.
• Conduct code reviews, enforce best practices, and mentor team members on Spark/Scala/Python patterns.
• Contribute to CI/CD-friendly development practices including testing, version control, and release readiness.

Additional Responsibilities

• Bachelors degree in Computer Science, Engineering, or a related field (or equivalent practical experience).
• 59 years of experience in data engineering or backend development with strong hands-on delivery ownership.
• Strong proficiency in Python and Apache Spark for large-scale data processing.
• Solid experience with Scala for Spark-based development and production-grade implementations.
• Working knowledge of Hive and SQL-based data querying/processing concepts.

Technical and Professional Requirements

• Technology->Analytics - Packages->Python - Big Data
• Technology->Big Data - Data Processing->Spark

Preferred Skills

• Technology->Analytics - Packages->Python - Big Data
• Technology->Big Data - Data Processing->Spark