SAP, SAP BW

Infosys; Hyderabad, Chennai District, Bengaluru; On-site; INR 2,600,000 - 4,000,000; Full time

Job summary

Infosys in Hyderabad, India seeks an experienced Data Engineer to lead end-to-end delivery of Databricks and PySpark-based data pipelines. You will optimize batch and near-real-time processing and collaborate with SAP BW stakeholders to translate requirements into scalable data platform designs.

The role requires 7-9 years of hands-on Databricks experience, solid data modeling skills, and a track record of delivering reliable, governed data solutions in an agile environment.

Qualifications

• 7-9 years of hands-on Databricks and PySpark experience for large-scale data processing.
• Strong SAP BW concepts knowledge to support analytics/reporting requirements.
• Solid data modeling, ETL/ELT patterns, and performance tuning in distributed environments.
• Ability to lead technical delivery and communicate across teams.
• Exposure to building scalable, reliable data pipelines with monitoring and governance.

Responsibilities

• Lead end-to-end delivery of data engineering solutions leveraging Databricks and PySpark for scalable processing and transformation.
• Design, build, and optimize batch/near-real-time pipelines with strong focus on performance, cost efficiency, and reliability.
• Collaborate with SAP BW stakeholders to understand source models, data semantics, and reporting needs; translate them into modern data platform designs.
• Develop reusable frameworks, coding standards, and best practices for Spark-based development and notebook/job orchestration.
• Ensure data quality through validations, reconciliation checks, and robust error handling/monitoring across pipelines.
• Drive technical discussions, estimate work, plan sprints, and manage delivery risks/dependencies for the team.
• Mentor engineers, conduct code reviews, and enforce secure, maintainable, and well-documented implementations.
• Partner with platform/DevOps teams to support CI/CD, environment management, and production readiness for Databricks workloads.

Skills

Databricks

PySpark

Spark optimization

Data modeling

ETL/ELT

Education

Bachelor's or Master's degree in CS/Engineering

Job description

Preferred Qualifications:

• Experience modernizing or integrating SAP BW data with cloud-based lakehouse architectures using Databricks.
• Strong knowledge of Spark optimization techniques (partitioning, caching, joins, file sizing) and production-grade pipeline design.
• Exposure to Delta Lake patterns (ACID tables, MERGE, time travel) and building curated layers for analytics consumption.
• Experience with workflow orchestration and operational monitoring for data pipelines, including alerting and SLA management.
• Proven ability to guide teams through design reviews, code quality improvements, and delivery governance in agile environments.

Key Responsibilities:

• Lead end-to-end delivery of data engineering solutions leveraging Databricks and PySpark for scalable processing and transformation.
• Design, build, and optimize batch/near-real-time pipelines with strong focus on performance, cost efficiency, and reliability.
• Collaborate with SAP BW stakeholders to understand source models, data semantics, and reporting needs; translate them into modern data platform designs.
• Develop reusable frameworks, coding standards, and best practices for Spark-based development and notebook/job orchestration.
• Ensure data quality through validations, reconciliation checks, and robust error handling/monitoring across pipelines.
• Drive technical discussions, estimate work, plan sprints, and manage delivery risks/dependencies for the team.
• Mentor engineers, conduct code reviews, and enforce secure, maintainable, and well-documented implementations.
• Partner with platform/DevOps teams to support CI/CD, environment management, and production readiness for Databricks workloads.

Minimum Qualifications:

• Bachelors or Masters degree (BTECH, MTECH, MCA, MSC) in Computer Science, Engineering, or a related field.
• 7-9 years of overall experience with strong hands-on expertise in Databricks and PySpark for large-scale data processing.
• Strong experience working with SAP BW concepts and data structures to support analytics/reporting requirements.
• Solid understanding of data modeling, ETL/ELT patterns, and performance tuning in distributed data processing environments.
• Ability to lead technical delivery, collaborate with stakeholders, and communicate clearly across teams.