Lead Big Data Engineer

S&P Global, Inc.; Rangareddy; On-site; INR 1,800,000 - 2,600,000; Full time

Benefits offered by this job

Health & Wellness

Flexible downtime

Continual learning

Student loan program

Family perks

Retail discounts

Job summary

S&P Global is looking for an experienced data engineering professional in India to design and implement scalable data processing pipelines for enterprise datasets. You will build and maintain ETL processes, deliver data-intensive applications, and enable real-time analytics across cloud platforms.

You will collaborate with cross-functional teams, provide production support, and help advance our data solutions with modern technologies and streaming capabilities.

Qualifications

• 8+ years of hands-on experience in technology application development and production support.
• 6+ years building data pipelines using Python, Scala, or Java.
• 3+ years ETL with cloud analytics platforms such as Databricks, Snowflake, or Azure Synapse.
• Experience building data-intensive applications using Java, Python, Scala, and SQL.
• Proficiency with cloud environments such as AWS, Azure, or Google Cloud Platform.

Responsibilities

• Design and develop scalable data processing pipelines using distributed computing frameworks.
• Build and maintain ETL processes that load data into strategic information products.
• Develop data-intensive applications and services on cloud-based analytics platforms.
• Implement real-time stream processing to support business-critical operations.
• Collaborate with cross-functional teams to translate business requirements into robust solutions.
• Provide production support and troubleshoot data systems for high availability.

Skills

ETL development

Streaming / real-time processing

Distributed computing

Cross-functional collaboration

Problem solving

Education

Bachelor's degree or equivalent

Tools

Databricks

Snowflake

Azure Synapse

Apache Spark

Dask

Hadoop MapReduce

SQL

Docker

Kubernetes

Apache NiFi

Job description

About the Team

Our enterprise data management team is at the forefront of designing and developing cutting‑edge data solutions that drive strategic business outcomes across the organization. We value quick learners who thrive in a dynamic technology environment and can seamlessly transition between collaborative teamwork and independent problem‑solving. The team fosters a culture of continuous learning and innovation, where members tackle emerging technologies while maintaining strong analytical rigor and effective cross‑functional communication.

Responsibilities and Impact

• Design and develop scalable data processing pipelines using distributed computing frameworks to handle large‑scale enterprise datasets
• Build and maintain ETL processes that extract, transform, and load data into strategic information products supporting organizational goals
• Develop data‑intensive applications and services using modern programming languages and cloud‑based analytics platforms
• Implement stream processing solutions using real‑time data processing technologies to support business‑critical operations
• Collaborate with cross‑functional teams to translate complex business requirements into robust technical solutions
• Provide production support and troubleshooting for data systems, ensuring high availability and performance

What We’re Looking For

Basic Required Qualifications

• 8+ years of hands‑on experience in technology application development and production support
• 6+ years of experience developing data pipelines that extract, transform, and load data using programming languages such as Python, Scala, or Java
• Minimum 3+ years of experience developing and supporting ETL processes using cloud‑based analytics platforms such as Databricks, Snowflake, or Azure Synapse
• Experience building data‑intensive applications using modern programming technologies (including but not limited to C#, Java, Python, Scala, and SQL)
• Proficiency with cloud computing environments such as AWS, Azure, or Google Cloud Platform

Additional Preferred Qualifications

• Experience with stream processing technologies such as Apache, Apache Pulsar, or Amazon Kinesis for real‑time data processing
• Hands‑on experience with distributed processing frameworks such as Apache Spark, Dask, or Hadoop MapReduce for large‑scale data analytics
• Knowledge of data integration tools such as Apache NiFi, Talend, or Informatica and database replication technologies
• Experience with CI/CD pipelines, containerization technologies such as Docker, Podman, or containerd, and orchestration platforms such as Kubernetes, Docker Swarm, or OpenShift

Benefits

Health & Wellness: Health care coverage designed for the mind and body.

Flexible Downtime: Generous time off helps keep you energized for your time on.

Continuous Learning: Access a wealth of resources to grow your career and learn valuable new skills.

Invest in Your Future: Secure your financial future through competitive pay, retirement planning, a continuing education program with a company‑matched student loan contribution, and financial wellness programs.

Family Friendly Perks: It’s not just about you. S&P Global has perks for your partners and little ones, too, with some best‑in‑class benefits for families.

Beyond the Basics: From retail discounts to referral incentive awards—small perks can make a big difference.

Equal Opportunity Employer

S&P Global is an equal opportunity employer and all qualified candidates will receive consideration for employment without regard to race/ethnicity, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, marital status, military veteran status, unemployment status, or any other status protected by law. Only electronic job submissions will be considered for employment.

If you need an accommodation during the application process due to a disability, please send an email to [e-mail] and your request will be forwarded to the appropriate person.

US Candidates Only: Know Your Rights: Workplace discrimination is illegal