Data Engineer, Data Platforms

S&P Global Inc.; Chennai, Tiruchirappalli, Coimbatore District, Bengaluru, Hyderabad; On-site; INR 1,100,000 - 1,800,000; Full time

Benefits offered by this job

Health & Wellness benefits

Flexible Downtime

Continuous Learning

Invest in Your Future

Family Friendly Perks

Beyond the Basics

Job summary

S&P Global Inc. is seeking an experienced Data Engineer in Chennai to lead the transition of existing pipelines to the enterprise data platform and optimize performance.

You will work with Databricks and AWS teams to enable secure, scalable data flows and establish repeatable patterns for ingestion, transformation, testing, deployment, and monitoring. The role emphasizes data onboarding, metadata alignment, and integration with the enterprise data mastering platform, while mentoring peers and

Qualifications

• 5–7+ years of experience in data engineering, data platforms, or related domains.
• Hands-on experience building and operating batch and/or streaming data pipelines.
• Experience with Databricks-based platforms and lakehouse architectures.
• Strong familiarity with AWS data engineering services (S3, Glue, Lambda, Lake Formation, Kinesis, IAM).
• Knowledge of Delta Lake, Apache Iceberg, and Databricks Unity Catalog.
• Understanding of data quality, schema management, lineage/metadata, access control, encryption, and operations.

Responsibilities

• Partner with Databricks and platform teams to transition pipelines to the enterprise data platform.
• Provide hands-on engineering support to stabilize and optimize pipelines post-transition.
• Implement repeatable ingestion, transformation, testing, deployment, and monitoring patterns.
• Support secure AWS deployment patterns and governed data access.
• Work with lakehouse technologies and metadata-driven pipeline design.
• Assist with observability practices (logging, telemetry, alerts, dashboards) for production pipelines.
• Support asset-agnostic onboarding and reusable framework development.
• Contribute to reusable assets, onboarding templates, and runbooks.
• Lead technical guidance through code/design reviews and CI/CD standards.
• Act as a bridge between offshore execution teams and global stakeholders.
• Leverage AI tools (Claude, GitHub Copilot) with context engineering for efficient engineering tasks.

Skills

Data engineering

Databricks

AWS

Lakehouse architecture

Delta Lake

Apache Iceberg

Unity Catalog

Data quality and security

CI/CD

Observability

Technical leadership

Tools

Databricks

AWS Glue

AWS Lambda

Amazon Kinesis

S3

Delta Lake

Apache Iceberg

Unity Catalog

Job description

About the Role:

Grade Level (for internal use): 10

Key Responsibilities

Data Pipeline Transition and Platform Delivery

• Partner with Databricks and platform teams to transition existing pipelines into the target enterprise data platform.
• Provide hands‑on engineering support to stabilize and optimize pipelines post‑transition, focusing on performance, scalability, maintainability, and operational reliability.
• Implement and follow repeatable engineering patterns for ingestion, transformation, testing, deployment, and monitoring.
• Support secure AWS deployment patterns and governed data access using AWS IAM, encryption, and platform guardrails.
• Work with lakehouse technologies such as Delta Lake, Apache Iceberg, and Databricks Unity Catalog, and support metadata‑driven pipeline design.
• Assist with establishing and maintaining observability practices (logging, telemetry, alerts, dashboards) for production pipelines.

Data Onboarding and Asset‑Agnostic Enablement

• Support asset‑agnostic onboarding by applying standard ingestion and transformation frameworks to new datasets.
• Help offshore teams adopt common onboarding standards, delivery practices, and technical templates.
• Contribute to reusable technical assets such as reference implementations, onboarding templates, documentation, and operational runbooks.
• Support both batch and streaming onboarding patterns using AWS services (e.g., AWS Glue, AWS Lambda, Amazon Kinesis) or event‑driven integrations as appropriate.

Data Mastering Platform Integration

• Support integration of platform datasets with the enterprise data mastering platform.
• Help establish and/or maintain data quality controls, reconciliation, and metadata alignment for trusted mastered data.
• Support golden record / golden copy generation and maintenance activities for key financial datasets (as applicable to the program).

Technical Leadership and Engineering Excellence

• Provide technical guidance to engineering peers through code reviews, design reviews, and implementation planning.
• Contribute to engineering standards and delivery rigor across CI/CD, infrastructure automation, testing practices, and operational readiness.
• Help ensure documentation quality and operational discipline (runbooks, support procedures, incident learnings).
• Act as a technical bridge between offshore execution teams and global platform stakeholders to resolve delivery blockers and keep alignment.

AI-Assisted Engineering and Context Engineering

• Use AI tools (Claude, GitHub CoPilot, LLMs) to accelerate engineering tasks such as drafting code, refactoring, writing documentation, generating unit tests, and troubleshooting.
• Apply basic prompt engineering and context engineering practices by providing the right technical background, constraints, schemas, examples, and acceptance criteria to improve output quality.
• Validate and verify AI‑generated outputs before adoption, especially for correctness, security, and maintainability.
• Follow responsible AI usage expectations, including protection of sensitive data, credentials, and regulated information.

Required Qualifications

Technical Experience

• 5–7+ years of experience in data engineering, data platforms, or related technical domains.
• Hands‑on experience building and operating batch and/or streaming data pipelines.
• Experience working with Databricks‑based platforms and lakehouse architectures.
• Strong familiarity with AWS data engineering services such as Amazon S3, AWS Glue, AWS Lambda, AWS Lake Formation, Amazon Kinesis, and AWS IAM.
• Working knowledge of Delta Lake, Apache Iceberg, and Databricks Unity Catalog (or equivalent governed lakehouse concepts).
• Solid understanding of data quality, schema management, lineage/metadata basics, access control, encryption, and operational support.

Leadership and Collaboration

• Ability to provide technical guidance without people management responsibility.
• Experience collaborating with global teams across time zones and working through ambiguous delivery problems.
• Strong communication skills and ability to translate technical requirements into implementable tasks.

AI Tooling (Required)

• Practical experience using AI‑assisted engineering tools such as Claude, GitHub CoPilot, or LLM‑based development features.
• Ability to use AI effectively with basic context engineering practices (inputs/constraints/examples/acceptance criteria).
• Demonstrated ability to validate AI‑generated outputs and follow responsible, secure usage practices.

Preferred Qualifications

• Experience with enterprise onboarding frameworks, reusable ingestion patterns, and pipeline templates.
• Familiarity with data cataloging/metadata management and governed data product concepts.
• Knowledge of semantic modeling (e.g., semantic layers, business definitions, canonical models) is a plus.
• Exposure to enterprise data mastering/MDM workflows, golden record concepts, and trusted data distribution.
• Experience in regulated environments with strong governance, auditability, and security expectations.
• Databricks and/or AWS certifications (or equivalent practical experience) are a plus.

About S&P Global

At S&P Dow Jones Indices, we provide iconic and innovative index solutions backed by unparalleled expertise across the asset‑class spectrum. By bringing transparency to the global capital markets, we empower investors everywhere to make decisions with conviction. We’re the largest global resource for index‑based concepts, data and research, and home to iconic financial market indicators, such as the S&P 500® and the Dow Jones Industrial Average®.

More assets are invested in products based upon our indices than any other index provider in the world. With over USD 7.4 trillion in passively managed assets linked to our indices and over USD 11.3 trillion benchmarked to our indices, our solutions are widely considered indispensable in tracking market performance, evaluating portfolios and developing investment strategies.

S&P Dow Jones Indices is a division of S&P Global (NYSE: SPGI). S&P Global is the world’s foremost provider of credit ratings, benchmarks, analytics and workflow solutions in the global capital, commodity and automotive markets. With every one of our offerings, we help many of the world’s leading organizations navigate the economic landscape so they can plan for tomorrow, today. For more information, visit www.spglobal.com/spdji.

What’s In It For You?

Advancing Essential Intelligence.

Our Mission:

Advancing Essential Intelligence.

Our People:

We're more than 35,000 strong worldwide—so we're able to understand nuances while having a broad perspective. Our team is driven by curiosity and a shared belief that Essential Intelligence can help build a more prosperous future for us all.From finding new ways to measure sustainability to analyzing energy transition across the supply chain to building workflow solutions that make it easy to tap into insight and apply it. We are changing the way people see things and empowering them to make an impact on the world we live in. We’re committed to a more equitable future and to helping our customers find new, sustainable ways of doing business. Join us and help create the critical insights that truly make a difference.

Our Values:

Integrity, Discovery, Partnership

Throughout our history, the world's leading organizations have relied on us for the Essential Intelligence they need to make confident decisions about the road ahead. We start with a foundation of integrity in all we do, bring a spirit of discovery to our work, and collaborate in close partnership with each other and our customers to achieve shared goals.

Benefits:

• Health & Wellness: Health care coverage designed for the mind and body.
• Flexible Downtime: Generous time off helps keep you energized for your time on.
• Continuous Learning: Access a wealth of resources to grow your career and learn valuable new skills.
• Invest in Your Future: Secure your financial future through competitive pay, retirement planning, a continuing education program with a company‑matched student loan contribution, and financial wellness programs.
• Family Friendly Perks: It’s not just about you. S&P Global has perks for your partners and little ones, too, with some best‑in class benefits for families.
• Beyond the Basics: From retail discounts to referral incentive awards—small perks can make a big difference.

For more information on benefits by country visit: https://spgbenefits.com/benefit-summaries

Global Hiring and Opportunity at S&P Global:

At S&P Global, we are committed to fostering a connected and engaged workplace where all individuals have access to opportunities based on their skills, experience, and contributions. Our hiring practices emphasize fairness, transparency, and merit, ensuring that we attract and retain top talent. By valuing different perspectives and promoting a culture of respect and collaboration, we drive innovation and power global markets.

Equal Opportunity Employer

S&P Global is an equal opportunity employer and all qualified candidates will receive consideration for employment without regard to race/ethnicity, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, marital status, military veteran status, unemployment status, or any other status protected by law. Only electronic job submissions will be considered for employment.

If you need an accommodation during the application process due to a disability, please send an email to [e-mail] and your request will be forwarded to the appropriate person.

US Candidates Only: The EEO is the Law Poster http://www.dol.gov/ofccp/regs/compliance/posters/pdf/eeopost.pdf describes discrimination protections under federal law.

Pay Transparency Nondiscrimination Provision - https://www.dol.gov/sites/dolgov/files/ofccp/pdf/pay-transp_%20English_formattedESQA508c.pdf