Data Engineer, Data Platforms

S&P Global; Indiana; On-site; USD 110,000 - 165,000; Full time

Job summary

S&P Global is seeking an experienced Data Engineer to transition existing data pipelines into the enterprise data platform and to optimize their performance and reliability on Databricks and AWS services.

You will collaborate with platform teams, implement repeatable data engineering patterns, and help maintain high data quality and observability across production pipelines.

Qualifications

• 5–7+ years of experience in data engineering, data platforms, or related technical domains.
• Hands-on experience building and operating batch and/or streaming data pipelines.
• Experience working with Databricks-based platforms and lakehouse architectures.
• Strong familiarity with AWS data engineering services (S3, Glue, Lambda, Lake Formation, Kinesis, IAM).
• Working knowledge of Delta Lake, Apache Iceberg, and Databricks Unity Catalog.
• Solid understanding of data quality, schema management, lineage/metadata basics, access control, encryption, and operational support.

Responsibilities

• Partner with Databricks and platform teams to transition existing pipelines into the target enterprise data platform.
• Provide hands-on engineering support to stabilize and optimize pipelines post-transition, focusing on performance, scalability, maintainability, and operational reliability.
• Implement and follow repeatable engineering patterns for ingestion, transformation, testing, deployment, and monitoring.
• Support secure AWS deployment patterns and governed data access using AWS IAM, encryption, and platform guardrails.
• Work with lakehouse technologies such as Delta Lake, Apache Iceberg, and Databricks Unity Catalog, and support metadata-driven pipeline design.
• Assist with establishing and maintaining observability practices (logging, telemetry, alerts, dashboards) for production pipelines.
• Provide technical guidance to engineering peers through code reviews, design reviews, and implementation planning.
• Contribute to engineering standards and delivery rigor across CI/CD, infrastructure automation, testing practices, and operational readiness.
• Help ensure documentation quality and operational discipline (runbooks, support procedures, incident learnings).
• Act as a technical bridge between offshore execution teams and global platform stakeholders to resolve delivery blockers and keep alignment.
• Use AI tools to accelerate engineering tasks, apply prompt/context engineering practices, and validate AI outputs for correctness and security.

Skills

Databricks

AWS

Pipelines

CI/CD

Observability

Tools

Delta Lake

Apache Iceberg

Unity Catalog

Databricks

Kinesis

S3

Glue

Lambda

Job description

Grade Level (for internal use)

10

Data Pipeline Transition and Platform Delivery

• Partner with Databricks and platform teams to transition existing pipelines into the target enterprise data platform.
• Provide hands‑on engineering support to stabilize and optimize pipelines post‑transition, focusing on performance, scalability, maintainability, and operational reliability.
• Implement and follow repeatable engineering patterns for ingestion, transformation, testing, deployment, and monitoring.
• Support secure AWS deployment patterns and governed data access using AWS IAM, encryption, and platform guardrails.
• Work with lakehouse technologies such as Delta Lake, Apache Iceberg, and Databricks Unity Catalog, and support metadata‑driven pipeline design.
• Assist with establishing and maintaining observability practices (logging, telemetry, alerts, dashboards) for production pipelines.

Data Onboarding and Asset‑Agnostic Enablement

• Support asset‑agnostic onboarding by applying standard ingestion and transformation frameworks to new datasets.
• Help offshore teams adopt common onboarding standards, delivery practices, and technical templates.
• Contribute to reusable technical assets such as reference implementations, onboarding templates, documentation, and operational runbooks.
• Support both batch and streaming onboarding patterns using AWS services (e.g., AWS Glue, AWS Lambda, Amazon Kinesis) or event‑driven integrations as appropriate.

Data Mastering Platform Integration

• Support integration of platform datasets with the enterprise data mastering platform.
• Help establish and/or maintain data quality controls, reconciliation, and metadata alignment for trusted mastered data.
• Support golden record / golden copy generation and maintenance activities for key financial datasets (as applicable to the program).

Technical Leadership and Engineering Excellence

• Provide technical guidance to engineering peers through code reviews, design reviews, and implementation planning.
• Contribute to engineering standards and delivery rigor across CI/CD, infrastructure automation, testing practices, and operational readiness.
• Help ensure documentation quality and operational discipline (runbooks, support procedures, incident learnings).
• Act as a technical bridge between offshore execution teams and global platform stakeholders to resolve delivery blockers and keep alignment.

AI‑Assisted Engineering and Context Engineering

• Use AI tools (Claude, GitHub CoPilot, LLMs) to accelerate engineering tasks such as drafting code, refactoring, writing documentation, generating unit tests, and troubleshooting.
• Apply basic prompt engineering and context engineering practices by providing the right technical background, constraints, schemas, examples, and acceptance criteria to improve output quality.
• Validate and verify AI‑generated outputs before adoption, especially for correctness, security, and maintainability.
• Follow responsible AI usage expectations, including protection of sensitive data, credentials, and regulated information.

Technical Experience

• 5–7+ years of experience in data engineering, data platforms, or related technical domains.
• Hands‑on experience building and operating batch and/or streaming data pipelines.
• Experience working with Databricks-based platforms and lakehouse architectures.
• Strong familiarity with AWS data engineering services such as Amazon S3, AWS Glue, AWS Lambda, AWS Lake Formation, Amazon Kinesis, and AWS IAM.
• Working knowledge of Delta Lake, Apache Iceberg, and Databricks Unity Catalog (or equivalent governed lakehouse concepts).
• Solid understanding of data quality, schema management, lineage/metadata basics, access control, encryption, and operational support.

Leadership and Collaboration

• Ability to provide technical guidance without people…