NYC-Based AI Data Engineer: Pipelines & ML Ops

1101 S&P Global Inc.; New York (NY); Hybrid; USD 90,000 - 150,000; Full time

Benefits offered by this job

Health care

Flexible time off

Learning resources

Retirement planning

Family perks

Job summary

S&P Global Inc. in NYC is seeking an experienced AI Data Engineer to design and operate large-scale data pipelines for model metadata, training data lineage, and monitoring.

You will work with Databricks, Spark, and MLflow to build feature stores, model registries, and real-time data workflows, collaborating with data scientists across the enterprise. Hybrid onsite work at 55 Water Street two days per week.

Qualifications

• Bachelor's or Master's degree in Computer Science, Data Science, Machine Learning, or related field.

Responsibilities

• Design and implement data pipelines for AI model metadata, training data lineage, and model performance metrics tracking.
• Build data infrastructure on Databricks leveraging Spark for large-scale distributed dataset processing.
• Develop MCP servers and enable AI data distribution via MCP.
• Develop feature engineering pipelines and data preprocessing workflows for AI model training and inference.
• Implement model versioning, experiment tracking, and model registry integration using MLflow or similar tools.
• Create automated workflows for AI agent discovery, classification, and inventory management across the enterprise.
• Design and maintain knowledge graph structures for representing AI model relationships, dependencies, and data lineage.
• Build real-time data pipelines for AI model monitoring, drift detection, and performance tracking.
• Develop data quality frameworks specific to AI training datasets and validation data.
• Collaborate with data scientists to optimize data access patterns and feature store implementations.
• Implement security and compliance controls for sensitive AI training data and model artifacts.
• Create comprehensive documentation for AI data architectures, schemas, and integration patterns.

Skills

Python

PyTorch

TensorFlow

scikit-learn

Databricks

Apache Spark

SQL

Data governance

AI ethics

Collaboration

Education

Bachelor's or Master's in CS/DS/ML

Tools

Databricks

MLflow

LangChain

HuggingFace

Job description

S&P Global Inc. in NYC is seeking an experienced AI Data Engineer to design and operate large-scale data pipelines for model metadata, training data lineage, and monitoring.

You will work with Databricks, Spark, and MLflow to build feature stores, model registries, and real-time data workflows, collaborating with data scientists across the enterprise. Hybrid onsite work at 55 Water Street two days per week.