Senior Data Engineer: AI-Driven Data Pipelines

IBM; New York (NY); On-site; USD 120,000 - 190,000; Full time

Job summary

IBM Consulting is seeking an experienced Data Engineer to support design and scaling of data pipelines and infrastructure for a high-priority Agentic AI engagement. You will work with AI architects to ensure the right data reaches the right systems in the right form, leveraging modern platforms for reliable pipelines and governance.

Responsibilities include building scalable data architectures, implementing data quality checks, and collaborating with AI teams to support analytics and AI/ML use

Qualifications

• 7+ years of experience designing, developing, and maintaining scalable data pipelines across Azure and AWS.
• Build and optimize enterprise data platforms using Azure Data Factory, Azure Data Lake, AWS S3, AWS Glue, Databricks, and Snowflake.
• Develop robust ETL/ELT frameworks for analytics, reporting, and AI/ML use cases in cloud/hybrid environments.
• Implement scalable ingestion and transformation pipelines for structured, semi-structured, and unstructured data sources.
• Support data industrialization through reusable pipelines, observability, CI/CD, and governance practices.

Responsibilities

• Data Pipeline Design & Development: design, build, and maintain robust pipelines for ingestion, transformation, and delivery of high-quality data across the platform.
• Data Infrastructure & Architecture: architect and maintain data lakehouse patterns and layered data models for AI and analytics.
• Data Quality & Governance: implement quality checks, lineage, cataloging, and access controls to ensure trustworthy data outputs.

Skills

Azure Data Factory

Databricks

Snowflake

ETL/ELT design

CI/CD

Data governance

Education

Master’s Degree

Tools

Azure Data Lake

AWS S3

Azure Synapse Analytics

Databricks

Job description

IBM Consulting is seeking an experienced Data Engineer to support design and scaling of data pipelines and infrastructure for a high-priority Agentic AI engagement. You will work with AI architects to ensure the right data reaches the right systems in the right form, leveraging modern platforms for reliable pipelines and governance.

Responsibilities include building scalable data architectures, implementing data quality checks, and collaborating with AI teams to support analytics and AI/ML use