Senior Data Engineer - Data Platform & Pipelines Lead

Intercontinental Exchange Holdings; Atlanta (GA); On-site; USD 140,000 - 170,000; Full time

Job summary

Intercontinental Exchange Holdings is seeking a Senior Data Engineer to lead design and implementation of our next-generation data platform in Atlanta, GA. You will build on-premises data pipelines with governance and data quality, collaborating with stakeholders to enable ML and AI workflows.

You will mentor engineers, implement CI/CD for data pipelines, and partner with ML teams to create scalable feature pipelines for batch and real-time processing. 5+ years in data engineering required.

Qualifications

• Bachelor's degree in CS, CE, data science, mathematics, or related fields.
• 5+ years in data engineering with at least 2 years on enterprise-scale platforms.
• Strong SQL and Python for data transformation and pipeline development.

Responsibilities

• Design, build, and maintain on-premises data orchestration platform using best-in-breed open source tools.
• Create self-service capabilities for teams to build data pipelines with minimal engineering support.
• Implement robust data quality testing frameworks to ensure data integrity across the lifecycle.
• Establish data engineering best practices, including version control, CI/CD for data pipelines, and automated testing.
• Collaborate with ML/AI teams to build scalable feature engineering pipelines for batch and real-time processing.
• Mentor junior engineers and advocate for engineering excellence in data practices.

Skills

SQL

Python

Data pipelines

Data governance

CI/CD

Kubernetes

Education

Bachelor's degree in CS/CE/Data Science

Tools

Airflow

Kubernetes

Docker

Job description

Intercontinental Exchange Holdings is seeking a Senior Data Engineer to lead design and implementation of our next-generation data platform in Atlanta, GA. You will build on-premises data pipelines with governance and data quality, collaborating with stakeholders to enable ML and AI workflows.

You will mentor engineers, implement CI/CD for data pipelines, and partner with ML teams to create scalable feature pipelines for batch and real-time processing. 5+ years in data engineering required.