Senior Data Engineer - Self-Hosted Data Platform Lead

Intercontinental Exchange Holdings, Inc.; Atlanta (GA); On-site; USD 125,000 - 180,000; Full time

Job summary

Intercontinental Exchange, Inc. is seeking a Senior Data Engineer to lead the design and implementation of a next-generation data platform. You will build scalable, self-service data pipelines with strong governance, collaborating with ML/AI teams and stakeholders across the organization.

The role emphasizes data quality, deployment strategies, and performance optimization on a Kubernetes-based platform, with mentorship for junior engineers and a focus on engineering excellence.

Qualifications

• Bachelor’s degree in science/engineering or related field.
• 5+ years of experience in data engineering on enterprise-scale platforms.
• Experience with data quality frameworks, testing, and governance.
• Strong SQL and Python skills for data transformation and pipeline development.

Responsibilities

• Design, build, and maintain on-premises data orchestration platform using open source tools.
• Create self-service capabilities for teams to build and deploy pipelines with minimal engineering support.
• Implement data quality testing frameworks to ensure data integrity across the lifecycle.
• Establish data engineering best practices including version control, CI/CD for data pipelines, and automated testing.
• Collaborate with ML/AI teams to build scalable feature engineering pipelines for batch and real-time processing.
• Develop reusable patterns for data integration across the organization.

Skills

Data engineering

SQL

Python

Orchestrating workflows

CI/CD pipelines

Education

Bachelor’s degree in Computer Science, Computer Engineering, Data Science, Mathematics, Engineering, or related disciplines

Job description

Intercontinental Exchange, Inc. is seeking a Senior Data Engineer to lead the design and implementation of a next-generation data platform. You will build scalable, self-service data pipelines with strong governance, collaborating with ML/AI teams and stakeholders across the organization.

The role emphasizes data quality, deployment strategies, and performance optimization on a Kubernetes-based platform, with mentorship for junior engineers and a focus on engineering excellence.