Hybrid Data Platform Engineer — AI-Ready Data Lake

Colonial Group

Greenwich (CT)

Hybrid

USD 140,000 - 210,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Competitive salary
Stock grants
401(k) retirement plan
Health benefits
Parental leave
Life insurance
Education reimbursement
Lunch allowance

Job summary

IBKR is seeking a Senior Data Platform Engineer to design, build, and operate a scalable data lake platform on AWS, enabling secure ingestion, governance, and analytics for internal teams.

You will partner with data scientists and developers to implement Terraform-based infrastructure, PySpark pipelines, and AI-ready knowledge bases, while championing observability and cost efficiency in a hybrid work model.

This position is based in Greenwich, CT with 3 days in office and 2 days remote.

Qualifications

  • 5+ years of hands-on data engineering or data platform experience.
  • Strong experience with AWS cloud services (S3, Glue, Athena, EMR).
  • Experience with infrastructure as code (Terraform).
  • Solid understanding of data lake architectures and cataloging.
  • Proficiency in Python and PySpark for ETL/ELT pipelines.
  • Experience with CI/CD practices and version control.

Responsibilities

  • Design, build, and operate the data lake platform and infrastructure on AWS.
  • Implement data governance and fine-grained access control using Lake Formation.
  • Automate infrastructure provisioning with Terraform and maintain self-service pipelines.
  • Contribute to GenAI knowledge base platform and Python-based services.
  • Build real-time streaming pipelines (Kafka) for analytics and AI workloads.
  • Maintain scalable ETL/ELT pipelines and data crawlers for diverse sources.
  • Monitor, troubleshoot, and optimize platform for performance and cost.

Skills

Python
PySpark
SQL
Git
Docker
Terraform
CI/CD
Kafka

Tools

AWS S3
AWS Glue
Athena
EMR
Terraform
Kubernetes
Airflow
Docker

Job description

IBKR is seeking a Senior Data Platform Engineer to design, build, and operate a scalable data lake platform on AWS, enabling secure ingestion, governance, and analytics for internal teams.

You will partner with data scientists and developers to implement Terraform-based infrastructure, PySpark pipelines, and AI-ready knowledge bases, while championing observability and cost efficiency in a hybrid work model.

This position is based in Greenwich, CT with 3 days in office and 2 days remote.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Platform Engineer - Cloud & GenAI (Hybrid)
Senior Data Platform Engineer - Cloud & GenAI (Hybrid)

Interactive Brokers • Greenwich (CT)

Hybrid
USD 140,000 - 210,000
Competitive salary
Bonus
Stock grants
+9
Remote Data Engineer: Lakehouse & AI Pipelines
Remote Data Engineer: Lakehouse & AI Pipelines

Bridgeway Benefit Technologies • Miami (FL)

Remote
USD 90,000 - 150,000
Hybrid Data Platform Engineer — Seattle AI-Secure Lakehouse
Hybrid Data Platform Engineer — Seattle AI-Secure Lakehouse

WideNet Consulting Group • Seattle (WA)

Hybrid
USD 124,000 - 138,000
Health benefits
401K
Employee Assistance Program
+1
Senior Lakehouse Data Engineer - Hybrid
Senior Lakehouse Data Engineer - Hybrid

MediData • Woodbridge Township (NJ)

Hybrid
USD 96,000 - 128,000
Senior Data Platform Engineer: Lakehouse & AI Infra
Senior Data Platform Engineer: Lakehouse & AI Infra

CreativeTechy • Lebanon

On-site
USD 150,000 - 190,000
Data Engineer
Data Engineer

Bridgeway Benefit Technologies • Miami (FL)

Remote
USD 90,000 - 150,000
AI-Driven Data Engineer: Lakehouse Pipelines on AWS
AI-Driven Data Engineer: Lakehouse Pipelines on AWS

IEBT Innovation • United States

Remote
USD 120,000 - 160,000
Health and dental insurance
Home office allowance
Meal allowance
+3
Senior Data Engineer – AI-Driven Lakehouse & Data Platform
Senior Data Engineer – AI-Driven Lakehouse & Data Platform

Myenergi Ltd • United States

Remote
USD 150,000 - 210,000
Free on-site EV charging
Private Medical Insurance
Cycle to Work Scheme
+2
Senior Data & AI Engineer: Lakehouse & Real-Time AI
Senior Data & AI Engineer: Lakehouse & Real-Time AI

27Global LLC • Leawood (KS)

Hybrid
USD 120,000 - 180,000
AI-Ready Lakehouse Architect for Scalable Data Platforms
AI-Ready Lakehouse Architect for Scalable Data Platforms

Tata Consultancy Services Limited • Marlborough (MA)

On-site
USD 150,000 - 160,000
Discretionary annual incentive
Comprehensive medical coverage
Parental leave
+4