AI-Driven Data Engineer: Lakehouse Pipelines on AWS

IEBT Innovation

United States

Remote

USD 120,000 - 160,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health and dental insurance
Home office allowance
Meal allowance
Birthday day off
Health and wellness program
IDP program

Job summary

IEBT Innovation in the United States seeks a data engineer to develop and maintain data lakehouse pipelines using Spark, Delta Lake, and AWS. You will prepare data for clinical studies, validate datasets, and investigate inconsistencies while documenting all changes.

The role involves monitoring pipelines for stability and cost efficiency, and collaborating with engineering, QA, and client data management teams on reviews, testing, and documentation.

Qualifications

  • Proficient in Python and SQL in production data pipelines.
  • Experience with Spark (PySpark) in data lake/lakehouse environments.
  • Experience with AWS data services (S3, Glue, Lambda).
  • Knowledge of Git, code reviews, and automated testing.
  • Background in AI and software engineering.

Responsibilities

  • Develop and maintain data lakehouse pipelines using Spark, Delta Lake, and AWS.
  • Prepare data for clinical studies and validate completeness/accuracy.
  • Investigate data inconsistencies and implement fixes with change history.
  • Monitor pipelines to improve stability and reduce costs.
  • Collaborate with engineering, QA, and client data management teams on reviews and testing.

Skills

Python
SQL
Fluent English
AI & software engineering

Tools

Spark (PySpark)
Delta Lake
AWS (S3, Glue, Lambda)
Git
Code reviews
Automated testing
Trino/Presto
Amazon EMR
Great Expectations
Pydantic
Claude Code
Cursor
Copilot

Job description

IEBT Innovation in the United States seeks a data engineer to develop and maintain data lakehouse pipelines using Spark, Delta Lake, and AWS. You will prepare data for clinical studies, validate datasets, and investigate inconsistencies while documenting all changes.

The role involves monitoring pipelines for stability and cost efficiency, and collaborating with engineering, QA, and client data management teams on reviews, testing, and documentation.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AWS Data Engineer: Lakehouse & AI-Driven Pipelines
Senior AWS Data Engineer: Lakehouse & AI-Driven Pipelines

Global Business Ser. 4u Inc. • New York (NY)

On-site
USD 140,000 - 190,000
Remote Data Engineer: Lakehouse & AI Pipelines
Remote Data Engineer: Lakehouse & AI Pipelines

Bridgeway Benefit Technologies • Miami (FL)

Remote
USD 90,000 - 150,000
Senior Data Engineer — AI-Driven Lakehouse & Pipelines
Senior Data Engineer — AI-Driven Lakehouse & Pipelines

TDIndustries, Inc. • Dallas (TX)

On-site
USD 108,000 - 132,000
Senior Data Engineer: Lakehouse Pipelines & Data Quality
Senior Data Engineer: Lakehouse Pipelines & Data Quality

Taskus India • United States

Remote
USD 140,000 - 190,000
Senior Data Engineer: Lakehouse + Cloud Data Pipelines
Senior Data Engineer: Lakehouse + Cloud Data Pipelines

FPT Asia Pacific • United States

Remote
USD 140,000 - 190,000
Data Engineer: AI-Driven Lakehouse & Pipelines Expert
Data Engineer: AI-Driven Lakehouse & Pipelines Expert

TriSearch • Denver (CO)

Hybrid
USD 160,000 - 190,000
Health, dental, vision coverage
Hybrid/Remote work options
401(k) with company match
+5
Data & AI Engineer II: Lakehouse & Real-Time Pipelines
Data & AI Engineer II: Lakehouse & Real-Time Pipelines

27Global LLC • Leawood (KS)

Hybrid
USD 120,000 - 170,000
Profit sharing
Quarterly bonus potential
Learning, development & certifications
+1
Data Engineer - Databricks Lakehouse & IoT Pipelines
Data Engineer - Databricks Lakehouse & IoT Pipelines

SmartRent • Oregon

Hybrid
USD 120,000 - 190,000
Competitive benefits package
Paid time off (PTO)
401(k) with employer contributions
+2
Data Engineer Consultant: Lakehouse & Cloud Pipelines
Data Engineer Consultant: Lakehouse & Cloud Pipelines

Indicium AI • New York (NY)

On-site
USD 120,000 - 175,000
Remote Data Engineer — AI-Ready Lakehouse Pipelines
Remote Data Engineer — AI-Ready Lakehouse Pipelines

7Seventy Recruiting • United States

Remote
USD 160,000 - 190,000
Remote working environment
Flexible, unlimited PTO
13 paid holidays (NFL/Super Bowl)**
+1