Lead Data Engineer - AI & GenAI Platform

Enterprise Products Partners L.P; Houston (TX); On-site; USD 150,000 - 190,000; Full time

Job summary

Enterprise Products Partners L.P. is seeking an experienced Analyst, Lead Data Engineering to architect and implement a cloud-based Data & AI platform supporting AI and analytics across midstream energy units.

You will shape architecture, set standards, and productionize data products and AI models on AWS and on-premises environments. The role emphasizes building batch/streaming pipelines, data cataloging, feature pipelines, and GenAI applications, with emphasis on MLOps, governance, and

Qualifications

• 10+ years of software engineering experience, with at least 2 years in a technical lead role.
• Expert level experience in Python with solid software engineering fundamentals including testing, code reviews, branching policies, version control, design patterns, and API design.
• Experience building data products leveraging Snowflake, DBT, and Airflow.
• Experience delivering classical ML and GenAI systems to production, including MLOps practices such as model registries, pipeline automation, and monitoring.
• Deep AWS experience building data and AI platforms including S3 Tables, Glue, Athena, Lake Formation, SageMaker, Bedrock, Lambda, ECS, RDS, IAM, and VPC.
• Experience in designing and implementing Infrastructure as Code (IaC) using Terraform, AWS CDK, or CloudFormation along with deployment automation with CI/CD pipelines.
• Practical experience with AI-assisted software development workflows using tools such as Claude Code or Cursor to design, implement, test, and refactor code.
• Experience in structuring work for AI coding agents such as writing clear specifications, providing context, breaking tasks down, and reviewing generated output critically.
• Clear communication skills and the ability to work with both technical and non-technical stakeholders.

Responsibilities

• Design and build reliable batch and streaming pipelines, including ingestion of industrial time-series and operational data alongside unstructured document sources
• Establish data modeling, quality, lineage, and cataloging practices for data product development
• Build feature pipelines and curated datasets that serve analytics, ML, and GenAI use cases
• Productionize classical ML models following MLOps practices
• Build GenAI applications such as retrieval-augmented generation over enterprise documents, including chunking, embeddings, vector search, prompt management, and evaluation
• Implement guardrails, observability, and cost controls for LLM-based systems
• Define evaluation approaches that make AI quality measurable and regressions visible
• Build self-service tooling, reusable templates, and CI/CD pipelines so teams can ship data and AI products consistently on AWS and on-premises environments
• Own infrastructure as code, observability, and reliability practices for the platform
• Embed security, governance, and compliance requirements into the platform by design
• Define and implement best practices around AI Assisted Software Development Life Cycle
• Mentor junior engineers in design and code reviews and implementation best practices

Skills

Python

Snowflake

DBT

Airflow

AWS

Terraform

CI/CD

GenAI

LLMs

Software Architecture

Tools

Claude Code

Cursor

Terraform

AWS CDK

Job description

Enterprise Products Partners L.P. is seeking an experienced Analyst, Lead Data Engineering to architect and implement a cloud-based Data & AI platform supporting AI and analytics across midstream energy units.

You will shape architecture, set standards, and productionize data products and AI models on AWS and on-premises environments. The role emphasizes building batch/streaming pipelines, data cataloging, feature pipelines, and GenAI applications, with emphasis on MLOps, governance, and