Lead Data Engineer, Cloud Pipelines & AI

Caterpillar Inc.; Irving (TX); On-site; USD 128,000 - 209,000; Full time

Benefits offered by this job

Medical benefits

401(k) plan

HSA

FSAs

Career development

Incentive bonus

Job summary

Caterpillar in Irving, TX is seeking a Lead Data Engineer to design, build, and maintain scalable data pipelines, microservices, and cloud platforms delivering high‑quality data to business and engineering teams. Working in an agile environment, you will drive data architecture, performance, reliability, and continuous improvement across modern data solutions.

You will collaborate with Principal Engineers to define architectures, lead real-time and batch processing, and ensure data quality,

Qualifications

• Bachelor’s degree in Computer Science, Computer Engineering, or related field.
• 8+ years of data engineering experience with increasing responsibility.
• Experience with modern, large-scale data platforms and Python production use.
• Experience deploying software via CI/CD tools (Azure DevOps, Jira, Jenkins).
• Strong SQL, data quality practices, and data integrity across systems.

Responsibilities

• Actively collaborate with Principal Software Engineers and Data Architects to define solution architecture.
• Lead the design and optimization of scalable data pipelines and microservices in Python for real-time and batch processing.
• Develop cloud-native data ingestion and streaming solutions using AWS services (Kinesis, S3, DynamoDB, EventBridge).
• Own data integration frameworks and source data pipelines supporting autonomy initiatives.
• Translate complex requirements into scalable data architectures and workflows.
• Establish automated testing, data quality controls, and validation frameworks.
• Lead operational monitoring, performance tuning, and root-cause analysis with observability tools.

Skills

Python

Java

AWS

Data pipelines

SQL

CI/CD

Observability

Education

Bachelor’s degree in Computer Science or related field

Tools

Jira

Azure DevOps

Jenkins

AWS data services

Job description

Caterpillar in Irving, TX is seeking a Lead Data Engineer to design, build, and maintain scalable data pipelines, microservices, and cloud platforms delivering high‑quality data to business and engineering teams. Working in an agile environment, you will drive data architecture, performance, reliability, and continuous improvement across modern data solutions.

You will collaborate with Principal Engineers to define architectures, lead real-time and batch processing, and ensure data quality,