Lead Data Engineer — Autonomous Construction AI Platform

Caterpillar Inc.; Chicago (IL); On-site; USD 128,000 - 209,000; Full time

Benefits offered by this job

Medical, dental, and vision benefits

Paid time off plan

401(k) savings plans

Health Savings Account (HSA)

Flexible Spending Accounts (FSAs)

Career Development

Incentive bonus

Parental leave

Tuition Reimbursement

Job summary

Caterpillar Inc. in Chicago, IL is seeking a Lead Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data platforms.

You will work in an agile environment to improve data architecture, performance, reliability, and measurement across modern data solutions. You will lead the development of real-time and batch data processing, collaborate with engineers and product teams, and ensure data quality and security across distributed systems.

Qualifications

• Bachelor's degree in computer science or engineering and 8+ years in data engineering.
• Strong Python and SQL, with experience building scalable data pipelines.
• Proven ability to lead data integration and microservice design for real-time processing.
• Hands-on experience with AWS data services and CI/CD tooling required.
• Experience with large-scale distributed data platforms and data quality practices.

Responsibilities

• Actively collaborate with Principal Software Engineers and Data Architects to define solution architecture
• Lead the solution design and optimization of scalable data pipelines and microservices in Python, enabling both real-time and batch data processing across enterprise platforms
• Drive the development of cloud-native data ingestion and streaming solutions leveraging AWS services including Kinesis, S3, DynamoDB, EventBridge, and related technologies
• Own the design, implementation, and operational excellence of data integration frameworks and source data pipelines supporting CI Autonomy initiatives
• Partner with business, product, and engineering stakeholders to translate complex requirements into scalable data architectures, workflows, mappings, and system designs
• Establish and enforce automated testing, data quality controls, and validation frameworks to ensure integrity, reliability, and compliance across distributed data ecosystems
• Lead operational monitoring, performance tuning, and root-cause analysis of production data platforms using observability tools such as CloudWatch to maintain high availability and service reliability

Skills

Python

SQL

Data pipelines

Distributed systems

AWS

Microservices

NoSQL databases

CI/CD

Real-time data ingestion

Education

Bachelor’s degree in Computer Science, Computer Engineering, or related field

Tools

Kinesis

S3

DynamoDB

EventBridge

Azure DevOps

Jira

Jenkins

Job description

Caterpillar Inc. in Chicago, IL is seeking a Lead Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data platforms.

You will work in an agile environment to improve data architecture, performance, reliability, and measurement across modern data solutions. You will lead the development of real-time and batch data processing, collaborate with engineers and product teams, and ensure data quality and security across distributed systems.