Senior Data Engineer - Physical AI Platform, Data Engineering

Caterpillar; Irving (TX); On-site; USD 120,000 - 180,000; Full time

Job summary

Caterpillar is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines, microservices, and cloud-based data platforms. You will work in an agile environment to improve data architecture, performance, reliability, and data quality across enterprise systems.

Responsibilities include implementing scalable data pipelines in Python/Java, enabling real-time and batch processing, and driving cloud-native ingestion with AWS services.

Qualifications

• Experience designing scalable data architectures and data platforms.
• Strong programming expertise in Python and Java for backend data systems.
• Hands-on with cloud-native data pipelines and services (AWS).
• Familiarity with agile development processes and CI/CD.

Responsibilities

• Actively collaborate with Lead Software Engineers and Data Architects to define and implement solution architecture
• Implement the solution design and optimize scalable data pipelines and microservices in Python, enabling real-time and batch data processing across enterprise platforms
• Drive the development of cloud-native data ingestion and streaming solutions leveraging AWS services including Kinesis, S3, DynamoDB, EventBridge, and related technologies
• Lead the design, implementation, and operational excellence of data integration frameworks and source data pipelines supporting CI Autonomy initiatives
• Partner with business, product, and engineering stakeholders to translate complex requirements into scalable data architectures, workflows, mappings, and system designs
• Establish and enforce automated testing, data quality controls, and validation frameworks to ensure integrity, reliability, and compliance across distributed data ecosystems
• Implement operational monitoring, performance tuning, and root-cause analysis of production data platforms using observability tools such as CloudWatch to maintain high availability and service reliability

Skills

Python

Java

Data pipelines

Cloud platforms

Agile

Tools

AWS

Kinesis

S3

DynamoDB

EventBridge

Job description

Career Area: Technology, Digital and Data

Job Description:

Your Work Shapes the World at Caterpillar Inc.

When you join Caterpillar, you’re joining a global team who cares not just about the work we do - but also about each other. We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable communities. We don’t just talk about progress and innovation here - we make it happen, with our customers, where we work and live. Together, we are building a better world, so we can all enjoy living in it.

Help Build the Future of Caterpillar -

At Caterpillar, technology always has a purpose, which is to solve our customers’ toughest challenges. Through Cat Technology, we are solving problems by building the intelligence layer that connects machines, data, and people to make jobsites safer, more productive, and more sustainable. By combining deep domain expertise in physical systems with software, connectivity, autonomy, and AI, we deliver solutions that work in the real world-on real jobsites, at global scale.

You’ll build and deploy against one of the most unique data foundations-over 1.6 million connected assets generating real-world data daily. These data and platform capabilities are enabling the development of AI models, edge computing architectures, and software systems that scale across fleets, products, and industries. The result will be a new generation of machines that continuously learn, improve, and deliver performance at scale.

Be Part of What’s Next in Autonomous Construction Sites

Construction autonomy is one of the most complex challenges in applied AI, and at Caterpillar, advancements in physical AI, simulation, sensing, and edge computing are turning things that once felt impossible-intelligent machines operating in dynamic jobsites-into reality.

Our connected ecosystem brings together massive volumes of high-quality data to create a foundation where engineers like you can build and deploy against.

In these roles, you’ll work at the intersection of the physical and digital worlds. You’ll help design and deliver intelligent systems that enable machines to perceive their environment, make informed decisions, and support safer, more productive operations.

Job Summary

As a Senior Data Engineer, you will design, build, and maintain scalable data pipelines, microservices, and cloud‑based data platforms that deliver reliable, high-quality data for business and engineering teams. Working in an agile environment, you will help develop data architecture, performance, reliability, and continuous improvement across modern data solutions.

What You Will Do:

• Actively collaborate with Lead Software Engineers and Data Architects to define and implement solution architecture
• Implement the solution design and optimize scalable data pipelines and microservices in Python, enabling both real-time and batch data processing across enterprise platforms
• Drive the development of cloud-native data ingestion and streaming solutions leveraging AWS services including Kinesis, S3, DynamoDB, EventBridge, and related technologies
• Lead the design, implementation, and operational excellence of data integration frameworks and source data pipelines supporting CI Autonomy initiatives
• Partner with business, product, and engineering stakeholders to translate complex requirements into scalable data architectures, workflows, mappings, and system designs
• Establish and enforce automated testing, data quality controls, and validation frameworks to ensure integrity, reliability, and compliance across distributed data ecosystems
• Implement operational monitoring, performance tuning, and root-cause analysis of production data platforms using observability tools such as CloudWatch to maintain high availability and service reliability

What You Will Have:

• Decision Making and Critical Thinking: Ability to analyze and resolve complex issues within distributed data platforms, designing scalable, and resilient solutions
• Effective Communications: Ability to communicate across teams by sharing feedback constructively, listening to others, and creating documentation that makes data systems and processes easy to understand and support
• Software Development : Extensive knowledge with design and development of backend systems and data pipelines using Python, Java, and modern frameworks, providing technical directions and ensuring the delivery of reliable, scalable solutions
• Software Development Life Cycle: Experience leading the delivery of data engineering solutions in an Agile environment by guiding work through the full development lifecycle, translating requirements into technical solutions, and ensuring projects are delivered with quality, reliability, and business value
• Software Integration Engineering: Capability to design and integrate APIs, data pipelines, streaming p