Data Engineering Lead

Cloud Analytics Technologies, LLC; Irving (TX); On-site; USD 84,000 - 120,000; Full time

Job summary

Cloud Analytics Technologies, LLC is seeking a senior Data Engineer to architect and optimize large-scale data pipelines and microservices in Python. You will drive cloud-native ingestion and streaming solutions using AWS services like Kinesis, S3, DynamoDB, and EventBridge.

You will lead teams to deploy production-grade data platforms, ensure data integrity, and collaborate with business and engineering stakeholders to translate complex requirements into scalable designs.

Qualifications

• Bachelor’s degree in Computer Science, Computer Engineering, or related field.
• 8+ years of experience in data engineering or related disciplines with increasing responsibility.
• Extensive experience with AWS data services (Kinesis, S3, DynamoDB, EventBridge) at scale.
• Strong Python development and production deployment experience.
• Experience leading teams to build high-throughput, scalable data pipelines.
• Strong SQL skills with data quality and validation practices.

Responsibilities

• Define solution architecture with Principal Engineers and Data Architects.
• Lead design and optimization of scalable data pipelines and microservices in Python.
• Develop cloud-native data ingestion and streaming solutions on AWS.
• Own data integration frameworks and source data pipelines supporting CI Autonomy initiatives.
• Collaborate with business, product, and engineering stakeholders to translate requirements into scalable architectures.
• Establish automated testing, data quality controls, and validation across distributed data ecosystems.
• Lead monitoring and performance tuning usingCloudWatch for high availability and reliability.

Skills

Python

AWS

SQL

Data pipelines

Education

Bachelor’s degree

Tools

Kinesis

S3

DynamoDB

EventBridge

CI/CD

Jira

Jenkins

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Duration: 8 Months + Extension
Bill Rate: $74/hour (C2C)
Job Type: W-2, 1099, C2C
Client: To Be Discussed Later
Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description:

• Actively collaborate with Principal Software Engineers and Data
  Architects to define solution architecture
  Lead the solution design and optimization of scalable data pipelines
  and microservices in Python, enabling both real-time and batch data
  processing across enterprise platforms
• Drive the development of cloud-native data ingestion and streaming
  solutions leveraging AWS services including Kinesis, S3, DynamoDB,
  EventBridge, and related technologies
• Own the design, implementation, and operational excellence of data
  integration frameworks and source data pipelines supporting CI Autonomy
  initiatives
• Partner with business, product, and engineering stakeholders to
  translate complex requirements into scalable data architectures, workflows,
  mappings, and system designs
• Establish and enforce automated testing, data quality controls, and
  validation frameworks to ensure integrity, reliability, and compliance
  across distributed data ecosystems
• Lead operational monitoring, performance tuning, and root-cause
  analysis of production data platforms using observability tools such as
  CloudWatch to maintain high availability and service reliability
• Bachelor’s degree in Computer Science, Computer Engineering, or
  related field
• 8+ years of experience in data engineering or related disciplines
  with increasing responsibility
• Extensive experience on modern, large scale, complex Caterpillar data
  platforms such as Helios Data Platform
• Strong foundation developing and deploying Python solutions to a
  production environment
• Experience leading teams to build high-throughput, scalable data
  pipelines
• Strong hands‑on experience with AWS data services (Kinesis, S3,
  DynamoDB, EventBridge, etc.) at scale
• Strong in SQL, including data quality and validation practices
• Experience in deploying software using CI/CD tools such as Azure
  DevOps, Jira, Jenkins, etc.
• Experience developing microservices that support real-time data
  ingestion
• Experience developing software applications using relational and
  noSQL databases
• Ability to ensure data integrity across distributed and streaming
  systems
• Experience with monitoring, testing, and automation in large-scale
  data environments

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Some of our Business Ventures - Knowledge Analytics

Do restrictive necessities, your capability for advanced analytics or your business model need custom-made applications and infrastructure configurations? opt for our hosted managed services to urge comprehensive analytics solutions supported industry-leading Cloud Analytics Technologies, LLC code - delivered by the Cloud Analytics Technologies, LLC consultants UN agency understand these solutions best. Through a mix of with-it technology and dedicated service and support, Cloud Analytics Technologies, LLC hosted managed services will facilitate you: