Senior Data Engineer: Real-Time & Batch Pipelines (Remote)

Committedstaff; United States; Remote; USD 140,000 - 200,000; Full time

Job summary

Committed Staff is seeking a Senior Data Engineer with deep expertise in AWS, batch processing, and real‑time streaming architectures. This hands‑on role designs and owns scalable data pipelines for GBs to TBs while collaborating with analysts, data scientists, and business teams.

You will mentor junior engineers, build data lakes/lakehouses on AWS, integrate APIs and SaaS sources, implement governance and security, and ensure reliability with monitoring and fault‑tolerant pipelines using Flink,

Qualifications

• 5+ years of experience in Data Engineering.
• Hands‑on AWS experience: S3, Glue, Athena, Lambda, EMR, Kinesis, IAM
• Strong batch processing experience with Spark/PySpark at scale
• Proven real‑time streaming experience with Apache Flink in production
• Experience building data lakes or data warehouses in production
• Experience integrating external systems via APIs

Responsibilities

• Build and maintain data lake / lakehouse architectures on AWS (S3‑based).
• Integrate multiple data sources — APIs, SaaS platforms, databases.
• Implement monitoring, logging, and alerting for all pipelines.
• Collaborate with analysts, data scientists, and business teams.
• Mentor junior engineers and enforce best practices.
• Own pipelines end‑to‑end in production.

Skills

Python

SQL

Scala

PySpark

Apache Flink

Apache Kafka

AWS Cloud

IAM Security

Real-time streaming

Batch processing

Tools

AWS Glue

Amazon EMR

Apache Airflow

Kinesis

Job description

Committed Staff is seeking a Senior Data Engineer with deep expertise in AWS, batch processing, and real‑time streaming architectures. This hands‑on role designs and owns scalable data pipelines for GBs to TBs while collaborating with analysts, data scientists, and business teams.

You will mentor junior engineers, build data lakes/lakehouses on AWS, integrate APIs and SaaS sources, implement governance and security, and ensure reliability with monitoring and fault‑tolerant pipelines using Flink,