Senior Data Engineer

Committedstaff; United States; Remote; USD 140,000 - 200,000; Full time

Job summary

Committed Staff is seeking a Senior Data Engineer with deep expertise in AWS, batch processing, and real‑time streaming architectures. This hands‑on role designs and owns scalable data pipelines for GBs to TBs while collaborating with analysts, data scientists, and business teams.

You will mentor junior engineers, build data lakes/lakehouses on AWS, integrate APIs and SaaS sources, implement governance and security, and ensure reliability with monitoring and fault‑tolerant pipelines using Flink,

Qualifications

• 5+ years of experience in Data Engineering.
• Hands‑on AWS experience: S3, Glue, Athena, Lambda, EMR, Kinesis, IAM
• Strong batch processing experience with Spark/PySpark at scale
• Proven real‑time streaming experience with Apache Flink in production
• Experience building data lakes or data warehouses in production
• Experience integrating external systems via APIs

Responsibilities

• Build and maintain data lake / lakehouse architectures on AWS (S3‑based).
• Integrate multiple data sources — APIs, SaaS platforms, databases.
• Implement monitoring, logging, and alerting for all pipelines.
• Collaborate with analysts, data scientists, and business teams.
• Mentor junior engineers and enforce best practices.
• Own pipelines end‑to‑end in production.

Skills

Python

SQL

Scala

PySpark

Apache Flink

Apache Kafka

AWS Cloud

IAM Security

Real-time streaming

Batch processing

Tools

AWS Glue

Amazon EMR

Apache Airflow

Kinesis

Job description

ABOUT THE ROLE

We are looking for a Senior Data Engineer with deep expertise in AWS, batch processing, and real‑time streaming architectures. This is a hands‑on role — you will design and own scalable data pipelines serving both high‑throughput batch workloads (GBs to TBs) and low‑latency streaming systems. You will collaborate with analysts, data scientists, and business teams, and help mentor junior engineers to raise the bar across the data platform.

Batch Processing

• Design & implement batch pipelines for large‑scale processing (GBs to TBs)
• Build ETL/ELT workflows using PySpark, SQL, Glue, and EMR
• Optimize jobs for performance, partitioning, and cost
• Manage data ingestion from DBs, files, and APIs into the data lake
• Ensure data quality, consistency, and reliability

Real‑Time / Streaming

• Design & implement real‑time pipelines using Apache Flink (Scala preferred)
• Process high‑throughput streams with low‑latency requirements
• Implement stateful processing, windowing, and event‑time handling
• Handle late/out‑of‑order events; exactly‑once / at‑least‑once semantics
• Configure checkpointing, fault tolerance, and recovery
• Integrate with Kafka or Kinesis

CORE RESPONSIBILITIES

• Build and maintain data lake / lakehouse architectures on AWS (S3‑based)
• Integrate multiple data sources — APIs, SaaS platforms, databases
• Ensure data governance, security, and access control (IAM)
• Implement monitoring, logging, and alerting for all pipelines
• Collaborate with analysts, data scientists, and business teams
• Mentor junior engineers and enforce best practices

TECH STACK

• Python
• SQL
• Scala
• PySpark
• Apache Flink
• Apache Kafka
• AWS Glue
• Amazon EMR
• Amazon S3
• AWS Lambda
• Amazon Athena
• Amazon Kinesis
• Delta Lake / Iceberg

REQUIRED QUALIFICATIONS

• 5+ years of experience in Data Engineering
• Strong skills in Python, SQL, and Scala
• Hands‑on AWS experience: S3, Glue, Athena, Lambda, EMR, Kinesis, IAM
• Strong batch processing experience with Spark / PySpark at scale
• Proven real‑time streaming experience with Apache Flink in production
• Solid understanding of batch vs streaming trade‑offs and event‑driven architectures
• Experience building data lakes or data warehouses in production
• Experience integrating external systems via APIs
• Distributed systems fundamentals: fault tolerance, partitioning, consistency

WHAT MAKES A STRONG CANDIDATE

• Has built both batch AND real‑time pipelines in production — not just one or the other
• Knows when to apply each paradigm: batch for cost‑efficient large volumes, streaming for low‑latency real‑time use cases
• Experience handling millions of records in batch and high event throughput in streaming systems
• Strong debugging and performance optimization skills across distributed data systems
• Hands‑on mindset — you build things and own them end‑to‑end, not just design them

NICE TO HAVE

• AWS certifications (Data Analytics Specialty, Solutions Architect)
• Experience with Delta Lake, Apache Iceberg, or open table formats
• Familiarity with dbt for transformation workflows
• Experience with orchestration tools (Apache Airflow, AWS Step Functions)
• Background in IoT, telemetry, or event‑heavy domains
• OpenTelemetry or advanced observability experience

THIS ROLE IS NOT FOR YOU IF

• You have only worked with batch OR only with streaming — this role requires both
• You need a fully defined architecture and runbook before making engineering decisions
• You are not comfortable owning pipelines end‑to‑end in production without daily oversight
• You optimize for theoretical elegance before proving a real cost or reliability problem exists
• You are not comfortable communicating directly in English with international teams

WHY COMMITTED STAFF

Committed Staff connects the top 1% of LATAM talent with high‑growth U.S. companies. We don't just place candidates — we build committed teams that scale. You'll get direct access to a real data engineering challenge with meaningful scope, full‑time remote work aligned with U.S. time zones, competitive LATAM‑market compensation, and a growing network of senior tech professionals across Latin America.