Senior Data Engineer – AWS & Real-Time Streaming

Incedo Inc.; New York (NY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Incedo Inc. in New York City, NY, is seeking a Mid–Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming, and data quality frameworks. The role focuses on designing, building, and optimizing scalable data pipelines that support enterprise analytics.

You will implement robust data validation, monitoring, and reconciliation, while collaborating with cross-functional teams to ensure data integrity and cost-efficient cloud operations.

Qualifications

• Mid–Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming technologies, and enterprise-grade data quality frameworks.
• Design, build, and optimize scalable batch and streaming pipelines in enterprise environments.
• Collaborate with cross-functional teams to ensure data integrity and production readiness.

Responsibilities

• Develop and maintain scalable ETL/ELT pipelines using AWS Glue, PySpark, and Python
• Design and manage real-time streaming solutions using Kafka, KSQL, and Apache Flink
• Implement and enforce comprehensive data quality frameworks, including validation, profiling, monitoring, and reconciliation
• Optimize data processing performance, scalability, reliability, and cost in cloud environments
• Collaborate with cross-functional teams to deliver reliable, production-grade data platforms and ensure data integrity across the pipeline

Skills

Python

PySpark

AWS Glue

Lambda

Kafka

Data quality frameworks

Distributed processing

Cloud architecture

Tools

Terraform

CloudFormation

Iceberg/Delta/Hudi

Apache Flink

KSQL

Job description

Incedo Inc. in New York City, NY, is seeking a Mid–Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming, and data quality frameworks. The role focuses on designing, building, and optimizing scalable data pipelines that support enterprise analytics.

You will implement robust data validation, monitoring, and reconciliation, while collaborating with cross-functional teams to ensure data integrity and cost-efficient cloud operations.