Senior Data Engineer

Incedo Inc.; New York (NY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Incedo Inc. in New York City, NY, is seeking a Mid–Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming, and data quality frameworks. The role focuses on designing, building, and optimizing scalable data pipelines that support enterprise analytics.

You will implement robust data validation, monitoring, and reconciliation, while collaborating with cross-functional teams to ensure data integrity and cost-efficient cloud operations.

Qualifications

• Mid–Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming technologies, and enterprise-grade data quality frameworks.
• Design, build, and optimize scalable batch and streaming pipelines in enterprise environments.
• Collaborate with cross-functional teams to ensure data integrity and production readiness.

Responsibilities

• Develop and maintain scalable ETL/ELT pipelines using AWS Glue, PySpark, and Python
• Design and manage real-time streaming solutions using Kafka, KSQL, and Apache Flink
• Implement and enforce comprehensive data quality frameworks, including validation, profiling, monitoring, and reconciliation
• Optimize data processing performance, scalability, reliability, and cost in cloud environments
• Collaborate with cross-functional teams to deliver reliable, production-grade data platforms and ensure data integrity across the pipeline

Skills

Python

PySpark

AWS Glue

Lambda

Kafka

Data quality frameworks

Distributed processing

Cloud architecture

Tools

Terraform

CloudFormation

Iceberg/Delta/Hudi

Apache Flink

KSQL

Job description

Incedo Inc. is a high-growth Digital, Data and AI Transformation Specialist firm headquartered in New Jersey. We

are a long-term strategy execution partner for Fortune 500 enterprises, operating at the intersection of business

and technology across Banking & Payments, Wealth Management, Telecom, Hi-Tech, and Life Sciences.

We are building Incedo 4.0 - an AI-native, execution-focused, founder-led organization designed for scale, speed,

and long-term impact.

Incedo delivers ROI from AI @ Scale through the “Power of 3”:

Engineering & Operations excellence

Job Title: Sr.Data Engineer– AWS & Streaming

Location: Fort Mill, SC or New York City, NY (Hybrid)

Role Summary:

We are seeking a Mid–Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming technologies, and enterprise-grade data quality frameworks. The ideal candidate will design, build, and optimize scalable batch and streaming data pipelines, implement robust data validation and monitoring processes, and support mission-critical analytics platforms.

Key Responsibilities:

• Develop and maintain scalable ETL/ELT pipelines using AWS Glue, PySpark, and Python
• Design and manage real-time streaming solutions using Kafka, KSQL, and Apache Flink
• Implement and enforce comprehensive data quality frameworks, including validation, profiling, monitoring, and reconciliation
• Optimize data processing performance, scalability, reliability, and cost in cloud environments
• Collaborate with cross-functional teams to deliver reliable, production-grade data platforms and ensure data integrity across the pipeline

Required Skills:

• Strong hands-on experience with Python and PySpark
• Proven expertise in AWS Glue, Lambda, and other cloud-native data services
• Solid experience with the Kafka ecosystem (topics, partitions, consumer groups, streaming patterns)
• Demonstrated experience building and supporting data quality frameworks (validation rules, reconciliation checks, profiling, anomaly detection)
• Strong understanding of distributed data processing and scalable architecture patterns

Good-to-Have Skills:

• Experience with Apache Flink for real-time stream processing and stateful computations
• Knowledge of KSQL or other streaming SQL engines
• Exposure to CI/CD pipelines, IaC (Terraform/CloudFormation), and DevOps practices
• Familiarity with data lake/lakehouse architectures and table formats such as Iceberg, Delta, or Hudi
• Experience working in enterprise or financial data environments