Senior AWS Data Engineer: Real-Time Pipelines & Lakehouse

Incedo Inc.; Florham Park (NJ); On-site; USD 120,000 - 180,000; Full time

Job summary

Incedo Inc. seeks an experienced AWS Data Engineer to design, build, and optimize large-scale data pipelines and ETL workflows on AWS.

The role focuses on cloud-native data services, data modeling, and pipeline orchestration for enterprise environments. The candidate will implement incremental and CDC pipelines, metadata-driven ETL, and Iceberg-based optimizations while ensuring governance, security, and cost efficiency across data platforms.

Qualifications

• 12+ years of experience in Data Engineering with 5+ years building AWS-based data platforms and data lakes.
• Hands-on expertise with AWS Glue, DMS, S3, Redshift, Athena, Lambda, Step Functions, EventBridge, CloudWatch, SNS, and Glue Data Catalog.
• Advanced PySpark, Spark, Python, SQL skills; experience building reusable ETL/ELT frameworks and high-volume data transformations.
• Experience designing incremental and CDC pipelines with AWS DMS, Glue, and related services.
• Strong Iceberg knowledge: partitioning, schema evolution, metadata, and catalog implementations.
• Experience building enterprise Data Lakes with Bronze/Silver/Gold layers, governance, and self-service analytics.
• Experience orchestrating complex workflows with AWS Step Functions, Glue Workflows, Lambda, EventBridge.
• Data modeling expertise across dimensional, star, snowflake, and lakehouse models.
• Performance tuning and cost optimization for Glue/Spark workloads (partitioning, bookmarks, tuning).
• Experience with real-time and batch architectures using Kafka, Kinesis, PostgreSQL, S3, Redshift.
• Data quality, lineage, observability, monitoring, and governance across enterprise platforms.

Responsibilities

• Design incremental and CDC data pipelines using AWS Glue, DMS, and Iceberg for near real-time analytics.
• Create metadata-driven ETL frameworks for reusability and scalability across data platforms.
• Implement partitioning, compaction, and optimization strategies for Iceberg datasets to cut latency and cost.
• Build and orchestrate complex workflows with AWS Step Functions, EventBridge, Lambda, and Glue Workflows.
• Tune Spark configurations and job parameters for performance and cost efficiency in AWS Glue jobs.
• Develop CI/CD pipelines for data engineering solutions using AWS CodePipeline, CodeBuild, GitHub, Jenkins, or Terraform.
• Design scalable data lake architectures following AWS best practices with governance and reliability.
• Automate data validation and reconciliation to ensure data accuracy across systems.
• Create and maintain Athena external tables, Iceberg catalogs, and Glue Data Catalog metadata for discovery.
• Enforce RBAC, data masking, encryption, and audits using Lake Formation and IAM policies.
• Support real-time and batch processing architectures integrating Kafka, Kinesis, PostgreSQL, S3, and Redshift.
• Monitor pipelines with CloudWatch, SNS, Glue Monitoring, and custom alerts to meet SLAs.

Skills

Data Engineering

AWS Cloud

PySpark

SQL

ETL/ELT

CDC pipelines

Iceberg

Data Lake

Spark tuning

Data Modeling

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or related field

Tools

AWS Glue

AWS DMS

Amazon S3

Amazon Redshift

Athena

Lambda

Step Functions

EventBridge

CloudWatch

SNS

Glue Data Catalog

Apache Iceberg

Kafka

Kinesis

PostgreSQL

Terraform

Job description

Incedo Inc. seeks an experienced AWS Data Engineer to design, build, and optimize large-scale data pipelines and ETL workflows on AWS.

The role focuses on cloud-native data services, data modeling, and pipeline orchestration for enterprise environments. The candidate will implement incremental and CDC pipelines, metadata-driven ETL, and Iceberg-based optimizations while ensuring governance, security, and cost efficiency across data platforms.