Data Architect

Incedo Inc.; United States; On-site; USD 120,000 - 150,000; Full time

Job summary

A leading IT services firm is seeking a Data Architect to design and implement scalable data pipelines using AWS technologies. This mid-senior level role involves leading the development of real-time data solutions and ensuring data quality across platforms. Ideal candidates will have over 6 years of data engineering experience and strong proficiency in AWS Glue and Apache Kafka. Join our team in fostering innovation for financial applications!

Qualifications

• 6+ years of hands-on data engineering experience in cloud-based environments (AWS preferred).
• Strong experience with Apache Kafka / AWS MSK including topic design and partitioning.
• Proficiency in AWS Glue (PySpark) and Flink / MSF for ETL.

Responsibilities

• Lead development of real-time data pipelines using AWS DMS and MSK.
• Build and optimize streaming and batch data pipelines using AWS Glue.
• Implement monitoring, logging, and event-based alerting.

Skills

Data engineering in cloud-based environments

Apache Kafka / AWS MSK

AWS Glue (PySpark)

AWS DMS

CloudWatch monitoring

Infrastructure as code (CDK/Terraform)

Tools

AWS Glue

Apache Kafka

AWS Flink

DynamoDB

Iceberg

Job description

Assistant Manager - Talent Acquisition @ Incedo Inc. | Global Hiring | Mastering LinkedIn Recruiter

About the Role:

We are seeking a Data Architect / Technical Lead with strong expertise in AWS-based streaming data pipelines, Apache Kafka (MSK), AWS Glue, AWS Flink (MSF), and PySpark, to help design and implement a scalable data ingestion, processing, and reconciliation framework.

You will play a key role in building out the Ingress (CDC) and TPM (Transformation & Processing) layers, and drive the data quality and reconciliation workflows for a high-impact data platform serving financial applications.

Key Responsibilities:

• Lead the development of real‑time data pipelines using AWS DMS, MSK, and Flink (MSF) for CDC ingestion from multiple SQL Server sources (RDS/on‑prem).
• Build and optimize streaming and batch data pipelines using AWS Glue (PySpark) to validate, transform, and normalize data to Iceberg and DynamoDB.
• Define and enforce data quality, lineage, and reconciliation logic with support for both streaming and batch use cases.
• Integrate with S3 Bronze/Silver layers and implement efficient schema evolution and partitioning strategies using Iceberg.
• Collaborate with architects, analysts, and downstream application teams to design API and file‑based egress layers.
• Implement monitoring, logging, and event‑based alerting using CloudWatch, SNS, and EventBridge.
• Mentor junior developers and enforce best practices for modular, secure, and scalable data pipeline development.

Required Skills:

• 6+ years of hands‑on data engineering experience in cloud‑based environments (AWS preferred)
• Strong experience with Apache Kafka / AWS MSK including topic design, partitioning, and Kafka Connect/Debezium
• Proficiency in AWS Glue (PySpark) and Flink / MSF for both batch and streaming ETL
• Working knowledge of AWS DMS, S3, Lake Formation, DynamoDB, and Iceberg
• Solid grasp of schema evolution, CDC patterns, and data reconciliation frameworks
• Experience with infrastructure‑as‑code (CDK/Terraform) and DevOps practices (CI/CD, Git)

Preferred:

• Familiarity with PostgreSQL, Athena, Step Functions, and Lambda
• Knowledge of event‑driven architectures and microservices‑based data APIs

Seniority Level

Mid‑Senior level

Employment Type

Full‑time

Job Function

Consulting

Industries

IT Services and IT Consulting, Financial Services, and Investment Banking