Sr. Data Engineer – AWS & Streaming - Full Time

Sabio infotech; Fort Mill (SC); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Sabio infotech is seeking a Sr. Data Engineer -- AWS & Streaming to design, build, and optimize scalable batch and streaming data pipelines in a hybrid Fort Mill/NYC environment.

The role emphasizes strong Python/PySpark skills, AWS Glue, Lambda, and Kafka-based streaming, with a focus on data quality and reliable analytics platforms.

Qualifications

• Hands-on Python and PySpark required.
• Proven AWS Glue, Lambda experience.
• Strong Kafka ecosystem knowledge.
• Experience building data quality frameworks (validation, profiling, monitoring).
• Understanding of distributed data processing and scalable architectures.

Responsibilities

• Develop and maintain scalable ETL/ELT pipelines using AWS Glue, PySpark, and Python.
• Build event-driven workflows using AWS Lambda.
• Design and manage real-time streaming solutions using Kafka, KSQL, and Apache Flink.
• Implement data quality frameworks with validation, profiling, monitoring, and reconciliation.
• Optimize data processing performance, scalability, reliability, and cost in cloud environments.
• Collaborate with cross-functional teams to deliver production-grade data platforms.

Skills

Python

PySpark

AWS Lambda

Kafka ecosystem

Data quality frameworks

Distributed data processing

Tools

AWS Glue

Apache Flink

KSQL

Terraform/CloudFormation

Job description

Job Title: Sr. Data Engineer -- AWS & Streaming

Location: Fort Mill SC/Charlotte, NC/New York, NY (2-3 Days Hybrid)

Position: Full-Time

Duration: 6 Months to Long Term

Experience level -- 10 Years

Role Summary:

We are seeking a Mid--Senior Data Engineer with strong expertise in AWS-based data engineering, real-time streaming technologies, and enterprise-grade data quality frameworks. The ideal candidate will design, build, and optimize scalable batch and streaming data pipelines, implement robust data validation and monitoring processes, and support mission-critical analytics platforms.

Key Responsibilities:

• Develop and maintain scalable ETL/ELT pipelines using AWS Glue, PySpark, and Python
• Build event-driven workflows using AWS Lambda
• Design and manage real-time streaming solutions using Kafka, KSQL, and Apache Flink
• Implement and enforce comprehensive data quality frameworks, including validation, profiling, monitoring, and reconciliation
• Optimize data processing performance, scalability, reliability, and cost in cloud environments
• Collaborate with cross-functional teams to deliver reliable, production-grade data platforms and ensure data integrity across the pipeline

Required Skills:

• Strong hands-on experience with Python and PySpark
• Proven expertise in AWS Glue, Lambda, and other cloud-native data services
• Solid experience with the Kafka ecosystem (topics, partitions, consumer groups, streaming patterns)
• Demonstrated experience building and supporting data quality frameworks (validation rules, reconciliation checks, profiling, anomaly detection)
• Strong understanding of distributed data processing and scalable architecture patterns

Good-to-Have Skills

• Experience with Apache Flink for real-time stream processing and stateful computations
• Knowledge of KSQL or other streaming SQL engines
• Exposure to CI/CD pipelines, IaC (Terraform/CloudFormation), and DevOps practices
• Familiarity with data lake/lakehouse architectures and table formats such as Iceberg, Delta, or Hudi
• Experience working in enterprise or financial data environments