AWS Data Engineer — End-to-End Pipelines, Data Quality

Incedo Inc.; Austin (TX); Hybrid; USD 120.000 - 160.000; Vollzeit

Zusammenfassung

Incedo Inc. in Austin, TX is seeking an experienced Data Engineer to design and build production-grade data pipelines across ingestion, transformation, storage, and consumption layers in a hybrid work environment.

You will own end-to-end data architectures, implement robust data quality checks, and optimize PySpark/SQL workloads on AWS-native stacks (S3, Glue, Athena) using Airflow or Step Functions, ensuring scalable, reliable data flows for business insights.

Qualifikationen

• 8+ years of hands-on data engineering experience building production data pipelines.
• End-to-end design and ownership of data pipelines across ingestion, transformation, storage and consumption.
• Experience with AWS-native data stack including S3, Glue (PySpark), Athena, and orchestration tools such as Airflow or Step Functions.
• Experience building automated data quality checks into pipelines: validation rules, reconciliation logic, anomaly detection.
• Strong understanding of data modeling and storage design including partitioning strategies and handling of small files.
• Hands-on data modeling expertise (dimensional modeling, normalization, slowly changing dimensions) for new datasets.
• Ownership of data quality frameworks with validation rules, thresholds and exception handling.
• Experience defining testing standards for data pipelines and driving adoption.
• Ability to make architectural trade-offs between Glue, Lambda, EMR or container-based approaches.
• Solid PySpark and SQL skills with performance tuning in distributed environments.
• Experience building reliable pipelines with error handling, retries, and monitoring.
• Experience reconciling data across sources and targets to secure production sign-off.

Aufgaben

• Design and own end-to-end data pipelines across ingestion, transformation, storage and consumption.
• Implement data quality checks and monitoring for pipelines.
• Optimize PySpark/SQL workloads on AWS-native stacks for performance and reliability.
• Make architectural decisions and trade-offs based on use case and scale.
• Reconcile data across source and target systems and secure production sign-off.

Kenntnisse

Data engineering

End-to-end pipelines

Data quality

Data modeling

Performance tuning

Production-grade pipelines

Monitoring & troubleshooting

Architectural decision-making

Data reconciliation

Data pipeline ownership

Tools

S3

Glue (PySpark)

Athena

Airflow

Step Functions

PySpark

SQL

EMR

Lambda

Jobbeschreibung

Incedo Inc. in Austin, TX is seeking an experienced Data Engineer to design and build production-grade data pipelines across ingestion, transformation, storage, and consumption layers in a hybrid work environment.

You will own end-to-end data architectures, implement robust data quality checks, and optimize PySpark/SQL workloads on AWS-native stacks (S3, Glue, Athena) using Airflow or Step Functions, ensuring scalable, reliable data flows for business insights.