Senior AWS Data Engineer — Python & PySpark

Valiant Technologies LLC; Reston (VA); On-site; USD 120,000 - 180,000; Full time

Job summary

VALIANT TECHNOLOGIES LLC is seeking a Data Engineer to build and maintain ETL pipelines using Python and PySpark on AWS Glue and other compute platforms. You will orchestrate workflows with AWS Step Functions and serverless components, implement messaging with SNS/SQS, and optimize Redshift storage and queries.

You will write performant SQL, ensure data quality and provide monitoring, while collaborating with stakeholders to translate requirements into robust data solutions.

Qualifications

• Strong experience with Python and PySpark for large-scale data processing.

Responsibilities

• Build and maintain ETL pipelines using Python and PySpark on AWS Glue and other compute platforms.
• Orchestrate workflows with AWS Step Functions and serverless components ( Lambda ).
• Implement messaging and event-driven patterns using AWS SNS and SQS.
• Design and optimize data storage and querying in Amazon Redshift.
• Write performant SQL for data transformations, validation, and reporting.
• Ensure data quality, monitoring, error handling and operational support for pipelines.
• Collaborate with data consumers, engineers, and stakeholders to translate requirements into solutions.
• Contribute to CI/CD , infrastructure-as-code, and documentation for reproducible deployments.

Skills

Python

PySpark

SQL

Data modeling

ETL best practices

Git

CI/CD

AWS services

Tools

AWS Lambda

AWS SNS

AWS SQS

AWS Glue

Amazon Redshift

AWS Step Functions

Job description

VALIANT TECHNOLOGIES LLC is seeking a Data Engineer to build and maintain ETL pipelines using Python and PySpark on AWS Glue and other compute platforms. You will orchestrate workflows with AWS Step Functions and serverless components, implement messaging with SNS/SQS, and optimize Redshift storage and queries.

You will write performant SQL, ensure data quality and provide monitoring, while collaborating with stakeholders to translate requirements into robust data solutions.