Senior Data Engineer

ValueMomentum; Illinois; On-site; USD 120,000 - 160,000; Full time

Job summary

ValueMomentum is seeking an experienced Cloud Data Platform Engineer to design and build scalable data pipelines on AWS. You will develop ingestion, processing, and quality validation components using Python, PySpark, Snowflake, and dbt, while orchestrating with Prefect.

Collaboration with global teams and strong communication are essential. Demonstrated expertise in data engineering, DevOps, and cloud-native tooling will drive production readiness and reliability across complex pipelines in a

Qualifications

• 8-10 years overall work experience in CS/Engineering field.
• 5+ years in building Cloud Data Platform data engineering solutions.
• Strong Python, PySpark, SparkSQL coding with tests and version control.
• Hands-on with Snowflake, dbt, Airbyte and Prefect orchestration.
• Experience with AWS data ecosystem and DevOps practices.

Responsibilities

• Design and build cloud data platform data pipelines for ingestion, processing, quality validations, and integration.
• Develop reliable Prefect workflows with retry and recovery for production pipelines.
• Work with S3-based data lakes, Databricks on AWS, and related services.
• Monitor, troubleshoot, and optimize end-to-end data workflows via Prefect UI/Cloud.
• Collaborate with on‑shore/off‑shore teams and ensure smooth solution delivery.

Skills

Python

PySpark

SparkSQL

Airbyte

Prefect

Snowflake

dbt

Databricks on AWS

AWS

Testing & CI

Software engineering practices

Education

B.E./B.Tech degree in Computer Science, Engineering, or a related field

Tools

Snowpark

Airbyte

Terraform

Docker

AWS CodePipeline/CodeBuild/CodeDeploy

CloudWatch

Secrets Manager/Parameter Store

MWAA / Airflow

EventBridge

Job description

Cloud Platform: AWS (Compute, Storage, Networking, IAM and related cloud resources)

Data Engineering Skills: Python, PySpark, SQL, Prefect, Snowflake, Snowpark, dbt, airbyte, Databricks on AWS

DevOps Skills: CI/CD (AWS CodePipeline/CodeBuild/CodeDeploy), Terraform (IaC), Docker, Amazon CloudWatch, Secrets Management, Release & Environment Management

Must-Have Skills

• B.E./B.Tech degree in Computer Science, Engineering, or a related field, with 8-10 years of overall work experience.
• 5+ years of hands‑on development experience building Cloud Data Platform Data Engineering solutions covering Data Ingestion, Data Quality Validations, Data Processing and Data Integration.
• Strong hands‑on coding experience in Python, PySpark, and SparkSQL, with solid software engineering practices including testing, version control and code reviews.
• Hands‑on experience in Airbyte, Snowflake and dbt.
• Hands‑on experience with Prefect for workflow orchestration, including designing flows and tasks, scheduling, deployments, and parameterized runs.
• Ability to build reliable, observable Prefect workflows with retry logic, failure handling, and rerun/recovery support for production pipelines.
• Hands‑on experience with Amazon S3-based data lakes, Databricks on AWS and other AWS-based data ecosystem services.
• Experience monitoring and troubleshooting orchestrated workflows via the Prefect UI/Cloud, including work pools, deployments and run history.
• Demonstrated willingness and ability to set up and own DevOps practices for the platforms you build (see DevOps Skills below).
• Proficiency in analytics use‑case analysis, source system analysis, and data quality assessment.
• Experience coordinating/collaborating with on‑shore and off‑shore teams for solution delivery.
• Excellent communication and presentation skills.

DevOps Skills

• Design and build CI/CD pipelines using AWS CodePipeline, CodeBuild, and CodeDeploy (or equivalent tools such as GitHub Actions/Jenkins) to automate build, test, and release cycles.
• Provision and manage AWS cloud resources using Infrastructure as Code (Terraform), including version‑controlled, reusable modules.
• Containerize applications and workflows with Docker; deploy and manage containers using Amazon ECS/EKS.
• Manage promotion of code and configuration across Development, Test, and Production environments with clear release and rollback strategies (blue‑green/canary deployments).
• Implement secure secrets and configuration management using AWS Secrets Manager or Parameter Store, with least‑privilege IAM policies.
• Set up monitoring, logging, and alerting using Amazon CloudWatch (metrics, dashboards, alarms) to maintain operational visibility into pipelines and workflows.
• Define and implement retry, recovery, and rerun strategies for failed jobs/workflows to ensure production reliability.
• Write automation scripts (Python/Bash) for deployment, operational tasks, and routine platform maintenance.
• Collaborate with data engineers, application developers, and cloud engineers to support end‑to‑end delivery, and troubleshoot production issues when required.

Nice-to-Have Skills

• Experience with Snowpark, and additional orchestration frameworks such as Amazon MWAA (Managed Airflow) or AWS Step Functions.
• Exposure to Docker and containerized deployments; familiarity with Amazon ECS/EKS (Kubernetes) is a plus.
• Experience with event‑driven architectures (Amazon EventBridge, SQS, SNS, Lambda) and REST API integration across distributed services.
• Exposure to Data Management principles including Data Governance, Data Cataloging, and Master Data Management.
• Exposure to cloud‑native MDM tooling.
• Exposure to BI analytics using Amazon QuickSight.