Senior Python Data Engineer — PySpark, AWS

Fourways Consulting; New York, Fort Mill (NY, SC); On-site; USD 120,000 - 180,000; Full time

Job summary

Fourways Consulting is seeking an experienced Python Developer for a 12+ month contract across NYC, NY and Fort Mill, SC. You will design, build, and optimize high-throughput distributed data pipelines and serverless cloud integrations on AWS, using Python, PySpark, AWS Glue, and AWS Lambda.

You will collaborate with solution architects and data engineers to translate complex requirements into scalable solutions, implement CI/CD pipelines, enforce automated tests, and optimize memory usage in

Qualifications

• Experience in backend Python development and data pipeline engineering.
• Proficient in PySpark for distributed data processing and DataFrame manipulation.
• Extensive experience deploying serverless functions with AWS Lambda.
• Hands-on expertise building, scheduling, and monitoring AWS Glue ETL pipelines.

Responsibilities

• Design, develop, and maintain high-performance, scalable data processing apps and backend services using Python.
• Build and optimize distributed data transformation workflows with PySpark across large datasets.
• Develop, configure, and orchestrate serverless event-driven workflows with AWS Lambda.
• Architect, catalog, and schedule automated ETL/ELT pipelines using AWS Glue.
• Manage Amazon S3 data lake storage and optimize query access across cloud databases.
• Troubleshoot bottlenecks, optimize memory in PySpark, and reduce cloud costs.
• Implement automated unit tests, integration tests, and data quality checks to ensure pipeline resilience.
• Collaborate with agile teams to participate in sprint ceremonies and code reviews.

Skills

Python

PySpark

AWS Lambda

AWS Glue

SQL

AWS S3

CloudWatch IAM

Tools

Docker

Apache Airflow

Databricks

Redshift

Snowflake

Job description

Fourways Consulting is seeking an experienced Python Developer for a 12+ month contract across NYC, NY and Fort Mill, SC. You will design, build, and optimize high-throughput distributed data pipelines and serverless cloud integrations on AWS, using Python, PySpark, AWS Glue, and AWS Lambda.

You will collaborate with solution architects and data engineers to translate complex requirements into scalable solutions, implement CI/CD pipelines, enforce automated tests, and optimize memory usage in