AWS PySpark Data Engineer - Python Pipelines

Polarits; Newark (NJ); On-site; USD 140,000 - 190,000; Full time

Job summary

Polarits in Newark, New Jersey, is seeking an experienced Python developer with strong AWS and PySpark expertise to join our data engineering team. You will design, build, and maintain scalable data pipelines and processing workflows in cloud environments.

The ideal candidate has 10+ years in software development with Python, hands-on PySpark, and a solid AWS background (S3, Glue, EMR, Redshift). You will collaborate across teams to deliver reliable, well-documented data solutions while

Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, or a related field.
• 10+ years of experience in software development with a strong focus on Python.
• Hands-on experience with PySpark for distributed data processing.
• Solid understanding of AWS cloud services such as S3, Glue, Lambda, EMR, Redshift, and Athena.
• Strong experience in ETL development and data pipeline orchestration.
• Familiarity with SQL and relational/non-relational databases.
• Excellent analytical, debugging, and communication skills.

Responsibilities

• Design, develop, and maintain data pipelines and ETL workflows using Python, PySpark, and AWS services.
• Build and optimize large-scale data processing and data transformation solutions.
• Integrate various data sources and ensure data quality, performance, and reliability.
• Collaborate with data engineers, analysts, and architects to deliver end-to-end data solutions.
• Implement best practices for code optimization, error handling, and data validation.
• Participate in code reviews, documentation, and deployment automation.
• Ensure adherence to data security and compliance standards.

Skills

Python

PySpark

AWS

ETL

SQL

Distributed processing

Communication

Debugging

Education

Bachelor's degree in Computer Science or related field

Tools

Airflow

Databricks

Git

Docker

Kubernetes

Job description

Polarits in Newark, New Jersey, is seeking an experienced Python developer with strong AWS and PySpark expertise to join our data engineering team. You will design, build, and maintain scalable data pipelines and processing workflows in cloud environments.

The ideal candidate has 10+ years in software development with Python, hands-on PySpark, and a solid AWS background (S3, Glue, EMR, Redshift). You will collaborate across teams to deliver reliable, well-documented data solutions while