AWS Python Developer with Pyspark Newark, NJ, New Jersey

Polarits; Newark (NJ); On-site; USD 140,000 - 190,000; Full time

Job summary

Polarits in Newark, New Jersey, is seeking an experienced Python developer with strong AWS and PySpark expertise to join our data engineering team. You will design, build, and maintain scalable data pipelines and processing workflows in cloud environments.

The ideal candidate has 10+ years in software development with Python, hands-on PySpark, and a solid AWS background (S3, Glue, EMR, Redshift). You will collaborate across teams to deliver reliable, well-documented data solutions while

Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, or a related field.
• 10+ years of experience in software development with a strong focus on Python.
• Hands-on experience with PySpark for distributed data processing.
• Solid understanding of AWS cloud services such as S3, Glue, Lambda, EMR, Redshift, and Athena.
• Strong experience in ETL development and data pipeline orchestration.
• Familiarity with SQL and relational/non-relational databases.
• Excellent analytical, debugging, and communication skills.

Responsibilities

• Design, develop, and maintain data pipelines and ETL workflows using Python, PySpark, and AWS services.
• Build and optimize large-scale data processing and data transformation solutions.
• Integrate various data sources and ensure data quality, performance, and reliability.
• Collaborate with data engineers, analysts, and architects to deliver end-to-end data solutions.
• Implement best practices for code optimization, error handling, and data validation.
• Participate in code reviews, documentation, and deployment automation.
• Ensure adherence to data security and compliance standards.

Skills

Python

PySpark

AWS

ETL

SQL

Distributed processing

Communication

Debugging

Education

Bachelor's degree in Computer Science or related field

Tools

Airflow

Databricks

Git

Docker

Kubernetes

Job description

Job Summary

We are seeking an experienced Python developer with strong expertise in AWS and PySpark to join our data engineering team. The ideal candidate will have hands-on experience developing scalable data pipelines, processing large data sets, and integrating with cloud-based environments. This role requires excellent problem-solving skills and a strong understanding of distributed data processing frameworks.

Responsibilities

• Design, develop, and maintain data pipelines and ETL workflows using Python, PySpark, and AWS services.
• Build and optimize large-scale data processing and data transformation solutions.
• Integrate various data sources and ensure data quality, performance, and reliability.
• Collaborate with data engineers, analysts, and architects to deliver end-to-end data solutions.
• Implement best practices for code optimization, error handling, and data validation.
• Participate in code reviews, documentation, and deployment automation.
• Ensure adherence to data security and compliance standards.

Required Skills & Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, or a related field.
• 10+ years of experience in software development with a strong focus on Python.
• Hands-on experience with PySpark for distributed data processing.
• Solid understanding of AWS cloud services such as S3, Glue, Lambda, EMR, Redshift, and Athena.
• Strong experience in ETL development and data pipeline orchestration.
• Familiarity with SQL and relational/non-relational databases.
• Excellent analytical, debugging, and communication skills.

Preferred Skills

• Experience with Airflow, Databricks, or other workflow management tools.
• Knowledge of CI/CD pipelines and version control tools like Git.
• Exposure to data lake or data warehouse architectures.
• Familiarity with Docker or Kubernetes for deployment.