Data Engineer

Quadrant IQ Solutions LLC; New York (NY); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading data solutions company in New York is seeking a Mid-Senior level Data Engineer. The role involves designing and optimizing high-performance Spark-based data pipelines using Python/PySpark. Candidates should have strong experience with AWS services and data warehousing tools such as Snowflake or Teradata. This is a full-time position that promises great opportunities for growth in a dynamic environment.

Qualifications

• Expert-level proficiency with Python & PySpark, with strong experience in Apache Spark.
• Hands-on experience with data warehousing tools like Snowflake or Teradata.
• Experience deploying data workloads on AWS ECS and using AWS Glue for data governance.

Responsibilities

• Design and optimize high-performance Spark-based data pipelines using Python/PySpark.
• Develop ETL/ELT workflows for large-scale structured and semi-structured data.
• Collaborate with analytics and engineering teams for data integration.

Skills

Python & PySpark

AWS Cloud

ECS (Elastic Container Service)

AWS Glue

Data Warehousing

Snowflake

Teradata

Job description

Get AI-powered advice on this job and more exclusive features.

Job Type: W2

Visa's Accepted: H4-EAD, GC-EAD, L2S, GC, USC, H1B

Must-Have Skills

• Python & PySpark – Expert-level proficiency with very strong Apache Spark experience (large-scale data processing, optimization, performance tuning).
• AWS Cloud
• ECS (Elastic Container Service) – Required
• AWS Glue and/or AWS Lake Formation – Strongly preferred
• Data Warehousing
• Snowflake or Teradata (hands-on experience with large analytical datasets)

Core Responsibilities

• Design, build, and optimize high-performance Spark-based data pipelines using Python/PySpark.
• Develop and maintain ETL/ELT workflows for large-scale structured and semi-structured data.
• Deploy and manage data workloads on AWS ECS.
• Implement data ingestion, transformation, and governance using AWS Glue / Lake Formation.
• Integrate data pipelines with Snowflake or Teradata for analytics and reporting.
• Ensure scalability, reliability, and performance of distributed data processing systems.
• Collaborate with analytics, platform, and cloud engineering teams.

Seniority level

• Mid-Senior level

Employment type

• Full-time

Job function

• Data Engineering

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.