Senior Big Data Engineer

Veriipro; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

Veriipro in New York City is seeking a Senior Big Data Engineer with 8+ years of IT experience and hands-on expertise in Hadoop, Spark, Python, and Snowflake. You will design scalable data pipelines, migrate data from Hadoop to Snowflake using PySpark, and develop enterprise ETL solutions.

The role requires building and optimizing ETL workflows, writing complex SQL, ensuring data quality, and collaborating with cross-functional teams to deliver robust data platforms for analytics.

Qualifications

• 8+ years of IT experience in data engineering or big data development.
• Hands-on experience with Hadoop, Spark, Python, and Snowflake.
• Experience migrating enterprise data from Hadoop to Snowflake using PySpark.
• Strong ETL, data integration, and data transformation knowledge.
• Strong SQL skills and ability to optimize complex queries.
• Excellent analytical and communication skills.
• Unix/Linux experience preferred.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Hadoop, Spark, Python, and Snowflake.
• Migrate data from Hadoop to Snowflake using PySpark frameworks and libraries.
• Build and optimize ETL workflows for large-scale data ingestion, transformation, and validation.
• Develop and optimize complex SQL queries for data processing and analytics.
• Ensure data quality, performance, and reliability across enterprise data platforms.
• Collaborate with cross-functional teams to gather requirements, troubleshoot issues, and deliver data solutions.

Skills

Hadoop

Apache Spark

Python

Snowflake

SQL

ETL

Unix/Linux

Data Migration

Tools

PySpark

Job description

We are seeking a Senior Big Data Engineer with 8+ years of IT experience and strong expertise in Hadoop, Spark, Python, and Snowflake. The ideal candidate will have hands‑on experience building scalable data pipelines, migrating data from Hadoop to Snowflake using PySpark, and developing enterprise ETL solutions.

Roles and Responsibilities

• Design, develop, and maintain scalable data pipelines using Hadoop, Spark, Python, and Snowflake.
• Perform data migration from Hadoop to Snowflake using PySpark frameworks and libraries.
• Build and optimize ETL workflows for large-scale data ingestion, transformation, and validation.
• Develop and optimize complex SQL queries for data processing and analytics.
• Ensure data quality, performance, and reliability across enterprise data platforms.
• Collaborate with cross‑functional teams to gather requirements, troubleshoot issues, and deliver data solutions.

Required Qualifications

• 8+ years of overall IT experience in data engineering or big data development.
• Strong hands‑on experience with Hadoop, Apache Spark, Python, and Snowflake.
• Experience migrating enterprise data from Hadoop to Snowflake using PySpark.
• Solid understanding of ETL processes, data integration, and data transformation techniques.
• Strong SQL skills with experience writing and optimizing complex queries.
• Excellent analytical, problem‑solving, and communication skills.
• Experience working in Unix/Linux environments. (Preferred)