Senior PySpark & Java Data Pipeline Engineer

Veriipro; Whitpain Township (PA); On-site; USD 90,000 - 120,000; Full time

Job summary

Veriipro in Whitpain Township, Pennsylvania, is looking for an experienced data engineer to design and maintain ETL pipelines for structured and unstructured data. You will work with technologies like PySpark, SQL Server, and Hadoop, ensuring data pipeline reliability and performance.

The ideal candidate should have over 5 years of experience in database development and a strong understanding of data processing applications. Join us to leverage your expertise in a dynamic environment.

Qualifications

• 5+ years of experience in Microsoft SQL Server and relational database development.
• Strong understanding of ETL concepts and large-scale data processing.
• Proven experience in performance tuning of SQL queries.

Responsibilities

• Design, develop, and maintain scalable ETL pipelines.
• Build and optimize data processing applications using PySpark and Java.
• Analyze and resolve performance bottlenecks in SQL procedures.

Skills

Microsoft SQL Server

ETL concepts

SQL performance tuning

Hadoop

PySpark

Hive

Impala

Python

Tools

Cloudera Hadoop Ecosystem

Kafka

Hue

Oozie

YARN

Sqoop

Job description

Veriipro in Whitpain Township, Pennsylvania, is looking for an experienced data engineer to design and maintain ETL pipelines for structured and unstructured data. You will work with technologies like PySpark, SQL Server, and Hadoop, ensuring data pipeline reliability and performance.

The ideal candidate should have over 5 years of experience in database development and a strong understanding of data processing applications. Join us to leverage your expertise in a dynamic environment.