Senior Data Engineer: Scale Data Pipelines & Quality

Veriipro; Center Square (PA); On-site; USD 120,000 - 160,000; Full time

Job summary

Veriipro is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines and ingestion processes for data lake environments. You will work with Spark, Scala, Hadoop, Hive, and SQL to ensure data quality and timely delivery.

The role requires 8–10 years in data engineering with strong SQL/ETL, Linux scripting, and hands-on Spark/Scala. You will mentor juniors and collaborate in an Agile/SAFe setting to meet project timelines.

Qualifications

• 8–10 years of industry experience with a related bachelor’s degree.
• 5+ years in data aggregation, standardization, linking, quality checks, and reporting.
• 5+ years with Hadoop and Spark.
• 5+ years with Oracle or SQL Server; strong SQL/ETL skills.
• Proficient in Linux environments, shell scripting, and data integration.
• Hands-on experience with Spark and Scala, plus Agile development.

Responsibilities

• Design, develop, and maintain complex data pipelines for data lake environments.
• Develop data transformation processes to aggregate, standardize, link, validate, and load data.
• Build data engineering solutions using Spark, Scala, Hadoop, Hive, SQL/T-SQL, and Shell scripting.
• Perform data validation, quality checks, analysis, and troubleshooting for accuracy.
• Analyze data issues, root-cause analysis, and implement improvements to quality and performance.
• Develop tools and processes to improve data processing efficiency, reliability, and performance.
• Collaborate with technical and operations teams to troubleshoot issues involving databases, Linux, storage, and servers.
• Support production environments and resolve critical data and pipeline issues as needed.
• Generate statistical and operational reports and perform program/data validation.
• Mentor junior data engineers on data engineering, troubleshooting, and quality best practices.
• Collaborate with cross-functional teams in an Agile/SAFe environment to deliver projects on time.

Skills

Spark

Scala

Hadoop

SQL & ETL

Linux scripting

Data integration

Data quality

Troubleshooting

Agile/SAFe

Data pipelines

Big Data technologies

Oracle/SQL Server

Education

Bachelor’s degree in Computer Science or related field

Tools

Hadoop

Hive

Oracle

SQL Server

Shell scripting

Job description

Veriipro is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines and ingestion processes for data lake environments. You will work with Spark, Scala, Hadoop, Hive, and SQL to ensure data quality and timely delivery.

The role requires 8–10 years in data engineering with strong SQL/ETL, Linux scripting, and hands-on Spark/Scala. You will mentor juniors and collaborate in an Agile/SAFe setting to meet project timelines.