Senior Data Engineer

Veriipro; Center Square (PA); On-site; USD 120,000 - 160,000; Full time

Job summary

Veriipro is seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines and ingestion processes for data lake environments. You will work with Spark, Scala, Hadoop, Hive, and SQL to ensure data quality and timely delivery.

The role requires 8–10 years in data engineering with strong SQL/ETL, Linux scripting, and hands-on Spark/Scala. You will mentor juniors and collaborate in an Agile/SAFe setting to meet project timelines.

Qualifications

• 8–10 years of industry experience with a related bachelor’s degree.
• 5+ years in data aggregation, standardization, linking, quality checks, and reporting.
• 5+ years with Hadoop and Spark.
• 5+ years with Oracle or SQL Server; strong SQL/ETL skills.
• Proficient in Linux environments, shell scripting, and data integration.
• Hands-on experience with Spark and Scala, plus Agile development.

Responsibilities

• Design, develop, and maintain complex data pipelines for data lake environments.
• Develop data transformation processes to aggregate, standardize, link, validate, and load data.
• Build data engineering solutions using Spark, Scala, Hadoop, Hive, SQL/T-SQL, and Shell scripting.
• Perform data validation, quality checks, analysis, and troubleshooting for accuracy.
• Analyze data issues, root-cause analysis, and implement improvements to quality and performance.
• Develop tools and processes to improve data processing efficiency, reliability, and performance.
• Collaborate with technical and operations teams to troubleshoot issues involving databases, Linux, storage, and servers.
• Support production environments and resolve critical data and pipeline issues as needed.
• Generate statistical and operational reports and perform program/data validation.
• Mentor junior data engineers on data engineering, troubleshooting, and quality best practices.
• Collaborate with cross-functional teams in an Agile/SAFe environment to deliver projects on time.

Skills

Spark

Scala

Hadoop

SQL & ETL

Linux scripting

Data integration

Data quality

Troubleshooting

Agile/SAFe

Data pipelines

Big Data technologies

Oracle/SQL Server

Education

Bachelor’s degree in Computer Science or related field

Tools

Hadoop

Hive

Oracle

SQL Server

Shell scripting

Job description

We are seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines and data ingestion processes using modern Big Data technologies. The ideal candidate will have strong experience in data aggregation, transformation, validation, quality management, ETL, and troubleshooting across large-scale data environments.

Roles and Responsibilities

• Design, develop, and maintain complex data pipelines and data ingestion processes for data lake environments.

• Develop data transformation processes to aggregate, standardize, link, validate, and load data from multiple sources.

• Build and maintain data engineering solutions using Spark, Scala, Hadoop, Hive, SQL/T-SQL, and Shell scripting.

• Perform data validation, quality checks, analysis, and troubleshooting to ensure data accuracy and integrity.

• Analyze data issues, perform root-cause analysis, and implement solutions to improve data quality and pipeline performance.

• Develop tools and processes to improve data processing efficiency, reliability, and performance.

• Work with technical and operations teams to troubleshoot complex issues involving databases, Linux environments, storage, and servers.

• Support production environments and resolve critical data and pipeline issues as needed.

• Generate statistical and operational reports and perform program/data validation.

• Mentor junior data engineers on data engineering, troubleshooting, and data quality best practices.

• Collaborate with cross-functional teams in an Agile/SAFe environment to deliver projects within defined timelines.

Required Qualifications

• 8–10 years of overall industry experience with a Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field.

• 5+ years of experience in data aggregation, standardization, data linking, quality checks, and reporting.

• 5+ years of experience with Big Data technologies such as Hadoop and Spark.

• 5+ years of experience with relational databases such as Oracle and Microsoft SQL Server.

• Strong experience with SQL, advanced SQL queries, ETL processes, and data integration tools.

• Strong understanding of Linux environments, shell scripting, and file systems.

• Hands-on experience with Spark and Scala.

• Strong data analysis, debugging, troubleshooting, and root-cause analysis skills.

• Experience working in an Agile development environment.

Preferred Qualifications

• Experience working with healthcare data or in a healthcare data operations environment.

• Experience with Hive and additional Big Data technologies.

• Experience with data lake and large-scale data ingestion platforms.

• Experience working in Scaled Agile Framework (SAFe) environments.

• Availability to support production issues outside standard business hours when required.

• EST time-zone availability is preferred.