Senior Data Engineer – Spark / Scala

DataJobs; Richmond (VA); Hybrid; USD 91,000 - 98,000; Full time

Benefits offered by this job

401(k)

Dental insurance

Health insurance

Vision insurance

Job summary

Astyra Corporation is seeking a Senior Data Engineer to modernize legacy ETL workloads and deliver Spark-based data pipelines. This contract role is based in Richmond, VA with hybrid remote work, focusing on accurate data movement and validation between legacy and target platforms.

You will design, build, and optimize enterprise data pipelines using Spark and Scala, migrate legacy ETL to modern architectures, and implement data quality checks across systems.

Qualifications

• 7+ years of data engineering experience.
• Experience designing and developing enterprise data pipelines.
• Advanced Spark experience.
• Strong data transformation, data quality and reconciliation knowledge.
• Experience migrating legacy ETL to Spark-based architectures.
• Experience supporting production data-processing workloads.
• Familiarity with modern CI/CD and software development practices.

Responsibilities

• Design, build, and maintain scalable data pipelines using Apache Spark and Scala.
• Migrate legacy ETL workloads to Spark-based framework.
• Develop and enhance data transformation and processing solutions.
• Perform data comparison, reconciliation, and validation between legacy and target platforms.
• Identify and troubleshoot data discrepancies throughout the migration process.
• Create data quality checks and controls to ensure completeness and accuracy.
• Optimize Spark jobs and pipelines for performance, scalability, and reliability.
• Collaborate with data engineers, architects, and other technical teams on modernization initiatives.
• Support testing, deployment, troubleshooting, and ongoing maintenance of production data pipelines.
• Contribute to broader feature engineering and data modernization efforts.

Skills

7+ years experience

Data engineering

Troubleshooting

CI/CD familiarity

Data quality & reconciliation

Data transformation

Production data workloads

Legacy modernization

Tools

Apache Spark

Scala

Python

CI/CD tooling

Job description

Astyra Corporation is seeking a Senior Data Engineer to help modernize legacy ETL workloads and deliver Spark-based data pipelines. This contract role is based in Richmond, VA with hybrid remote work, with a focus on accurate data movement, Spark job optimization, and validation between legacy and target platforms.

What You’ll Do

• Design, build, and maintain scalable data pipelines using Apache Spark and Scala
• Support the migration of legacy ETL workloads into a modern Spark-based data framework
• Develop and enhance data transformation and processing solutions
• Perform data comparison, reconciliation, and validation between legacy and target platforms
• Identify and troubleshoot data discrepancies throughout the migration process
• Create data quality checks and controls to ensure completeness and accuracy
• Optimize Spark jobs and pipelines for performance, scalability, and reliability
• Collaborate with data engineers, architects, and other technical teams on modernization initiatives
• Support testing, deployment, troubleshooting, and ongoing maintenance of production data pipelines
• Contribute to broader feature engineering and data modernization efforts

What You Bring

• 7+ years of experience in data engineering or a related technical discipline
• Strong hands-on experience designing and developing enterprise data pipelines
• Advanced experience with Apache Spark
• Strong development experience with Scala
• Experience with ETL development, modernization, and migration initiatives
• Strong understanding of data transformation, data quality, and reconciliation
• Experience comparing and validating data across legacy and modernized systems
• Experience developing and supporting production data-processing workloads
• Familiarity with modern CI/CD and software development practices
• Strong troubleshooting and problem-solving skills

Technologies

• Apache Spark
• Spark
• Scala
• Python
• ETL
• CI/CD

Preferred Qualifications

• Experience migrating legacy ETL platforms to Spark-based architectures
• Experience supporting large-scale technology or data modernization initiatives
• Experience within financial services or other highly regulated environments
• Python experience
• Exposure to Generative AI or AI-assisted development and modernization tools
• Previous experience within a large enterprise engineering environment

Compensation and Schedule

• Pay: $66.00 - $71.00 per hour
• Expected hours: 40.0 per week

Benefits

• 401(k)
• Dental insurance
• Health insurance
• Vision insurance

Location: Richmond, VA 23238 (Required). Work location: Hybrid remote in Richmond, VA 23238.