Senior Data Engineer – Spark / Scala

Astyra Corporation; Richmond (VA); On-site; USD 150,000 - 190,000; Full time

Job summary

Astyra’s client is seeking a hands-on Senior Data Engineer to modernize large-scale data workloads using Spark and Scala. You will design, build, and optimize pipelines, migrate legacy ETL processes, and ensure data quality across platforms.

This role focuses on performance, scalability, and reliable data processing in a modern data framework. Ideal candidates have 7+ years in data engineering, deep Spark experience, and strong Scala skills, with familiarity in data transformation,

Qualifications

• 7+ years of experience in data engineering or a related technical discipline.
• Strong hands-on experience designing and developing enterprise data pipelines.
• Experience with ETL development, modernization, and migration initiatives.

Responsibilities

• Design, build, and maintain scalable data pipelines using Spark and Scala.
• Support the migration of legacy ETL workloads into a modern Spark-based data framework.
• Develop and enhance data transformation and processing solutions.
• Perform data comparison, reconciliation, and validation between legacy and target platforms.
• Identify and troubleshoot data discrepancies throughout the migration process.
• Develop data quality checks and controls to ensure completeness and accuracy.
• Optimize Spark jobs and pipelines for performance, scalability, and reliability.
• Work closely with data engineers, architects, and other technical teams throughout the modernization effort.
• Support testing, deployment, troubleshooting, and ongoing maintenance of production data pipelines.
• Contribute to broader feature engineering and data modernization initiatives.

Skills

Apache Spark

Scala

Data pipeline design

ETL development

Data quality & reconciliation

CI/CD practices

Troubleshooting

Production workloads

Team collaboration

Job description

Astyra’s client is seeking a hands-on Senior Data Engineer to support a large-scale data and technology modernization initiative. This position will focus on modernizing legacy ETL workloads, developing Spark-based data pipelines, and ensuring data accuracy and consistency as workloads transition to modern platforms.
The ideal candidate has deep hands-on experience with Spark and Scala and a strong background designing, building, optimizing, and validating enterprise data pipelines.

What You’ll Do

• Design, build, and maintain scalable data pipelines using Spark and Scala
• Support the migration of legacy ETL workloads into a modern Spark-based data framework
• Develop and enhance data transformation and processing solutions
• Perform data comparison, reconciliation, and validation between legacy and target platforms
• Identify and troubleshoot data discrepancies throughout the migration process
• Develop data quality checks and controls to ensure completeness and accuracy
• Optimize Spark jobs and pipelines for performance, scalability, and reliability
• Work closely with data engineers, architects, and other technical teams throughout the modernization effort
• Support testing, deployment, troubleshooting, and ongoing maintenance of production data pipelines
• Contribute to broader feature engineering and data modernization initiatives

What We’re Looking For

• 7+ years of experience in data engineering or a related technical discipline
• Strong hands-on experience designing and developing enterprise data pipelines
• Advanced experience with Apache Spark
• Strong development experience with Scala
• Experience with ETL development, modernization, and migration initiatives
• Strong understanding of data transformation, data quality, and reconciliation
• Experience comparing and validating data across legacy and modernized systems
• Experience developing and supporting production data-processing workloads
• Familiarity with modern CI/CD and software development practices
• Strong troubleshooting and problem-solving skills

Preferred Qualifications

• Experience migrating legacy ETL platforms to Spark-based architectures
• Experience supporting large-scale technology or data modernization initiatives
• Experience within financial services or other highly regulated environments
• Python experience
• Exposure to Generative AI or AI-assisted development and modernization tools
• Previous experience within a large enterprise engineering environment