Senior Data Engineer - Spark/Scala, Modern Data Pipelines

Astyra Corporation; Richmond (VA); On-site; USD 150,000 - 190,000; Full time

Job summary

Astyra’s client is seeking a hands-on Senior Data Engineer to modernize large-scale data workloads using Spark and Scala. You will design, build, and optimize pipelines, migrate legacy ETL processes, and ensure data quality across platforms.

This role focuses on performance, scalability, and reliable data processing in a modern data framework. Ideal candidates have 7+ years in data engineering, deep Spark experience, and strong Scala skills, with familiarity in data transformation,

Qualifications

• 7+ years of experience in data engineering or a related technical discipline.
• Strong hands-on experience designing and developing enterprise data pipelines.
• Experience with ETL development, modernization, and migration initiatives.

Responsibilities

• Design, build, and maintain scalable data pipelines using Spark and Scala.
• Support the migration of legacy ETL workloads into a modern Spark-based data framework.
• Develop and enhance data transformation and processing solutions.
• Perform data comparison, reconciliation, and validation between legacy and target platforms.
• Identify and troubleshoot data discrepancies throughout the migration process.
• Develop data quality checks and controls to ensure completeness and accuracy.
• Optimize Spark jobs and pipelines for performance, scalability, and reliability.
• Work closely with data engineers, architects, and other technical teams throughout the modernization effort.
• Support testing, deployment, troubleshooting, and ongoing maintenance of production data pipelines.
• Contribute to broader feature engineering and data modernization initiatives.

Skills

Apache Spark

Scala

Data pipeline design

ETL development

Data quality & reconciliation

CI/CD practices

Troubleshooting

Production workloads

Team collaboration

Job description

Astyra’s client is seeking a hands-on Senior Data Engineer to modernize large-scale data workloads using Spark and Scala. You will design, build, and optimize pipelines, migrate legacy ETL processes, and ensure data quality across platforms.

This role focuses on performance, scalability, and reliable data processing in a modern data framework. Ideal candidates have 7+ years in data engineering, deep Spark experience, and strong Scala skills, with familiarity in data transformation,