Cloud-Native Spark Migration Architect (Serverless)

Virtasant; Austin (TX); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Virtasant is seeking a Senior Platform Engineer to lead migrations of Apache Spark workloads from on-prem Hadoop/Spark environments to cloud-native and serverless architectures across AWS and GCP. This hands-on role covers assessment, architecture, refactoring, migration execution, performance optimization, automation, and production readiness.

You will determine target architectures per workload and establish repeatable patterns for enterprise-scale migrations, balancing legacy systems with

Qualifications

• 8+ years in data engineering, distributed systems, cloud engineering, or platform engineering.
• 5+ years of hands-on Apache Spark experience in enterprise environments.
• Strong PySpark and/or Scala development experience.
• Proven experience migrating large-scale Spark workloads between infrastructure platforms.
• Hands-on experience with both AWS and GCP.
• Experience with on-premise Hadoop/Spark ecosystems, including HDFS, YARN and Hive.

Responsibilities

• Lead migrations of enterprise Spark workloads from on-premise environments to AWS and GCP.
• Assess Spark applications, clusters, configurations, dependencies, data flows, and resource utilization.
• Determine migration approaches: rehost, replatform, refactor, modernize, or retire.
• Modernize traditional cluster workloads for serverless Spark where appropriate.
• Design architectures using AWS EMR Serverless, S3, Glue, Lake Formation, GCP Dataproc Serverless, GCS, and BigQuery.
• Refactor PySpark/Scala/Spark SQL applications for cloud portability and reliability.
• Migrate workloads from Hadoop, HDFS, YARN, Hive, and on-prem Spark clusters.
• Troubleshoot and optimize Spark workloads: partitioning, joins, data skew, execution plans, and SQL execution.
• Benchmark performance and optimize serverless workloads for performance and cost.
• Build reusable migration tooling, automation, and frameworks.
• Implement CI/CD and Infrastructure as Code using Terraform and related tools.
• Define testing, validation, cutover, rollback, observability, and production-readiness patterns.
• Partner with Data Engineering, ML, Cloud Architecture, Platform Engineering, DevOps/SRE, Security, Governance, and FinOps teams.

Skills

PySpark

Scala

SQL

Cloud architecture

Tools

AWS

GCP

Hadoop

YARN

Hive

Terraform

Spark

Job description

Virtasant is seeking a Senior Platform Engineer to lead migrations of Apache Spark workloads from on-prem Hadoop/Spark environments to cloud-native and serverless architectures across AWS and GCP. This hands-on role covers assessment, architecture, refactoring, migration execution, performance optimization, automation, and production readiness.

You will determine target architectures per workload and establish repeatable patterns for enterprise-scale migrations, balancing legacy systems with