Spark Migration Architect for Big Data Pipelines

HMG AMERICA LLC; San Francisco (CA); On-site; USD 120,000 - 150,000; Full time

Job summary

A tech data solutions provider is seeking a Spark job migration specialist in San Francisco, California. This role focuses on migrating data pipelines, JAR tasks, and analytics workloads to modern platforms, requiring over 5 years of experience with Apache Spark and cloud services like Azure or AWS. Responsibilities include refactoring code, performance optimization, and regression testing. Ideal candidates have strong skills in HDFS and the Hadoop ecosystem, as well as expertise in SQL and scripting.

Qualifications

• 5+ years experience with Apache Spark (PySpark/Scala) and Cloud platforms.
• Strong experience with HDFS and Hadoop ecosystem.

Responsibilities

• Migrate JVM workloads and Spark-Submit tasks to Databricks.
• Convert HiveQL scripts and Oozie workflows into optimized Spark applications.
• Implement Adaptive Query Execution in Spark 3 to improve performance.
• Perform regression testing to validate output consistency.

Skills

Apache Spark (PySpark/Scala)

HDFS

Cloud platforms (Azure/AWS)

Data migration

SQL and performance tuning

Scripting (Python, Shell, Scala)

Job description

A tech data solutions provider is seeking a Spark job migration specialist in San Francisco, California. This role focuses on migrating data pipelines, JAR tasks, and analytics workloads to modern platforms, requiring over 5 years of experience with Apache Spark and cloud services like Azure or AWS. Responsibilities include refactoring code, performance optimization, and regression testing. Ideal candidates have strong skills in HDFS and the Hadoop ecosystem, as well as expertise in SQL and scripting.