Remote Spark Big Data Engineer for AI Training

ixolabs.ai; France; À distance; EUR 82 656 - 165 312; Temps partiel

Résumé du poste

ixolabs.ai is seeking a Big Data Engineer (Spark) to help design and optimize Spark-based AI training pipelines. You will write and review Spark code across Scala, PySpark, and Spark SQL, focusing on scalable data processing, partitioning, and efficient joins.

Ideal candidates have extensive Spark production experience, strong knowledge of Spark internals, and hands-on with modern table formats and cloud platforms. Flexible, remote, contractor engagement with 10–25 hours per week is offered.

Qualifications

• 6 years in big data engineering, including 4 years writing production Spark.
• Strong familiarity with both Scala Spark and PySpark.
• Deep knowledge of Spark internals (Catalyst, Tungsten, AQE).
• Experience with at least one modern table format (Delta, Iceberg, Hudi).
• Comfort with cloud data platforms (Databricks, EMR, Dataproc, Synapse).
• Familiarity with Kafka, Airflow, or dbt is a plus.

Responsabilités

• Generate and evaluate Spark instruction-response pairs covering DataFrame, SQL, and Structured Streaming APIs.
• Review AI-generated code in Scala Spark, PySpark, and Spark SQL.
• Provide feedback on partitioning strategies, broadcast joins, and skew handling.
• Validate AI handling of Delta Lake, Iceberg, and Hudi table formats.
• Evaluate cluster sizing, dynamic allocation, and Spark-on-Kubernetes patterns.
• Identify subtle issues in shuffle behavior, serialization, and AQE-related regressions.

Connaissances

Big data engineering

Spark production experience

Scala Spark

PySpark

Cloud data platforms

Outils

Spark (Core)

Delta Lake

Iceberg

Hudi

Databricks

EMR

Dataproc

Synapse

Kafka

Airflow

dbt

Spark on Kubernetes

Description du poste

ixolabs.ai is seeking a Big Data Engineer (Spark) to help design and optimize Spark-based AI training pipelines. You will write and review Spark code across Scala, PySpark, and Spark SQL, focusing on scalable data processing, partitioning, and efficient joins.

Ideal candidates have extensive Spark production experience, strong knowledge of Spark internals, and hands-on with modern table formats and cloud platforms. Flexible, remote, contractor engagement with 10–25 hours per week is offered.