Remote Apache Spark Engineer - Big Data & Cloud Analytics

Bright Vision Technologies; Oregon; Hybrid; USD 125.000 - 185.000; Vollzeit

Zusammenfassung

Bright Vision Technologies is seeking an experienced Apache Spark Developer to design, develop, and optimize large-scale distributed data processing applications supporting analytics, machine learning, real-time reporting, and cloud platforms.

This role focuses on building high-performance Spark applications capable of handling billions of records across structured and semi-structured data while delivering scalable, reliable data pipelines.

Qualifikationen

• 6+ years in software/data engineering.
• 4+ years Apache Spark development in production.
• Strong PySpark/Scala/Spark SQL skills.
• Knowledge of Spark architecture and distributed systems.

Aufgaben

• Design, develop and maintain high-performance distributed data processing apps with Spark.
• Build scalable batch and real-time ETL/ELT pipelines for large data.
• Develop Spark apps using PySpark, Scala, or Spark SQL for data transformation and analytics.
• Optimize Spark jobs for memory, partitioning, shuffle, and efficiency.
• Integrate with data warehouses, lakehouses, and reporting platforms.
• Collaborate with cloud teams to deploy Spark on Databricks, EMR, Azure Synapse, or Kubernetes.
• Implement data quality validation, monitoring, and automated error handling.
• Participate in architecture reviews, code reviews, and Agile development.

Kenntnisse

6+ years experience in software/data

Apache Spark development

PySpark/Scala/Spark SQL

Spark architecture knowledge

Distributed computing concepts

Advanced SQL skills

Hadoop ecosystem knowledge

Streaming data processing

Cloud platforms knowledge

Git/CI-CD tools

Agile Scrum

Tools

Databricks

Delta Lake

Apache Iceberg

Apache Hudi

Kubernetes

Docker

Airflow

Terraform

Prometheus

Grafana

OpenTelemetry

Jobbeschreibung

Bright Vision Technologies is seeking an experienced Apache Spark Developer to design, develop, and optimize large-scale distributed data processing applications supporting analytics, machine learning, real-time reporting, and cloud platforms.

This role focuses on building high-performance Spark applications capable of handling billions of records across structured and semi-structured data while delivering scalable, reliable data pipelines.