Senior Apache Spark Developer – Remote, Enterprise Data

Bright Vision Technologies; United States; Remote; USD 125,000 - 185,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Apache Spark Developer for a 100% remote role in the United States. The role focuses on designing, developing, and optimizing large-scale distributed data processing applications supporting analytics, ML, real-time reporting, and cloud-based data platforms.

The ideal candidate has 6+ years of experience with PySpark/Scala, Spark SQL, and Spark architecture, plus strong SQL and Hadoop ecosystem knowledge.

Qualifications

• Six+ years of professional software or data engineering experience.

Responsibilities

• Design, develop, and maintain high-performance distributed data processing applications using Apache Spark.
• Build scalable batch and real-time ETL/ELT pipelines processing large volumes of enterprise data.
• Develop Spark applications using PySpark, Scala, or Spark SQL for data transformation and analytics.
• Optimize Spark jobs for memory usage, partitioning, shuffle performance, and execution efficiency.
• Collaborate with cloud engineering teams to deploy Spark workloads on Databricks, EMR, Azure Synapse, or Kubernetes.
• Implement data quality validation, monitoring, and error handling across pipelines.
• Participate in architecture and code reviews and Agile development activities.
• Troubleshoot production issues involving distributed processing and data quality.

Skills

PySpark

Scala

Spark SQL

Spark architecture

Distributed computing

SQL proficiency

Hadoop ecosystem

Streaming

Cloud platforms

Delta Lake

Data warehousing

Git / CI/CD

Agile Scrum

Tools

Hive

HDFS

YARN

Parquet

Delta Lake

Iceberg

Hudi

Kafka

Databricks

EMR

Azure Synapse

Kubernetes

Docker

Git

CI/CD

Job description

Bright Vision Technologies is seeking an experienced Apache Spark Developer for a 100% remote role in the United States. The role focuses on designing, developing, and optimizing large-scale distributed data processing applications supporting analytics, ML, real-time reporting, and cloud-based data platforms.

The ideal candidate has 6+ years of experience with PySpark/Scala, Spark SQL, and Spark architecture, plus strong SQL and Hadoop ecosystem knowledge.