Senior Apache Spark Developer - Remote & Scalable Pipelines

Bright Vision Technologies; Chesterfield (MO); Remote; USD 125,000 - 185,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Apache Spark Developer for a 100% remote role in the United States. The candidate will design, develop, and optimize large-scale distributed data processing applications supporting enterprise analytics, ML, and real-time reporting across cloud-based platforms.

The role requires 6+ years in software or data engineering, 4+ years with Spark, and strong PySpark/Scala/Spark SQL skills.

Qualifications

• Six or more years of professional software or data engineering experience.
• Four or more years of hands-on Apache Spark development experience in enterprise production environments.
• Strong proficiency in PySpark, Scala, or Spark SQL for distributed data processing.
• Deep understanding of Apache Spark architecture including RDDs, DataFrames, Datasets, Catalyst Optimizer, DAG execution, and Tungsten engine.
• Strong experience with distributed computing concepts including partitioning, shuffling, caching, broadcast joins, and fault tolerance.
• Advanced SQL skills with databases such as SQL Server, Oracle, PostgreSQL, Snowflake, or Teradata.
• Experience working with Hadoop ecosystem technologies including Hive, HDFS, YARN, and Parquet.
• Experience processing streaming data using Spark Structured Streaming, Apache Kafka, or Event Hubs.
• Hands-on experience with cloud platforms including Azure Databricks, AWS EMR, AWS Glue, Azure Synapse Analytics, or Google Dataproc.
• Experience integrating Spark applications with Delta Lake, Apache Iceberg, or Apache Hudi.
• Strong understanding of data warehousing concepts, dimensional modeling, and data lake architecture.
• Experience using Git, CI/CD pipelines, Azure DevOps, GitHub Actions, or Jenkins.
• Strong debugging, troubleshooting, and Spark performance tuning skills.
• Experience working in Agile Scrum development environments.

Responsibilities

• Design, develop, and maintain high-performance distributed data processing applications using Apache Spark.
• Build scalable batch and real-time ETL/ELT pipelines processing large volumes of enterprise data.
• Develop Spark applications using PySpark, Scala, or Spark SQL for data transformation, aggregation, and analytics.
• Optimize Spark jobs for memory utilization, partitioning strategies, shuffle performance, and execution efficiency.
• Process structured, semi-structured, and streaming data from enterprise databases, APIs, Kafka, cloud storage, and data lakes.
• Develop reusable Spark libraries, data processing frameworks, and metadata-driven ingestion pipelines.
• Collaborate with cloud engineering teams to deploy Spark workloads on Databricks, EMR, Azure Synapse, or Kubernetes.
• Implement data quality validation, reconciliation, monitoring, and automated error handling across distributed pipelines.
• Integrate Spark applications with enterprise data warehouses, lakehouses, and reporting platforms.
• Participate in architecture reviews, code reviews, technical design discussions, and Agile development activities.
• Troubleshoot production issues involving distributed processing, cluster performance, resource utilization, and data quality.
• Support cloud migration initiatives by modernizing legacy ETL workloads into Spark-based architectures.

Skills

PySpark

Scala

Spark SQL

Spark Architecture

Distributed computing

SQL

Hadoop Ecosystem

Structured Streaming

Cloud platforms

Delta Lake

Data Warehousing

Git

CI/CD

Agile Scrum

Tools

Databricks

AWS EMR

Azure Synapse

Google Dataproc

Kafka

Job description

Bright Vision Technologies is seeking an experienced Apache Spark Developer for a 100% remote role in the United States. The candidate will design, develop, and optimize large-scale distributed data processing applications supporting enterprise analytics, ML, and real-time reporting across cloud-based platforms.

The role requires 6+ years in software or data engineering, 4+ years with Spark, and strong PySpark/Scala/Spark SQL skills.