Senior Apache Spark Developer - Remote, Big Data & AI

Bright Vision Technologies; Overland Park (KS); Remote; USD 125,000 - 185,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Apache Spark Developer for a 100% remote role in the United States. The position focuses on designing, building, and optimizing large-scale distributed data processing applications that support analytics, ML, real-time reporting, and cloud-based platforms.

You will collaborate with data architects, data engineers, cloud platform teams, ML engineers, and BI developers to deliver scalable Spark-based data pipelines.

Qualifications

• 6+ years of professional software or data engineering experience.
• 4+ years hands-on Apache Spark development in enterprise production environments.
• Experience with PySpark, Scala, or Spark SQL for distributed data processing.
• Deep understanding of Apache Spark architecture including RDDs, DataFrames, Datasets, Catalyst Optimizer, DAG execution, and Tungsten engine.
• Strong experience with distributed computing concepts including partitioning, shuffling, caching, broadcast joins, and fault tolerance.
• Advanced SQL skills with databases such as SQL Server, Oracle, PostgreSQL, Snowflake, or Teradata.
• Experience with Hadoop ecosystem technologies including Hive, HDFS, YARN, and Parquet.
• Experience processing streaming data using Spark Structured Streaming, Kafka, or Event Hubs.
• Hands-on experience with cloud platforms including Azure Databricks, AWS EMR, AWS Glue, Azure Synapse Analytics, or Google Dataproc.
• Experience integrating Spark applications with Delta Lake, Iceberg, or Hudi.
• Strong understanding of data warehousing concepts, dimensional modeling, and data lake architecture.
• Experience using Git, CI/CD pipelines, Azure DevOps, GitHub Actions, or Jenkins.
• Strong debugging, troubleshooting, and Spark performance tuning skills.
• Experience working in Agile Scrum development environments.

Responsibilities

• Design, develop, and maintain high-performance distributed data processing applications using Apache Spark.
• Build scalable batch and real-time ETL/ELT pipelines processing large volumes of enterprise data.
• Develop Spark applications using PySpark, Scala, or Spark SQL for data transformation, aggregation, and analytics.
• Optimize Spark jobs for memory utilization, partitioning strategies, shuffle performance, and execution efficiency.
• Process structured, semi-structured, and streaming data from enterprise databases, APIs, Kafka, cloud storage, and data lakes.
• Develop reusable Spark libraries, data processing frameworks, and metadata-driven ingestion pipelines.
• Collaborate with cloud engineering teams to deploy Spark workloads on Databricks, EMR, Azure Synapse, or Kubernetes.
• Implement data quality validation, reconciliation, monitoring, and automated error handling across distributed pipelines.
• Integrate Spark applications with enterprise data warehouses, lakehouses, and reporting platforms.
• Participate in architecture reviews, code reviews, technical design discussions, and Agile development activities.
• Troubleshoot production issues involving distributed processing, cluster performance, resource utilization, and data quality.
• Support cloud migration initiatives by modernizing legacy ETL workloads into Spark-based architectures.

Skills

PySpark

Scala

Spark SQL

SQL

Spark architecture

Distributed systems

Hadoop ecosystem

Kafka

Databricks

Azure/AWS cloud

Tools

Databricks

EMR

Azure Synapse

Kubernetes

Docker

Delta Lake

Iceberg

Hudi

Kafka

Git

CI/CD

Job description

Bright Vision Technologies is seeking an experienced Apache Spark Developer for a 100% remote role in the United States. The position focuses on designing, building, and optimizing large-scale distributed data processing applications that support analytics, ML, real-time reporting, and cloud-based platforms.

You will collaborate with data architects, data engineers, cloud platform teams, ML engineers, and BI developers to deliver scalable Spark-based data pipelines.