Senior Spark Developer — Remote, Big Data & Cloud

Bright Vision Technologies; Austin (TX); Remote; USD 125,000 - 185,000; Full time

Job summary

Bright Vision Technologies seeks an experienced Apache Spark Developer to design, develop, and optimize large-scale distributed data processing applications supporting analytics, ML, real-time reporting, and cloud platforms. The role emphasizes building high-performance Spark pipelines across structured and semi-structured data, delivering scalable, cost-efficient solutions.

You will collaborate with data architects, cloud engineers, ML engineers, and BI teams to leverage Spark, Databricks, and

Qualifications

• 6+ years of professional software or data engineering experience.
• 4+ years of hands-on Apache Spark development in production.
• Strong proficiency in PySpark, Scala, or Spark SQL for distributed processing.
• Deep understanding of Spark architecture (RDDs, DataFrames, Datasets, Catalyst, DAG, Tungsten).
• Experience with distributed computing concepts (partitioning, shuffling, caching, broadcasts, fault tolerance).
• Advanced SQL skills with SQL Server, Oracle, PostgreSQL, Snowflake, or Teradata.

Responsibilities

• Design, develop, and maintain high-performance distributed data-processing applications using Spark.
• Build scalable batch and real-time ETL/ELT pipelines for enterprise data.
• Develop Spark apps using PySpark, Scala, or Spark SQL for data transformation and analytics.
• Optimize Spark jobs for memory, partitioning, and shuffle performance.
• Ingest data from enterprise sources, APIs, Kafka, and data lakes.
• Collaborate with cloud teams to deploy Spark on Databricks, EMR, Azure Synapse, or Kubernetes.
• Implement data quality validation, monitoring, and error handling across pipelines.
• Integrate Spark with data warehouses, lakehouses, and BI tools.
• Participate in architecture and code reviews and Agile ceremonies.
• Troubleshoot production issues and optimize performance in distributed setups.
• Support cloud migration by modernizing legacy ETL into Spark-based architectures.

Skills

PySpark

Scala

Spark SQL

Spark architecture

Distributed systems

SQL proficiency

Tools

Databricks

Kafka

Hadoop ecosystem

Delta Lake

Kubernetes

Job description

Bright Vision Technologies seeks an experienced Apache Spark Developer to design, develop, and optimize large-scale distributed data processing applications supporting analytics, ML, real-time reporting, and cloud platforms. The role emphasizes building high-performance Spark pipelines across structured and semi-structured data, delivering scalable, cost-efficient solutions.

You will collaborate with data architects, cloud engineers, ML engineers, and BI teams to leverage Spark, Databricks, and