Senior Apache Spark Engineer – Remote

Bright Vision Technologies; Flower Mound (TX); On-site; USD 125,000 - 185,000; Full time

Job summary

Bright Vision Technologies in the United States seeks an experienced Apache Spark Developer to design, develop, and optimize large-scale distributed data processing applications supporting analytics, ML, real-time reporting, and cloud-based data platforms.

This role focuses on building high-performance Spark pipelines across structured and semi-structured data, delivering scalable, reliable, and cost-efficient data pipelines.

Qualifications

• Six or more years of professional software or data engineering experience.
• Four or more years of hands-on Apache Spark development experience in enterprise production environments.
• Strong proficiency in PySpark, Scala, or Spark SQL for distributed data processing.
• Deep understanding of Apache Spark architecture including RDDs, DataFrames, Datasets, Catalyst Optimizer, DAG execution, and Tungsten engine.
• Strong experience with distributed computing concepts including partitioning, shuffling, caching, broadcast joins, and fault tolerance.
• Advanced SQL skills with databases such as SQL Server, Oracle, PostgreSQL, Snowflake, or Teradata.
• Experience working with Hadoop ecosystem technologies including Hive, HDFS, YARN, and Parquet.
• Experience processing streaming data using Spark Structured Streaming, Apache Kafka, or Event Hubs.
• Hands‑on experience with cloud platforms including Azure Databricks, AWS EMR, AWS Glue, Azure Synapse Analytics, or Google Dataproc.
• Experience integrating Spark applications with Delta Lake, Apache Iceberg, or Apache Hudi.
• Strong understanding of data warehousing concepts, dimensional modeling, and data lake architecture.
• Experience using Git, CI/CD pipelines, Azure DevOps, GitHub Actions, or Jenkins.
• Strong debugging, troubleshooting, and Spark performance tuning skills.
• Experience working in Agile Scrum development environments.

Responsibilities

• Design, develop, and maintain high-performance distributed data processing applications using Apache Spark.
• Build scalable batch and real-time ETL/ELT pipelines processing large volumes of enterprise data.
• Develop Spark applications using PySpark, Scala, or Spark SQL for data transformation, aggregation, and analytics.
• Optimize Spark jobs for memory utilization, partitioning strategies, shuffle performance, and execution efficiency.
• Process structured, semi-structured, and streaming data from enterprise databases, APIs, Kafka, cloud storage, and data lakes.
• Develop reusable Spark libraries, data processing frameworks, and metadata-driven ingestion pipelines.
• Collaborate with cloud engineering teams to deploy Spark workloads on Databricks, EMR, Azure Synapse, or Kubernetes.
• Implement data quality validation, reconciliation, monitoring, and automated error handling across distributed pipelines.
• Integrate Spark applications with enterprise data warehouses, lakehouses, and reporting platforms.
• Participate in architecture reviews, code reviews, technical design discussions, and Agile development activities.
• Troubleshoot production issues involving distributed processing, cluster performance, resource utilization, and data quality.
• Support cloud migration initiatives by modernizing legacy ETL workloads into Spark-based architectures.

Skills

Spark development

PySpark/Scala/Spark SQL

Distributed systems

SQL proficiency

Cloud platforms (Databricks/EMR/Azure)

Agile/Scrum

Data processing pipelines

Tools

Databricks

EMR

Azure Synapse

Kubernetes

Delta Lake

Kafka

Hive

HDFS

YARN

Job description

Bright Vision Technologies in the United States seeks an experienced Apache Spark Developer to design, develop, and optimize large-scale distributed data processing applications supporting analytics, ML, real-time reporting, and cloud-based data platforms.

This role focuses on building high-performance Spark pipelines across structured and semi-structured data, delivering scalable, reliable, and cost-efficient data pipelines.