Remote Apache Spark Engineer for Big Data Analytics

United States Digital Space LLC; United States; Remote; USD 125,000 - 185,000; Full time

Job summary

Bright Vision Technologies seeks an experienced Apache Spark Developer to design, develop, and optimize large-scale distributed data processing applications for analytics, ML, real-time reporting, and cloud platforms. You will work with data architects, cloud platform teams, ML engineers, and BI developers to build modern data pipelines using Spark, Databricks, and related technologies.

The ideal candidate has deep Spark knowledge, distributed systems experience, and strong debugging and

Qualifications

• Six+ years of professional software or data engineering experience.
• Four+ years of hands-on Apache Spark development in enterprise environments.
• Strong proficiency in PySpark, Scala, or Spark SQL for distributed data processing.
• Deep understanding of Spark architecture including RDDs, DataFrames, Datasets, Catalyst Optimizer, DAG execution, and Tungsten engine.
• Experience with distributed computing concepts including partitioning, shuffling, caching, and fault tolerance.
• Advanced SQL skills with SQL Server, Oracle, PostgreSQL, Snowflake, or Teradata.
• Experience with Hadoop ecosystem technologies including Hive, HDFS, YARN, Parquet.
• Experience processing streaming data using Spark Structured Streaming, Kafka, or Event Hubs.
• Hands-on experience with cloud platforms including Azure Databricks, AWS EMR, AWS Glue, Azure Synapse, or Google Dataproc.
• Experience integrating Spark applications with Delta Lake, Iceberg, or Hudi.
• Strong understanding of data warehousing concepts and data lake architecture.
• Experience with Git, CI/CD pipelines, Azure DevOps, GitHub Actions, or Jenkins.
• Strong debugging and Spark performance tuning skills.
• Experience working in Agile Scrum environments.

Responsibilities

• Design, develop, and maintain high-performance distributed data processing applications using Apache Spark.
• Build scalable batch and real-time ETL/ELT pipelines processing large volumes of enterprise data.
• Develop Spark applications using PySpark, Scala, or Spark SQL.
• Optimize Spark jobs for memory, partitioning, shuffle, and execution efficiency.
• Process structured, semi-structured, and streaming data from various sources.
• Develop reusable Spark libraries and ingestion pipelines.
• Collaborate with cloud engineering teams to deploy Spark workloads on Databricks, EMR, Azure Synapse, or Kubernetes.
• Implement data quality validation, reconciliation, and monitoring across pipelines.
• Integrate Spark with data warehouses, lakehouses, and reporting platforms.
• Participate in architecture reviews, code reviews, and Agile development.
• Troubleshoot production issues related to distributed processing and data quality.
• Support cloud migration by modernizing legacy ETL into Spark-based architectures.

Skills

PySpark

Scala

Spark SQL

Spark architecture

Distributed computing

Advanced SQL

Hadoop Ecosystem

Structured Streaming

Cloud platforms

Delta Lake

Data warehousing

Git & CI/CD

Agile/Scrum

Tools

Databricks

Kafka

Kubernetes

Terraform

Airflow

Job description

Bright Vision Technologies seeks an experienced Apache Spark Developer to design, develop, and optimize large-scale distributed data processing applications for analytics, ML, real-time reporting, and cloud platforms. You will work with data architects, cloud platform teams, ML engineers, and BI developers to build modern data pipelines using Spark, Databricks, and related technologies.

The ideal candidate has deep Spark knowledge, distributed systems experience, and strong debugging and