Remote Spark Engineer - Big Data & Streaming Specialist

Userc Dehradun; United States; Remote; USD 125,000 - 185,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Apache Spark Developer for a 100% remote role in the United States. The position requires 6+ years of software or data-engineering experience and 4+ years in Spark development in enterprise environments.

You will design and optimize large-scale distributed data processing applications, build real-time and batch ETL pipelines, and collaborate with cloud and BI teams to deliver scalable data platforms.

Qualifications

• 6+ years of professional software or data engineering experience.
• 4+ years hands-on Apache Spark development in enterprise environments.
• Strong proficiency in PySpark, Scala, or Spark SQL.
• Deep understanding of Spark architecture and distributed systems.
• Experience with Hadoop ecosystem: Hive, HDFS, YARN.
• Experience with streaming data: Spark Structured Streaming, Kafka.
• Experience with cloud platforms: Databricks, AWS EMR, Azure Synapse, Google Dataproc.
• Experience with Delta Lake / Iceberg.
• Strong SQL skills against SQL Server, Oracle, PostgreSQL, Snowflake, or Teradata.

Responsibilities

• Design, develop, and maintain high-performance distributed data processing applications using Apache Spark.
• Build scalable batch and real-time ETL/ELT pipelines processing large volumes of enterprise data.
• Develop Spark applications using PySpark, Scala, or Spark SQL for data transformation and analytics.
• Optimize Spark jobs for memory, partitioning, shuffle, and execution efficiency.
• Process structured, semi-structured, and streaming data from databases, APIs, Kafka, and data lakes.
• Collaborate with cloud teams to deploy Spark workloads on Databricks, EMR, or Kubernetes.
• Implement data quality validation, monitoring, and error handling across pipelines.
• Integrate Spark apps with data warehouses, lakehouses, and BI tools.
• Participate in architecture reviews and Agile development activities.
• Troubleshoot production issues and optimize performance.

Skills

PySpark

Scala

Spark SQL

Distributed systems

SQL

Kafka

Databricks

AWS EMR

Azure Databricks

Kubernetes

Tools

Databricks

EMR

Azure Synapse

Kubernetes

Docker

Kafka

Delta Lake

Terraform

Job description

Bright Vision Technologies is seeking an experienced Apache Spark Developer for a 100% remote role in the United States. The position requires 6+ years of software or data-engineering experience and 4+ years in Spark development in enterprise environments.

You will design and optimize large-scale distributed data processing applications, build real-time and batch ETL pipelines, and collaborate with cloud and BI teams to deliver scalable data platforms.