Remote Big Data Engineer: Spark & Data Platform

Bright Vision Technologies; San Diego, 4S Ranch (CA, CA); Remote; USD 90,000 - 110,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Big Data Engineer to design and operate large-scale data processing pipelines on Hadoop ecosystems. You will ingest, transform, and analyze massive data to support analytics, ML, and reporting workloads.

The role requires deep Hadoop/Spark expertise, strong software engineering fundamentals, and the ability to deliver reliable, cost-effective data platforms in production environments.

Qualifications

• Bachelor's degree in CS, Engineering or related technical discipline.
• 5+ years designing and operating big-data pipelines on Hadoop.
• Strong hands-on Spark (Scala, Python, or Java) in production.
• Solid experience with Hive, HDFS, Sqoop, HBase and Hadoop ecosystem.
• Hands-on experience with streaming platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL skills with relational and NoSQL stores.
• Experience with Airflow or Oozie for workflows.
• Understanding of distributed systems concepts like partitioning and replication.
• Strong Python or Shell scripting and excellent debugging skills.

Responsibilities

• Design, build, and operate large-scale data processing pipelines and analytics platforms on Hadoop.
• Ingest, transform, and analyze massive structured and unstructured data.
• Deliver reliable, performant, and cost-effective data platforms in production environments.

Skills

Apache Spark

SQL

Python

Shell scripting

Hadoop ecosystem

Kafka/Spark Streaming/Flink

Airflow/Oozie

Distributed systems

Education

Bachelor's degree in CS/Engineering

Tools

Hive/HDFS/Sqoop/HBase

Job description

Bright Vision Technologies is seeking an experienced Big Data Engineer to design and operate large-scale data processing pipelines on Hadoop ecosystems. You will ingest, transform, and analyze massive data to support analytics, ML, and reporting workloads.

The role requires deep Hadoop/Spark expertise, strong software engineering fundamentals, and the ability to deliver reliable, cost-effective data platforms in production environments.