Remote Big Data Engineer - Spark & Analytics Platform

Bright Vision Technologies; Palo Alto (CA); On-site; USD 90,000 - 110,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Big Data Engineer to design, build, and operate large-scale data pipelines on Hadoop and related ecosystems. This role focuses on ingesting, transforming, and analyzing vast data to support analytics, ML, and reporting workloads, with production-grade reliability.

The ideal candidate will have 6+ years of experience, deep Hadoop and Spark expertise, and a strong foundation in distributed systems and scripting.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related technical discipline.
• 5+ years designing and operating big-data pipelines on Hadoop.
• Strong hands‑on Apache Spark (Scala, Python, or Java) in production.
• Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem.
• Hands‑on experience with streaming data platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL skills and experience working with both relational and NoSQL data stores.
• Experience with workflow orchestration tools such as Airflow or Oozie.
• Solid understanding of distributed systems concepts, including partitioning, replication, and fault tolerance.
• Strong scripting skills in Python or Shell.
• Excellent troubleshooting, debugging, and documentation skills.

Responsibilities

• Design, build, and operate large-scale data processing pipelines on Hadoop and related ecosystems.
• Ingest, transform, and analyze massive volumes of structured and unstructured data for analytics, ML, and reporting workloads.
• Deliver reliable, performant, and cost-effective data platforms in production.

Skills

Hadoop

Apache Spark

Hive/HDFS/Sqoop

Kafka

Spark Streaming

Flink

SQL

Airflow

Oozie

Python

Shell

Distributed systems

Troubleshooting

Education

Bachelor’s degree in Computer Science/Engineering

Tools

CI/CD

Infrastructure as Code

Job description

Bright Vision Technologies is seeking an experienced Big Data Engineer to design, build, and operate large-scale data pipelines on Hadoop and related ecosystems. This role focuses on ingesting, transforming, and analyzing vast data to support analytics, ML, and reporting workloads, with production-grade reliability.

The ideal candidate will have 6+ years of experience, deep Hadoop and Spark expertise, and a strong foundation in distributed systems and scripting.