Remote Hadoop Big Data Engineer - Spark & Pipelines

Bright Vision Technologies; Andover (MA); Remote; USD 100,000 - 150,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Hadoop Big Data Developer to design, build, and operate large-scale data pipelines on Hadoop and related ecosystems. You will ingest, transform, and analyze massive volumes of data to support enterprise analytics, ML, and reporting workloads.

The ideal candidate combines deep technical expertise with strong software engineering fundamentals and a focus on reliable, scalable, and cost-efficient production data platforms.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related technical discipline.
• 5+ years designing and operating Hadoop-based big-data pipelines.
• Hands-on Apache Spark in production (Scala, Python, or Java).
• Strong experience with Hive, HDFS, Sqoop, HBase in Hadoop.
• Experience with streaming platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL and experience with relational and NoSQL stores.
• Experience with Airflow or Oozie workflows.
• Understanding distributed systems concepts including partitioning and fault tolerance.
• Python or Shell scripting proficiency.
• Excellent troubleshooting and documentation skills.

Responsibilities

• Design, build, and operate large-scale data processing pipelines on Hadoop.
• Ingest, transform, and analyze massive volumes of structured and unstructured data.
• Support analytics, machine learning, and reporting workloads in production.
• Deliver reliable, performant, and cost-effective data platforms.

Skills

Apache Spark

SQL

Python

Shell

Distributed systems

Kafka

Spark Streaming

Flink

HDFS

Hive

Sqoop

HBase

Oozie

Airflow

CI/CD

Infrastructure as Code

Education

Bachelor’s degree in Computer Science, Engineering, or related

Tools

Airflow

Oozie

CI/CD

Infrastructure as Code

Job description

Bright Vision Technologies is seeking an experienced Hadoop Big Data Developer to design, build, and operate large-scale data pipelines on Hadoop and related ecosystems. You will ingest, transform, and analyze massive volumes of data to support enterprise analytics, ML, and reporting workloads.

The ideal candidate combines deep technical expertise with strong software engineering fundamentals and a focus on reliable, scalable, and cost-efficient production data platforms.