Remote Hadoop Data Pipeline Engineer

Bright Vision Technologies; Highlands Ranch (CO); Remote; USD 100,000 - 150,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design, build, and operate large-scale data processing pipelines on Hadoop and related big-data ecosystems. In this role you will be responsible for ingesting, transforming, and analyzing massive volumes of structured and unstructured data to support enterprise analytics, machine learning, and reporting workloads.

The ideal candidate will combine deep technical expertise across the Hadoop ecosystem with strong

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related technical discipline.
• Five or more years of professional experience designing and operating big-data pipelines on Hadoop.
• Strong hands-on expertise with Apache Spark (Scala, Python, or Java) in production environments.

Responsibilities

• Design, develop, and operate end-to-end big-data pipelines on Hadoop, ingesting data from diverse sources.
• Build robust ETL/ELT workflows using Apache Spark, Hive, Pig, and Sqoop, with data quality focus.
• Develop high-throughput streaming data pipelines using Kafka, Spark Streaming, or Flink.
• Optimize Spark and MapReduce jobs for performance and cost.
• Design and maintain data models and storage layouts on HDFS, Hive, HBase, and lakehouse formats.
• Implement data governance, lineage, and quality controls in collaboration with teams.
• Build monitoring, alerting, and logging strategies for big-data pipelines.
• Automate pipeline orchestration using Airflow, Oozie, or similar engines.
• Lead performance reviews and architecture audits; mentor junior engineers.

Skills

Apache Spark

Scala

Python

Java

Hadoop Ecosystem

HDFS

Hive

Sqoop

Kafka

Flink

Spark Streaming

Oozie

Airflow

Kubernetes

SQL

Python Shell

CI/CD

IaC

Education

Bachelor’s degree in Computer Science or related field

Tools

Airflow

Oozie

AWS EMR

Azure HDInsight

Databricks

Kubernetes (Spark-on-K8s)

Terraform

Job description

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design, build, and operate large-scale data processing pipelines on Hadoop and related big-data ecosystems. In this role you will be responsible for ingesting, transforming, and analyzing massive volumes of structured and unstructured data to support enterprise analytics, machine learning, and reporting workloads.

The ideal candidate will combine deep technical expertise across the Hadoop ecosystem with strong