Remote Hadoop Data Pipeline Engineer

Bright Vision Technologies; Plymouth (MN); Remote; USD 100,000 - 150,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design and operate large‑scale data pipelines on Hadoop. You will build ETL/ELT workflows using Spark, Hive, and Sqoop, while ensuring data quality and reliability in production environments.

The role focuses on streaming pipelines with Kafka, Spark Streaming, or Flink, and optimizing jobs for SLAs and cost. You will collaborate with data scientists to deliver curated datasets and maintain data models across HDFS,

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related technical discipline.
• Five or more years of professional experience designing and operating big‑data pipelines on Hadoop.
• Strong hands‑on expertise with Apache Spark (Scala, Python, or Java) in production environments.
• Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem.
• Hands‑on experience with streaming data platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL skills and experience working with both relational and NoSQL data stores.
• Experience with workflow orchestration tools such as Airflow or Oozie.
• Solid understanding of distributed systems concepts, including partitioning, replication, and fault tolerance.
• Strong scripting skills in Python or Shell.
• Excellent troubleshooting, debugging, and documentation skills.

Responsibilities

• Design, develop, and operate end‑to‑end big‑data pipelines on Hadoop, ingesting data from diverse sources.
• Build robust ETL/ELT workflows using Spark, Hive, Pig, and Sqoop with data quality and recoverability.
• Develop high‑throughput streaming pipelines using Kafka, Spark Streaming, or Flink and integrate with downstream systems.
• Optimize Spark and MapReduce jobs for SLAs at minimal cost through tuning and skew handling.
• Design and maintain data models and storage layouts on HDFS, Hive, HBase, and lakehouse formats.
• Implement data governance, lineage, and quality controls in collaboration with data governance teams.
• Build monitoring, alerting, and logging strategies for big‑data pipelines with clear ownership.
• Partner with data scientists to deliver curated, reliable datasets for analytics and ML.
• Automate pipeline orchestration using Airflow or Oozie with clean dependency management.
• Continuously evaluate new technologies in big‑data and cloud ecosystems.

Skills

Strong SQL skills

Distributed systems understanding

Troubleshooting

Documentation

Education

Bachelor’s degree in Computer Science, Engineering, or related

Tools

Apache Spark

Hive

HDFS

Sqoop

HBase

Kafka

Spark Streaming

Flink

Airflow

Oozie

Delta Lake

Iceberg

Hudi

Kubernetes

CI/CD

Job description

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design and operate large‑scale data pipelines on Hadoop. You will build ETL/ELT workflows using Spark, Hive, and Sqoop, while ensuring data quality and reliability in production environments.

The role focuses on streaming pipelines with Kafka, Spark Streaming, or Flink, and optimizing jobs for SLAs and cost. You will collaborate with data scientists to deliver curated datasets and maintain data models across HDFS,