Remote Hadoop Solutions Engineer – Big Data Pipelines

Bright Vision Technologies; Woodbury (MN); Remote; USD 100,000 - 150,000; Full time

Job summary

Bright Vision Technologies is seeking a Hadoop Solutions Developer to design, build, and operate large-scale data pipelines on Hadoop and related ecosystems. You will ingest, transform, and analyze massive data to support analytics, ML, and reporting workloads.

The ideal candidate combines deep Hadoop expertise with software engineering fundamentals to deliver reliable, cost-effective data platforms in production environments.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• Five+ years designing and operating big-data pipelines on Hadoop.
• Hands-on with Apache Spark in production (Scala, Python, or Java).
• Experience with Hive, HDFS, Sqoop, HBase, and Hadoop ecosystem.
• Streaming data platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL and experience with relational and NoSQL stores.
• Workflow orchestration tools like Airflow or Oozie.
• Understanding of distributed systems concepts and fault tolerance.

Responsibilities

• Design, develop, and operate end-to-end big-data pipelines on Hadoop.
• Build ETL/ELT workflows using Spark, Hive, Pig, and Sqoop with data quality and recoverability.
• Develop high-throughput streaming pipelines using Kafka, Spark Streaming, or Flink.
• Optimize Spark/MapReduce jobs for performance and cost efficiency.
• Design data models and storage layouts on HDFS, Hive, HBase, and lakehouse formats.
• Implement data governance, lineage, and quality controls in collaboration with governance teams.
• Monitor, alert, and log big-data pipelines with SLAs and failure detection.
• Mentor junior engineers and contribute to engineering standards.

Skills

Hadoop

Spark

Hive

Kafka

Airflow

Python

Scala

MapReduce

SQL

Shell

Education

Bachelor’s degree in CS/Eng or related

Tools

Spark

Hive

HDFS

Sqoop

HBase

Kafka

Flink

Airflow

Oozie

Databricks

Job description

Bright Vision Technologies is seeking a Hadoop Solutions Developer to design, build, and operate large-scale data pipelines on Hadoop and related ecosystems. You will ingest, transform, and analyze massive data to support analytics, ML, and reporting workloads.

The ideal candidate combines deep Hadoop expertise with software engineering fundamentals to deliver reliable, cost-effective data platforms in production environments.