Remote: Senior Hadoop Data Pipeline Engineer

Bright Vision Technologies; Bothell (WA); On-site; USD 100,000 - 150,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer for 100% remote work in the U.S. You will design, build, and operate large-scale data processing pipelines on Hadoop and related ecosystems, ingesting, transforming, and analyzing data to support enterprise analytics, machine learning, and reporting workloads.

The ideal candidate has 6+ years of experience, strong Spark and Hadoop skills, and a proven track record delivering reliable, scalable data platforms in

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related technical discipline.
• 6+ years designing and operating big-data pipelines on Hadoop.
• Hands-on expertise with Apache Spark in production environments.
• Solid experience with Hive, HDFS, Sqoop, HBase, and the Hadoop ecosystem.
• Experience with streaming platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL and experience with relational and NoSQL stores.
• Workflow orchestration via Airflow or Oozie.
• Knowledge of distributed systems concepts: partitioning, replication, fault tolerance.
• Scripting in Python or Shell.
• Excellent troubleshooting, debugging, and documentation skills.

Responsibilities

• Design, develop, and operate end-to-end big-data pipelines on Hadoop.
• Build ETL/ELT workflows with Spark, Hive, Pig, and Sqoop.
• Develop high-throughput streaming pipelines with Kafka, Spark Streaming, or Flink.
• Tune Spark/MapReduce jobs for SLAs and cost efficiency.
• Design data models and storage on HDFS/Hive/HBase and modern lakehouse formats.
• Implement data governance, lineage, and quality controls.
• Monitor, alert, and log big-data pipelines with robust practices.
• Collaborate with data scientists to provide reliable datasets.
• Automate pipelines with Airflow, Oozie, or similar tools.
• Evaluate new big-data and cloud technologies (EMR, Databricks, Snowflake, BigQuery).
• Lead performance reviews and architecture audits.
• Document architectures, schemas, and runbooks for production support.
• Mentor junior engineers and uphold engineering standards.

Skills

Spark proficiency (Scala/Python/Java)

Distributed systems

Troubleshooting

Documentation

Mentoring juniors

Education

Bachelor’s degree in CS/Engineering or related

Tools

Apache Spark

Hive

HDFS

Sqoop

Kafka

Flink

Airflow

Oozie

Databricks

Spark-on-K8s

Job description

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer for 100% remote work in the U.S. You will design, build, and operate large-scale data processing pipelines on Hadoop and related ecosystems, ingesting, transforming, and analyzing data to support enterprise analytics, machine learning, and reporting workloads.

The ideal candidate has 6+ years of experience, strong Spark and Hadoop skills, and a proven track record delivering reliable, scalable data platforms in