Remote Hadoop Solutions Engineer | Big-Data Pipelines & Spark

brightvisiontechnologies; United States; Remote; USD 100,000 - 150,000; Full time

Job summary

Bright Vision Technologies seeks an experienced Hadoop Solutions Developer to design, build, and operate large-scale data pipelines on Hadoop ecosystems. You will ingest, transform, and analyze massive data to support analytics, ML, and reporting workloads, delivering reliable, cost-efficient platforms in production.

You will partner with data scientists and engineers, implement governance and monitoring, and drive optimization initiatives across the stack.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field.
• Five or more years designing and operating big-data pipelines on Hadoop.
• Strong hands-on Apache Spark (Scala, Python, or Java) in production.
• Solid experience with Hive, HDFS, Sqoop, HBase, and Hadoop ecosystem.
• Hands-on with streaming platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL skills and experience with relational and NoSQL stores.
• Experience with Airflow or Oozie for workflow orchestration.
• Solid understanding of distributed systems concepts and scripting (Python/Shell).

Responsibilities

• Design, develop, and operate end-to-end big-data pipelines on Hadoop.
• Build ETL/ELT workflows using Spark, Hive, Pig, and Sqoop with data quality.
• Develop high-throughput streaming pipelines with Kafka, Spark Streaming, or Flink.
• Tune Spark and MapReduce for performance and cost efficiency.
• Model data and storage layouts on HDFS, Hive, HBase, and lakehouse formats.
• Implement data governance, lineage, and quality controls with security teams.
• Build monitoring, alerting, and logging for pipelines and SLAs.
• Collaborate with data scientists to deliver reliable datasets.
• Automate orchestration with Airflow, Oozie; manage dependencies clearly.
• Evaluate new tech (EMR, Databricks, Snowflake, BigQuery) for improvements.
• Lead performance reviews and architecture audits of pipelines.

Skills

Big-data fundamentals

Distributed systems

SQL proficiency

Python scripting

Shell scripting

Education

Bachelor’s degree in CS/Engineering

Tools

Apache Spark

Hive

HDFS

Sqoop

HBase

Kafka

Spark Streaming

Flink

Airflow

Oozie

CI/CD

Kubernetes

Databricks

Job description

Bright Vision Technologies seeks an experienced Hadoop Solutions Developer to design, build, and operate large-scale data pipelines on Hadoop ecosystems. You will ingest, transform, and analyze massive data to support analytics, ML, and reporting workloads, delivering reliable, cost-efficient platforms in production.

You will partner with data scientists and engineers, implement governance and monitoring, and drive optimization initiatives across the stack.