Remote Senior Data Engineer - Hadoop & Big Data Pipelines

United States Digital Space LLC; United States; Remote; USD 100,000 - 150,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Senior Data Engineer - Hadoop to design, build, and operate large-scale data processing pipelines on Hadoop and related ecosystems. You will ingest, transform, and analyze massive data to support analytics, ML, and reporting workloads.

The ideal candidate has 6+ years of hands-on experience with Spark, Hive, HDFS, and streaming platforms, and a strong software engineering background focused on reliable, cost-efficient production data platforms

Qualifications

• Bachelor's degree in CS/engineering or related field.
• 5+ years designing and operating big-data pipelines on Hadoop.
• Strong hands-on Spark experience in production.
• Solid Hive, HDFS, Sqoop, HBase experience.
• Streaming data platform experience with Kafka, Spark Streaming or Flink.
• Strong SQL and relational/NoSQL data store skills.
• Experience with Airflow or Oozie for workflows.
• Understanding of distributed systems, partitioning, replication, fault tolerance.
• Scripting in Python or Shell; good debugging and docs.
• Excellent troubleshooting and documentation skills.

Responsibilities

• Design, develop, and operate end-to-end big-data pipelines on Hadoop.
• Build ETL/ELT workflows using Spark, Hive, Pig, and Sqoop.
• Develop streaming pipelines with Kafka, Spark Streaming, or Flink.
• Tune Spark/MapReduce jobs for SLA targets and cost efficiency.
• Model and store data on HDFS, Hive, HBase; use lakehouse formats.
• Implement governance, lineage, and quality controls with teams.
• Monitor, alert, and log pipelines with reliable runbooks.
• Collaborate with data scientists to provide curated datasets.
• Automate workflows with Airflow, Oozie; manage dependencies.
• Evaluate new big-data/cloud tech (EMR, Databricks, Snowflake).
• Lead performance reviews and architecture audits of pipelines.
• Document architectures and runbooks; mentor junior engineers.

Skills

Apache Spark

Hive

HDFS

Sqoop

Kafka

Spark Streaming

Flink

SQL

Airflow

Oozie

Python

Shell

Kubernetes

CI/CD

Education

Bachelor's degree in Computer Science or related

Tools

Hadoop

Databricks

YARN

Job description

Bright Vision Technologies is seeking an experienced Senior Data Engineer - Hadoop to design, build, and operate large-scale data processing pipelines on Hadoop and related ecosystems. You will ingest, transform, and analyze massive data to support analytics, ML, and reporting workloads.

The ideal candidate has 6+ years of hands-on experience with Spark, Hive, HDFS, and streaming platforms, and a strong software engineering background focused on reliable, cost-efficient production data platforms