Remote Hadoop Data Pipeline Engineer

Bright Vision Technologies; South Jordan, Draper (UT, UT); Remote; USD 100.000 - 150.000; Vollzeit

Zusammenfassung

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design, build, and operate large-scale data processing pipelines on Hadoop and related big-data ecosystems. You will ingest, transform, and analyze massive volumes of data to support analytics, ML, and reporting workloads.

The ideal candidate will combine deep technical expertise across the Hadoop ecosystem with strong software engineering fundamentals and a clear understanding of how to deliver reliable,

Qualifikationen

• Bachelor's degree in Computer Science, Engineering, or a related technical field.
• 5+ years designing and operating big-data pipelines on Hadoop.
• Hands-on Spark in production (Scala, Python, or Java).
• Experience with Hive, HDFS, Sqoop, HBase, and Hadoop ecosystem.
• Experience with streaming platforms like Kafka, Spark Streaming, or Flink.

Aufgaben

• Design, build, and operate end-to-end big-data pipelines on Hadoop.
• Ingest data from relational, file, streaming, and API sources.
• Develop ETL/ELT workflows with Spark, Hive, Pig, Sqoop.
• Build high-throughput streaming pipelines with Kafka, Spark Streaming, or Flink.
• Tune Spark/MapReduce for performance and cost.
• Design data models on HDFS, Hive, HBase and lakehouse formats.
• Implement data governance, lineage, and quality controls.
• Develop monitoring, alerting, and logging for pipelines.
• Collaborate with data scientists to curate datasets.
• Automate orchestration with Airflow or Oozie.
• Evaluate new technologies (EMR, Databricks, Snowflake, etc.).
• Lead architecture reviews and document runbooks.
• Mentor junior engineers and contribute to standards.

Kenntnisse

Big-data architecture

Mentoring

Troubleshooting

Documentation

Python/Scala/Java programming

Ausbildung

Bachelor's degree in Computer Science/Engineering

Tools

Apache Spark

Hive

HDFS

Sqoop

Kafka

Oozie

Airflow

Kubernetes

Databricks

Jobbeschreibung

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design, build, and operate large-scale data processing pipelines on Hadoop and related big-data ecosystems. You will ingest, transform, and analyze massive volumes of data to support analytics, ML, and reporting workloads.

The ideal candidate will combine deep technical expertise across the Hadoop ecosystem with strong software engineering fundamentals and a clear understanding of how to deliver reliable,