Senior Data Engineer: Hadoop & Spark, Remote

Bright Vision Technologies; Round Rock (TX); On-site; USD 145,000 - 165,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Senior Data Engineer - Hadoop to design, build, and operate large-scale data processing pipelines on Hadoop and related ecosystems. You will ingest, transform, and analyze massive structured and unstructured data to support analytics and reporting workloads.

The ideal candidate has 6+ years of professional big-data pipeline design and operations experience, strong Spark, Hive, and SQL skills, and a proven ability to deliver reliable,

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related technical discipline.
• 5+ years designing and operating big-data pipelines on Hadoop.
• Hands-on Spark (Scala, Python, or Java) in production.
• Experience with Hive, HDFS, Sqoop, HBase in Hadoop ecosystems.
• Experience with Kafka, Spark Streaming, or Flink.
• Strong SQL and relational/NoSQL data stores.
• Airflow or Oozie for workflow orchestration.
• Understanding of distributed systems, partitioning, replication, fault tolerance.
• Python or Shell scripting.
• Troubleshooting and documentation skills.

Responsibilities

• Ingest, transform, and analyze large volumes of data for analytics and reporting.
• Design, build, and operate data pipelines on Hadoop and related ecosystems.
• Deliver reliable, scalable, and cost-effective data platforms in production.

Skills

Spark

SQL

Python

Shell

Troubleshooting

Documentation

Education

Bachelor’s degree in Computer Science or engineering

Tools

Hive

HDFS

Sqoop

HBase

Kafka

Spark Streaming

Flink

Airflow

Oozie

AWS EMR

Azure HDInsight

Databricks

Kubernetes (Spark-on-K8s)

Trino

CI/CD & IaC tools

Job description

Bright Vision Technologies is seeking an experienced Senior Data Engineer - Hadoop to design, build, and operate large-scale data processing pipelines on Hadoop and related ecosystems. You will ingest, transform, and analyze massive structured and unstructured data to support analytics and reporting workloads.

The ideal candidate has 6+ years of professional big-data pipeline design and operations experience, strong Spark, Hive, and SQL skills, and a proven ability to deliver reliable,