Remote Hadoop Data Engineer & Big Data Architect

Bright Vision Technologies; United States; Remote; USD 100,000 - 150,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design, build, and operate large-scale data pipelines on Hadoop in a remote setup. You will drive ingestion, transformation, and analytics workloads for enterprise use.

The role requires deep Hadoop ecosystem knowledge, strong software engineering fundamentals, and the ability to deliver reliable, cost-efficient data platforms in production environments across the US.

Qualifications

• Bachelor’s degree in CS, engineering, or related field.
• 5+ years designing and operating Hadoop data pipelines.
• Hands-on Apache Spark in production (Scala, Python, or Java).
• Experience with Hive, HDFS, Sqoop, HBase and Hadoop ecosystem.
• Streaming data platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL and experience with relational and NoSQL stores.
• Experience with Airflow or Oozie.
• Distributed systems concepts: partitioning, replication, fault tolerance.
• Scripting in Python or Shell.
• Excellent troubleshooting and documentation.

Responsibilities

• Design, develop, and operate end-to-end big-data pipelines on Hadoop.
• Build ETL/ELT workflows with Spark, Hive, Pig, and Sqoop.
• Develop streaming pipelines with Kafka, Spark Streaming, or Flink.
• Tune Spark and MapReduce for performance and cost.
• Design data models and storage on HDFS, Hive, HBase; lakehouse formats.
• Implement data governance, lineage, and quality controls.
• Build monitoring, alerting, and logging for pipelines.
• Collaborate with data scientists to provide curated datasets.
• Automate pipelines with Airflow or Oozie.
• Evaluate new technologies like EMR, Databricks, Snowflake, BigQuery.
• Lead performance reviews and architecture audits.
• Document architectures and runbooks.
• Mentor junior engineers and contribute to standards.

Skills

Hadoop

Apache Spark

Kafka

Airflow

Oozie

Hive

HDFS

Sqoop

Python

Shell

SQL

Kubernetes

Databricks

Delta Lake

Iceberg

Hudi

Education

Bachelor’s degree in CS/Engineering

Tools

Hive

HDFS

Sqoop

HBase

Spark

Job description

Bright Vision Technologies is seeking an experienced Hadoop Solutions Developer to design, build, and operate large-scale data pipelines on Hadoop in a remote setup. You will drive ingestion, transformation, and analytics workloads for enterprise use.

The role requires deep Hadoop ecosystem knowledge, strong software engineering fundamentals, and the ability to deliver reliable, cost-efficient data platforms in production environments across the US.