Remote Data Platform Engineer - Big Data & Spark Expert

Bright Vision Technologies; Columbus, New Albany, Hilliard (OH, OH, OH); Remote; USD 100,000 - 150,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Data Platform Engineer to design, build, and operate large-scale data processing pipelines on Hadoop. You will manage ingestion, transformation, and analysis of massive datasets to support enterprise analytics and reporting workloads.

The ideal candidate has 6+ years of hands-on Hadoop and Spark experience, strong SQL skills, and familiarity with cloud data platforms. This is a 100% remote, full-time role with competitive compensation.

Qualifications

• 5+ years designing and operating big-data pipelines on Hadoop.
• Strong hands-on expertise with Apache Spark (Scala, Python, or Java) in production environments.
• Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem.
• Hands-on experience with streaming data platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL skills and experience working with both relational and NoSQL data stores.
• Experience with workflow orchestration tools such as Airflow or Oozie.
• Solid understanding of distributed systems concepts, including partitioning, replication, and fault tolerance.
• Strong scripting skills in Python or Shell.
• Excellent troubleshooting, debugging, and documentation skills.

Responsibilities

• Design, build, and operate large-scale data processing pipelines and analytics platforms on Hadoop.
• Ingest, transform, and analyze massive volumes of structured and unstructured data to support analytics, ML, and reporting workloads.
• Deliver reliable, performant, and cost-effective data platforms in production environments.

Skills

Big-data pipelines

Spark (Scala/Python/Java)

Hadoop ecosystem

Kafka

Spark Streaming

Flink

SQL

Airflow

Oozie

Distributed systems

Python/Shell

Troubleshooting

Tools

AWS EMR

Azure HDInsight

Databricks

Delta Lake

Iceberg

Hudi

Apache Atlas

Collibra

Spark-on-K8s

Kubernetes

Terraform

Job description

Bright Vision Technologies is seeking an experienced Data Platform Engineer to design, build, and operate large-scale data processing pipelines on Hadoop. You will manage ingestion, transformation, and analysis of massive datasets to support enterprise analytics and reporting workloads.

The ideal candidate has 6+ years of hands-on Hadoop and Spark experience, strong SQL skills, and familiarity with cloud data platforms. This is a 100% remote, full-time role with competitive compensation.