Senior Big Data Engineer - Remote Data Pipelines

Bright Vision Technologies; Wesley Chapel (FL); Remote; USD 100,000 - 150,000; Full time

Job summary

Bright Vision Technologies is seeking an experienced Big Data Application Developer to design, build, and operate large-scale data processing pipelines on Hadoop and related ecosystems. The role focuses on ingesting, transforming, and analyzing massive data volumes to support analytics, ML, and reporting workloads.

The ideal candidate will have deep Hadoop expertise, production-grade software engineering skills, and a clear understanding of delivering scalable data platforms in production

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related technical discipline.
• 5+ years designing and operating big-data pipelines on Hadoop.
• Strong hands-on Apache Spark in production environments.
• Solid experience with Hive, HDFS, Sqoop, HBase and Hadoop ecosystem.
• Hands-on with streaming platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL skills with relational and NoSQL data stores.
• Experience with Airflow or Oozie for workflow orchestration.
• Understanding of distributed systems concepts like partitioning and fault tolerance.
• Strong scripting in Python or Shell.
• Excellent troubleshooting, debugging and documentation skills.

Responsibilities

• Design, build and operate large-scale data processing pipelines.
• Ingest, transform and analyze massive volumes of data for analytics and ML workloads.
• Deliver reliable, performant, and cost-effective data platforms in production.

Skills

SQL

Distributed systems concepts

Python

Shell

Troubleshooting

Documentation

Education

Bachelor’s degree in Computer Science or related field

Tools

Apache Spark

Hive/HDFS/Sqoop/HBase

Kafka/Spark Streaming/Flink

Airflow/Oozie

CI/CD / Infrastructure as Code

Kubernetes

Job description

Bright Vision Technologies is seeking an experienced Big Data Application Developer to design, build, and operate large-scale data processing pipelines on Hadoop and related ecosystems. The role focuses on ingesting, transforming, and analyzing massive data volumes to support analytics, ML, and reporting workloads.

The ideal candidate will have deep Hadoop expertise, production-grade software engineering skills, and a clear understanding of delivering scalable data platforms in production