Senior Big Data Engineer: Scalable Pipelines & Streaming

Highbrow LLC; Basking Ridge (NJ); On-site; USD 120,000 - 180,000; Full time

Job summary

Highbrow LLC in Basking Ridge, NJ, seeks a Big Data Engineer to design and lead end-to-end data integration and streaming solutions. You will optimize data flows, implement scalable architectures, and work on data lakes/warehouses on containerized clusters. Strong MS/BS in CS or related with 5+ years in big data environments is required.

The role emphasizes performance tuning, collaboration with QA, and staying current with AI/ML trends to enhance product capabilities.

Qualifications

• 5+ years of hands-on experience designing and implementing data applications in production using Java, Python, or Scala on big data platforms.
• 5+ years of hands-on experience with streaming technologies, specifically with Kafka.
• Proven ability to optimize data flow and design efficient, scalable data pipelines, including data lakes and warehouses on containerized clusters.
• Strong understanding of distributed data architecture concepts (caching, MapReduce, stream processing).
• Solid knowledge of data security, governance, and compliance.
• Proficiency in programming languages: Java, Python, Scala, and SQL.
• Strong analytical capabilities, creativity, and critical thinking skills.
• Effective written and verbal communication, with the ability to facilitate team collaboration.

Responsibilities

• Design and lead end-to-end data integration.
• Perform performance tuning, diagnose, and resolve complex big data and streaming issues.
• Design and implement scalable solutions across the technology stack including data processing, database, web services, network, and security layers.
• Perform integration testing and support QA teams on large-scale projects.
• Contribute to requirements analysis, project estimation, system design, coding, and testing.
• Evaluate and implement third-party libraries and software to enhance product capabilities.
• Stay current with emerging techniques and trends in big data and AI/ML.

Skills

Java

Python

Scala

SQL

Kafka

Big data platforms

Data security & governance

Data pipeline optimization

Tools

Apache Kafka

Hadoop ecosystem

Linux

Containerized clusters

Job description

Highbrow LLC in Basking Ridge, NJ, seeks a Big Data Engineer to design and lead end-to-end data integration and streaming solutions. You will optimize data flows, implement scalable architectures, and work on data lakes/warehouses on containerized clusters. Strong MS/BS in CS or related with 5+ years in big data environments is required.

The role emphasizes performance tuning, collaboration with QA, and staying current with AI/ML trends to enhance product capabilities.