Big Data Engineer

Highbrow LLC; Basking Ridge (NJ); On-site; USD 120,000 - 180,000; Full time

Job summary

Highbrow LLC in Basking Ridge, NJ, seeks a Big Data Engineer to design and lead end-to-end data integration and streaming solutions. You will optimize data flows, implement scalable architectures, and work on data lakes/warehouses on containerized clusters. Strong MS/BS in CS or related with 5+ years in big data environments is required.

The role emphasizes performance tuning, collaboration with QA, and staying current with AI/ML trends to enhance product capabilities.

Qualifications

• 5+ years of hands-on experience designing and implementing data applications in production using Java, Python, or Scala on big data platforms.
• 5+ years of hands-on experience with streaming technologies, specifically with Kafka.
• Proven ability to optimize data flow and design efficient, scalable data pipelines, including data lakes and warehouses on containerized clusters.
• Strong understanding of distributed data architecture concepts (caching, MapReduce, stream processing).
• Solid knowledge of data security, governance, and compliance.
• Proficiency in programming languages: Java, Python, Scala, and SQL.
• Strong analytical capabilities, creativity, and critical thinking skills.
• Effective written and verbal communication, with the ability to facilitate team collaboration.

Responsibilities

• Design and lead end-to-end data integration.
• Perform performance tuning, diagnose, and resolve complex big data and streaming issues.
• Design and implement scalable solutions across the technology stack including data processing, database, web services, network, and security layers.
• Perform integration testing and support QA teams on large-scale projects.
• Contribute to requirements analysis, project estimation, system design, coding, and testing.
• Evaluate and implement third-party libraries and software to enhance product capabilities.
• Stay current with emerging techniques and trends in big data and AI/ML.

Skills

Java

Python

Scala

SQL

Kafka

Big data platforms

Data security & governance

Data pipeline optimization

Tools

Apache Kafka

Hadoop ecosystem

Linux

Containerized clusters

Job description

Title: Big Data Engineer

Location: Basking Ridge, NJ (5-day onsite/week)

IRC : https://recruitment.globallogic.com/apps/recruitment/vacancies/IRC303514

Job Responsibilities:

• Design and lead end-to-end data integration
• Perform performance tuning, diagnose, and resolve complex big data and streaming technical issues.
• Design and implement scalable solutions across the technology stack, including data processing, database, web services, network, and security layers.
• Perform integration testing and provide support to QA teams on large-scale projects.
• Contribute to requirements analysis, project estimation, system design, coding, and testing.
• Evaluate and implement appropriate third‑party libraries and software to enhance product capabilities.
• Stay current with emerging techniques and trends in big data and AI/ML.

Job Requirement:

• 5+ years of hands‑on experience designing and implementing data applications in production using Java, Python, or Scala on big data platforms, with deep expertise in big data technologies and data optimization techniques.
• 5+ years of hands‑on experience with streaming technologies, specifically with a strong focus on Kafka, and various open‑source software platforms (e.g., Apache ecosystem, Linux).
• Proven ability to optimize data flow and design efficient, scalable data pipelines, including data lakes and warehouses on containerized clusters.
• Strong understanding of distributed data architecture concepts (e.g., caching, MapReduce, stream processing).
• Solid knowledge of data security, governance, and compliance.
• Proficiency in programming languages: Java, Python, Scala, and SQL.
• Strong analytical capabilities, creativity, and critical thinking skills.
• Effective written and verbal communication, with the ability to facilitate team collaboration.