Big Data Engineer

TechDigital Group; Santa Clara (CA); On-site; USD 120,000 - 150,000; Full time

Job summary

A leading tech firm in Santa Clara is seeking a Principal Big Data Engineer to drive architecture, design, and development in data ingestion and processing platforms. The ideal candidate will have over 6 years of experience with large data environments, processing tools like Dataflow and GKE, and Java programming. Strong problem-solving and teamwork skills are essential for success in this role.

Qualifications

• 6+ years of experience in design and implementation in an environment with hundreds of terabytes of data.
• 6+ years of experience with large data processing tools.
• 6+ years of experience with Java.

Responsibilities

• Architect, design and develop data ingestion and processing platforms.
• Resolve bottlenecks by communicating with various teams.
• Implement scalable data processing and storage architectures.

Skills

Experience in design and implementation in a large data environment

Proficiency with large data processing tools

Java programming skills

Problem-solving attitude

Interpersonal and teamwork skills

Tools

Dataflow

Google Kubernetes Engine (GKE)

BigQuery (BQ)

Apache Beam

Job description

As a Principal Big Data Engineer, you will be an integral member of our data ingestion and processing platform team responsible for architecture, design and development. Having the dynamic ability to adapt to conventional big-data frameworks and tools with the use-cases required by the project Ability to communicate with research and development teams and data scientists, finding bottlenecks and resolving them Design and implement different architectural models for our scalable data processing, as well as scalable data storage Build tools for proper data ingestion from multiple heterogeneous sources
Skills
1) 6+ years of experience in design and implementation in an environment with hundreds of terabytes of data
2) 6+ years of experience with large data processing tools such as: Dataflow, GKE, BQ, Beam
3) 6+ years of experience with Java
4) Can-do attitude on problem solving, quality and ability to execute
5) Excellent inter-personal and teamwork skills