Real-Time GCP BigQuery Data Engineer (NiFi & Kafka)

Virtues

Irving (TX)

On-site

USD 100,000 - 110,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

Virtues is seeking a GCP BigQuery Data Engineer to design cloud-native, scalable data platforms and build real-time ingestion pipelines on Google Cloud. This onsite role in Irving, TX focuses on event-driven streaming architectures that support near real-time analytics and enterprise reporting.

Responsibilities include designing and optimizing data warehouses with BigQuery, building streaming pipelines with NiFi and Kafka, and implementing ETL/ELT processes for scalable data lakes and analytics

Qualifications

7+ years of experience in Data Engineering, Data Warehousing, or Big Data platforms.

Responsibilities

Design, develop, and optimize enterprise data warehouse solutions using Google BigQuery.
Design, develop, and maintain real-time data ingestion pipelines using Apache NiFi to extract, transform, and route streaming data into Apache Kafka topics.
Build scalable event-driven streaming architectures using Apache NiFi, Apache Kafka, and Google BigQuery to enable high-throughput, fault-tolerant, low-latency processing.
Configure and optimize Apache NiFi processors for extraction, transformation, routing, filtering, schema validation, error handling, retry mechanisms, and reliable delivery to Kafka.
Develop streaming ingestion solutions that allow Google BigQuery to consume Kafka event streams and transform near real-time events into analytical tables and reporting datasets.
Design data models and implement ETL/ELT processes to move data from raw to curated and published layers.
Design, create, and manage large-scale BigQuery datasets, including temporary/permanent and internal/external tables.
Optimize BigQuery workloads using query tuning, partitioning, clustering, and cost optimization techniques to improve performance and reduce cloud costs.
Monitor, troubleshoot, and optimize streaming workloads by tuning NiFi flows, Kafka topics/partitions, and BigQuery streaming ingestion to support high availability, data integrity, minimal latency, and cost-efficient processing.
Build scalable data lake frameworks and ingestion pipelines using cloud-native GCP technologies.
Develop reporting and visualization solutions using Looker, Looker Studio (Data Studio), Connected Sheets, and other BigQuery reporting tools.
Collaborate with data architects, modelers, developers, DevOps engineers, project managers, and business stakeholders to deliver scalable enterprise analytics solutions and continuous improvements to the data platform.