Senior GCP Data Engineer — AI/ML & Real-Time Pipelines

Valiant Technologies LLC; Hartford (CT); On-site; USD 120,000 - 180,000; Full time

Job summary

Valiant Technologies LLC is seeking a skilled Data Engineer to design and implement robust data pipelines on Google Cloud Platform (GCP) using BigQuery, Cloud Storage, and Pub/Sub. You will develop workflows with Cloud Composer for orchestration and write clean Python code for scalable data processing.

Ideal candidates will have hands-on experience with Apache Kafka, MongoDB, and SQL optimization in BigQuery, plus Linux scripting and API integration skills.

Qualifications

• Design and implement robust data pipelines using GCP services such as BigQuery, Cloud Storage, and Pub/Sub.
• Develop workflows using Cloud Composer (Airflow) for scheduling and orchestration.
• Write clean, efficient, and scalable Python code.
• Craft complex SQL queries in BigQuery with window functions, CTEs, and performance tuning.
• Build and maintain real-time data processing with Apache Kafka.
• Model and manage MongoDB with scalable schema design.
• Utilize Shell scripting and Linux system administration to support data infrastructure.
• Conduct data profiling and implement validation techniques to ensure data quality and integrity.
• Develop and maintain API integration scripts for automation and data exchange.
• Troubleshoot and resolve data-related issues with strong analytical skills.
• Create and maintain data flow diagrams to communicate architecture clearly.
• Develop and manage workflows using Cloud Composer 3.

Responsibilities

• Design and implement data pipelines on GCP.
• Develop and manage Cloud Composer workflows.
• Write Python code for data processing tasks.
• Query and optimize BigQuery SQL for performance.
• Implement real-time data pipelines with Kafka.
• Manage MongoDB data models and schemas.
• Perform Linux system and shell scripting tasks for infra support.
• Profile data and validate quality across pipelines.
• Create API integrations for automation and data exchange.
• Troubleshoot data issues and document resolutions.
• Diagram data flows for stakeholders.

Skills

AI and ML usage (GPT, Gemini)

Distributed systems

Python programming

SQL (BigQuery)

Real-time data processing

NoSQL (MongoDB)

Shell scripting

Linux administration

Data profiling & validation

API integration

GitHub

RTC

Agile / SAFe

Onshore/offshore collaboration

Communication & documentation

Tools

BigQuery

Cloud Storage

Pub/Sub

Cloud Composer (Airflow)

Apache Airflow

Kafka

MongoDB

Linux

Shell scripting

GitHub

RTC

Cloud Composer 3

Job description

Valiant Technologies LLC is seeking a skilled Data Engineer to design and implement robust data pipelines on Google Cloud Platform (GCP) using BigQuery, Cloud Storage, and Pub/Sub. You will develop workflows with Cloud Composer for orchestration and write clean Python code for scalable data processing.

Ideal candidates will have hands-on experience with Apache Kafka, MongoDB, and SQL optimization in BigQuery, plus Linux scripting and API integration skills.