GCP Data Engineer

Programmers.io; California (MO); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading data engineering company in California is seeking a skilled Data Engineer to design and optimize analytical datasets and maintain robust data pipelines in cloud environments. The ideal candidate will have extensive experience with Google Cloud Platform, particularly BigQuery, strong skills in Python and SQL, and a background in data pipeline orchestration. This role promises opportunities to work collaboratively with product managers and DevOps teams to deliver impactful data solutions.

Qualifications

• Expert-level skills in data ingestion and performance optimization within a petabyte-scale environment.
• Proficiency in Python using data engineering libraries.
• Solid command of advanced SQL for complex querying and data processing.

Responsibilities

• Design, implement, and optimize analytical datasets for reporting and business analysis.
• Architect and maintain scalable data pipelines for business intelligence.
• Implement Big Data solutions for both batch and streaming analytics.

Skills

Cloud Platform Expertise (GCP Focus)

Python

SQL

Data Pipeline Orchestration

DevOps/CI/CD

Monitoring & Automation

Tools

BigQuery

Apache Airflow

Git

Docker

Kubernetes

Job description

Note: (GC, GC-EAD, OPT, CPT on C2C will not be workable)

Core Responsibilities

• Design, implement, and optimize clean, well-structured, and performant analytical datasets to support high-volume reporting, business analysis, and data science model development.
• Architect, build, and maintain scalable and robust data pipelines for diverse applications, including business intelligence and advanced analytics.
• Implement and support Big Data solutions for both batch (scheduled) and real-time/streaming analytics.
• Work closely with product managers and business teams to understand data requirements and translate them into technical solutions.
• Collaborate with DevOps teams to ensure smooth deployment, monitoring, and maintenance of data pipelines and infrastructure in cloud environments.

Required Skills & Experience

• Cloud Platform Expertise (GCP Focus): Extensive hands-on experience working in dynamic cloud environments, with a strong preference for Google Cloud Platform (GCP) services, specifically:
• BigQuery: Expert-level skills in data ingestion, performance optimization, and data modeling within a petabyte-scale environment.
• Experience with other relevant GCP services like Cloud Storage, Cloud Dataflow/Beam, or Pub/Sub.
• Python: Expert-level programming proficiency in Python, including experience with relevant data engineering libraries.
• SQL: A solid command of advanced SQL for complex querying, data processing, and performance tuning.

• Data Pipeline Orchestration: Prior experience using workflow management and orchestration tools (e.g., Apache Airflow, Cloud Composer, Dagster, or similar).

• DevOps/CI/CD:
• Strong understanding of DevOps principles and practices.
• Experience with CI/CD pipelines, automation tools, and deployment strategies.
• Familiarity with version control systems (Git) and tools like GitLab CI/CD, GitHub Actions, or Jenkins.
• Knowledge of containerization (Docker) and orchestration tools (Kubernetes) is a plus.

• Monitoring & Automation: Ability to implement monitoring solutions and automate operational tasks to ensure reliability and scalability.