GCP Data Engineer

TechDigital Group; Raleigh (NC); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading technology firm in Raleigh, North Carolina is seeking a GCP Data Engineer to design and implement scalable data pipelines using Google Cloud services. You will oversee deployments with Kubernetes, create CI/CD pipelines with Jenkins, and optimize data workflows for performance. Ideal candidates will have over 3 years of experience with Python and SQL, and familiarity with container orchestration. Join us to deliver value through innovative data solutions.

Qualifications

• 3+ years of experience as a data engineer.
• Proficient in Python and SQL.
• Experience with Kubernetes and Jenkins.

Responsibilities

• Design and implement scalable data pipelines using GCP services.
• Oversee containerized deployments using Helm and Kubernetes.
• Implement CI/CD pipelines using Jenkins for automation.
• Monitor and optimize data workflows for performance.

Skills

Google Cloud Platform Tools

Kubernetes

Jenkins

Python

SQL

Docker

Terraform

Job description

Mandatory: Big Query, Big Table, Google Cloud Storage, PubSub, Data Fusion, Dataflow, Dataproc

Skills

• Google Cloud Platform (GCP) Tools (Preferred): BigQuery, Cloud Storage, Dataflow, Cloud Functions, Pub/Sub, Cloud Run, Cloud Composer (Airflow), Cloud Spanner, Bigtable.
• Container Orchestration: Kubernetes (preferred on GKE) and Helm for managing and deploying containerized applications.
• CI/CD and Automation: Jenkins for building CI/CD pipelines to automate deployment and testing of data pipelines.
• Programming Languages: Proficient in Python for data processing and automation, SQL for querying and data manipulation. Experience with Java is a plus.
• DevOps Tools: Familiarity with Terraform or Deployment Manager for Infrastructure as Code (IaC) to manage GCP resources.
• Monitoring and Logging: Experience with Cloud Monitoring, Datadog, or other monitoring solutions to track pipeline performance and ensure operational efficiency.
• Data Engineering: Expertise in ETL/ELT pipelines, data modeling, and data integration across large datasets.
• Strong understanding of data warehousing and real-time data processing workflows.
• Strong communication skills to work effectively with cross-functional teams and mentor junior developers. Proven ability to lead in an Agile environment.
• 3+ years of experience as a data engineer, with hands-on experience in Kubernetes, Helm, Python and Jenkins.
• Strong experience building and optimizing data pipelines and services in any cloud platform.
• Proficiency in Python and SQL. Familiarity with Java and Docker is a plus.

Job Description

• The GCP Data Engineer will work iteratively on the cloud platform to design, develop and implement scalable, high performance solutions that offer measurable business value to customers.
• Data Solutions: Design and implementation of data pipelines using GCP services.
• Manage Deployments with Helm and Kubernetes: Oversee containerized deployments of data applications on Kubernetes Engine using Helm for package management, ensuring efficient orchestration of services.
• Automation with Jenkins: Implement CI/CD pipelines using Jenkins to automate data pipeline deployment, testing, and integration with other services, ensuring quick iterations and deployments.
• Develop and Optimize Pipelines: Write efficient Python and SQL scripts to build data pipelines and ETL/ELT processes. Continuously monitor and optimize data workflows for performance and cost-effectiveness.
• Data Integration and Orchestration: Design workflows to integrate data from various sources using GCP services, and orchestrate complex tasks with Cloud Composer (Apache Airflow).
• Security and Compliance: Ensure data security and compliance by implementing IAM policies, encryption, and other security measures in GCP, adhering to best practices for handling sensitive data.
• Collaborate Across Teams: Work closely with application developers, data architects, and business stakeholders to define and deliver robust data-driven solutions. Provide technical leadership and ensure alignment between business and technical objectives.
• Monitoring and Logging: Set up monitoring, logging, and alerting using Cloud Monitoring (formerly Stackdriver), Datadog, or other tools to ensure visibility into pipeline performance and quickly identify and resolve issues.