GCP Data Engineer: BigQuery & Real-Time Pipelines (Onsite)

Sun Technologies, Inc.; Johns Creek (GA); On-site; USD 69,000 - 90,000; Part time

Benefits offered by this job

Medical Insurance

401(k)

Sick leave

Job summary

Sun Technologies, Inc. is seeking an onsite Data Engineer with strong GCP and SQL skills to design and maintain scalable data pipelines.

You will develop ETL/ELT pipelines with Dataflow, Dataproc, Cloud Composer, and BigQuery, and implement batch and real-time processing with Spark, PySpark, and Dataflow. Responsibilities include data ingestion from APIs and databases, data lake design with GCS, BigQuery modeling, and CI/CD with Cloud Build and Terraform.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Google Cloud Platform services.
• Build and manage ETL/ELT pipelines using Cloud Dataflow, Dataproc, Cloud Composer, and BigQuery.
• Develop complex SQL queries, stored procedures, views, and data transformations in BigQuery.
• Implement batch and real-time data processing using Apache Spark, PySpark, and Dataflow.
• Develop data ingestion pipelines from multiple sources such as APIs, databases, files, and streaming systems.
• Work with Cloud Storage (GCS) for data lake implementation and manage data in structured, semi-structured, and unstructured formats.
• Design and optimize BigQuery data models, partitioning, clustering, and performance strategies.
• Implement real-time streaming pipelines using Pub/Sub and Dataflow.
• Orchestrate and schedule data workflows using Cloud Composer (Apache Airflow).
• Develop PySpark jobs on Dataproc for large-scale distributed data processing.
• Implement data quality, validation, reconciliation, and error-handling frameworks.
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
• Implement CI/CD pipelines for data engineering workloads using Cloud Build, Git, and Terraform.
• Manage security and access using IAM, service accounts, Secret Manager, and encryption.
• Monitor pipelines and troubleshoot failures using Cloud Monitoring, Cloud Logging, and alerting.
• Collaborate with data architects, analysts, data scientists, and business teams to understand data requirements.
• Implement data governance, metadata management, lineage, and security best practices.
• Migrate on-premises or other cloud data platforms to GCP, including data warehouses and ETL workloads.
• Automate infrastructure and GCP resources using Terraform / Infrastructure as Code.
• Create technical documentation for data pipelines, data models, architecture, and operational procedures.

Job description

Sun Technologies, Inc. is seeking an onsite Data Engineer with strong GCP and SQL skills to design and maintain scalable data pipelines.

You will develop ETL/ELT pipelines with Dataflow, Dataproc, Cloud Composer, and BigQuery, and implement batch and real-time processing with Spark, PySpark, and Dataflow. Responsibilities include data ingestion from APIs and databases, data lake design with GCS, BigQuery modeling, and CI/CD with Cloud Build and Terraform.