Cloud Data Engineer: BigQuery & Spark

Openkyber, LLC; Atlanta (GA); Hybrid; USD 66,000 - 103,000; Part time

Job summary

OpenKyber is seeking a Data Engineer to design, develop, and maintain enterprise-scale data pipelines and platforms within a cloud-based data ecosystem. You will collaborate with architects, engineers, analysts, and stakeholders to deliver scalable data solutions for analytics, reporting, and operations.

The ideal candidate will have hands-on experience with GCP, BigQuery, Spark, and a strong background in ETL/ELT design, Python/Java/Scala, and data modeling.

Qualifications

• Hands-on experience with Google Cloud Platform (GCP) and cloud-native data services.
• Advanced experience with BigQuery, including SQL development, query optimization, and large-scale data processing.
• Hands-on experience with Apache Spark and distributed computing frameworks.
• Experience with GCP Dataproc, Google Cloud Storage (GCS), and cloud-based data processing environments.
• Experience working with Apache Hadoop, Apache Hive, and other big data technologies.
• Proven experience building and supporting production-grade data engineering solutions.
• Strong expertise in ETL/ELT design, development, and optimization.
• Strong SQL skills and experience processing large, complex datasets.
• Proficiency in Python and/or Java/Scala.
• Understanding of data modeling, partitioning strategies, distributed processing concepts, and common data storage formats.
• Experience with DevOps practices, CI/CD pipelines, Git version control, automated deployments, and production support.
• Strong analytical, troubleshooting, and problem-solving skills.
• Experience developing reporting or analytics solutions using Looker and/or Power BI.

Responsibilities

• Design, develop, test, and maintain scalable data pipelines and ETL/ELT workflows for large-volume datasets.
• Build and optimize distributed data processing solutions using Apache Spark and GCP Dataproc.
• Develop and maintain analytical datasets and data workloads within BigQuery.
• Create reliable batch and streaming data processing solutions using technologies such as Spark, Hadoop, and Kafka.
• Develop data ingestion and transformation pipelines across source systems, data lakes, and analytics platforms.
• Optimize Spark and BigQuery workloads for performance, scalability, reliability, and cost efficiency.
• Troubleshoot production data pipelines, perform root-cause analysis, and implement long-term solutions.
• Implement data quality controls, monitoring, alerting, and operational support processes.
• Apply software engineering and DevOps best practices, including source control, automated testing, CI/CD, and deployment automation.
• Collaborate with cross-functional teams to translate business requirements into production-ready data solutions.
• Participate in code reviews and contribute to engineering standards, reusable frameworks, and best practices.

Skills

GCP Data Engineering

BigQuery

Apache Spark

Dataproc

Python

SQL

Looker/Power BI

ETL/ELT

Job description

OpenKyber is seeking a Data Engineer to design, develop, and maintain enterprise-scale data pipelines and platforms within a cloud-based data ecosystem. You will collaborate with architects, engineers, analysts, and stakeholders to deliver scalable data solutions for analytics, reporting, and operations.

The ideal candidate will have hands-on experience with GCP, BigQuery, Spark, and a strong background in ETL/ELT design, Python/Java/Scala, and data modeling.