Cloud Data Engineer: Spark, BigQuery & GCP

Corsogen; Houston (TX); On-site; USD 100,000 - 150,000; Full time

Job summary

Corporate Solutions General Inc. is hiring a Software Engineer to work with distributed computing platforms and cloud technologies such as Big Data, Spark, and GCP. You will optimize Spark jobs on large data volumes, develop Python automation scripts, and design efficient BigQuery queries.

Role requires a Master’s in CS/Engineering and 2 years’ experience, including hands-on GCP, BigQuery, Dataproc, Airflow, SQL, Python, and Spark. Houston, TX-based with potential travel and relocation support.

Qualifications

• Masters in Computer Science, Engineering, or related field.
• Minimum 2 years of experience; at least 2 years with GCP, BigQuery, Dataproc, Airflow, SQL, Python and Spark.

Responsibilities

• Work with distributed computing platforms/cloud tech (Big Data) and optimize Spark jobs.
• Develop Python scripts for automation and data output generation.
• Analyze customer data and build models to derive insights.
• Design and optimize BigQuery queries to reduce processing time.
• Migrate existing processes to GCP for better performance.
• Monitor Spark jobs/logs, fix issues, and optimize performance.
• Build scalable data pipelines using BigQuery and GCP (Airflow, Dataproc).

Skills

GCP

BigQuery

Dataproc

Airflow

SQL

Python

Spark

Big Data

Education

Master's degree in Computer Science or Engineering

Tools

BigQuery

Dataproc

Airflow

Python

Spark

SQL

GCP

Job description

Corporate Solutions General Inc. is hiring a Software Engineer to work with distributed computing platforms and cloud technologies such as Big Data, Spark, and GCP. You will optimize Spark jobs on large data volumes, develop Python automation scripts, and design efficient BigQuery queries.

Role requires a Master’s in CS/Engineering and 2 years’ experience, including hands-on GCP, BigQuery, Dataproc, Airflow, SQL, Python, and Spark. Houston, TX-based with potential travel and relocation support.