GCP Data Engineer

Prophecy Technologies Corp; Northern (KY); On-site; USD 90,000 - 120,000; Full time

Job summary

Prophecy Technologies Corp is seeking a data engineer to profile data, build efficient ingestion processes, and develop semantic data layers to enable analytics tools.

You will monitor and support the data extraction, transformation, and loading infrastructure, using SQL and Google Cloud Platform big data technologies, across a variety of data sources.

Qualifications

• 6-8 years of experience in data engineering roles.
• Strong Python and PySpark skills with data pipelines.
• Experience with GCP data tools and BigQuery.
• Familiarity with MySQL and SQL window functions.

Responsibilities

• Profile data, design efficient ingestion pipelines and semantic data layers.
• Support analytics tools via the data pipeline to drive insights.
• Monitor and support infrastructure for ETL processes.
• Work with SQL and GCP big data technologies across sources.

Skills

Python Programming

PySpark

SQL

Window functions

Tools

GCP Data Proc

Composer

GCS

BigQuery

MySQL

Job description

This role involves working on data engineering solutions, leveraging best practices and industry-leading tools and technologies. The primary focus is on profiling data, developing efficient ingestion processes, and building semantic data layers to support analytics tools. The position also entails monitoring and supporting the infrastructure for optimal data extraction, transformation, and loading, utilizing a variety of data sources including SQL and GCP big data technologies.

Key Responsibilities:

• Work on data engineering solutions to profile data, develop efficient ingestion processes, and build semantic data layers.
• Support analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.
• Support and monitor the infrastructure required for optimal extraction, transformation, and loading of data.
• Work with a wide variety of data sources using SQL and GCP big data technologies.

Required Skills:

• Strong skills in Python Programming.
• Strong experience in PySpark.
• Experience with data pipeline and workflow management tools: GCP Data Proc, Composer, GCS.
• Strong experience in handling window functions in SQL.
• BigQuery experience.
• Familiarity with MySQL.

Qualifications:

• 6-8 years of experience in relevant data engineering roles.