Senior Data Engineer

Jobtailor; Dearborn (MO); On-site; USD 100,000 - 130,000; Full time

Job summary

Jobtailor in Dearborn, Missouri is seeking a data engineer to automate pipelines and build scalable analytics platforms on GCP. You will organize massive datasets, work with architects to productionalize pipelines, and translate business needs into technical requirements.

The role emphasizes data products, streaming and batch ingestion, and expertise in GCP services and Terraform. Strong communication and collaboration with product teams are essential.

Qualifications

• Master’s degree in Computer Science or related field with 5 years of related experience.
• 5 years SQL development for analyzing and integrating large datasets across data sources.
• 5 years analytics and data product development translating business problems to data requirements.
• 3 years GCP experience with production-scale solutions.
• 2 years GCP native services including BigQuery, GCS, Pub/Sub, Dataflow, Dataproc, Cloud Build.
• 2 years Airflow for scheduling and orchestration of data pipelines.
• 2 years Terraform for Infrastructure as Code.
• 2 years Java or Python for cloud run functions and event-driven processing.

Responsibilities

• Implement methods to automate pipeline stages, reducing manual labor in development and production.
• Analyze complex data, organize raw data, and integrate large datasets into analytical domains.
• Collaborate with architects to productionalize data ingestion, curation, and consumption pipelines.
• Translate business problems into technical data requirements with product management alignment.
• Develop analytical data products using streaming and batch ingestion on Google Cloud Platform.
• Serve as Subject Matter Expert in Data Engineering focusing on GCP native services and related technologies.

Skills

SQL Development

data analysis

data integration

data product development

data ingestion

data curation

data consumption

cloud development

Infrastructure as Code

event-driven programming

Education

Master’s degree in Computer Science

Tools

BigQuery

Google Cloud Storage

Pub/Sub

Dataflow

Dataproc

Cloud Build

Airflow

Terraform

Java

Python

Job description

Responsibilities

• Implement methods for automation of all parts of the pipeline to minimize labor in development and production.
• Analyze complex data, organizing raw data, and integrating massive datasets from multiple data sources to build analytical domains and reusable data products.
• Work with architects to evaluate and productionalize data pipelines for data ingestion, curation, and consumption.
• Communicate with stakeholders to formulate business problems as technical data requirements, identify and implement technical solutions while ensuring key business drivers are captured in collaboration with product management.
• Develop exceptional analytical data products using both streaming and batch ingestion patterns on Google Cloud Platform with solid data warehouse principles.
• Serve as the Subject Matter Expert in Data Engineering with a focus on GCP native services and other well integrated third-party technologies.

Requirements

• Master’s degree or foreign equivalent in Computer Science or related field and 5 years of experience in the job offered or related occupation.
• 5 years of experience with SQL Development for analyzing complex data, organizing raw data, and integrating massive datasets from multiple data sources into data factory data lake.
• 5 years of experience with Analytics and data product development to formulate business problems as technical data requirements, identify and implement technical solutions while ensuring key business drivers are captured in collaboration with product deliverables.
• 3 years of experience with Google Cloud Platform (GCP) experience with solutions designed and implemented at production scale.
• 2 years of experience with GCP native (or equivalent) services including Big Query, Google Cloud Storage, PubSub, Dataflow, Dataproc, and Cloud Build to evaluate and productionalize data pipelines for data ingestion, curation, and consumption.
• 2 years of experience working with Airflow for scheduling and orchestration of data pipelines.
• 2 years of experience utilizing Terraform to provision Infrastructure as Code.
• 2 years of experience using Java or Python for development of cloud run http and event-driven functions which handle http requests and events from cloud environment and use event triggers.

Hard Skills

• SQL Development
• data analysis
• data integration
• data product development
• data ingestion
• data curation
• data consumption
• cloud development
• Infrastructure as Code
• event-driven programming

Soft Skills

• communication
• collaboration
• problem-solving
• stakeholder engagement
• analytical thinking

Certifications and Qualifications

• Master’s degree in Computer Science