Cloud Data Engineer: Pipelines, Modeling & Analytics

Insight Global (United States); Woonsocket (RI); Remote; USD 76,000 - 83,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

HSA

FSA

DCFSA

401k

Job summary

Insight Global is seeking a Data Engineer to lead the design of data marts and data models that power Data Science and internal analytics teams. You will develop large-scale cloud-based data pipelines, write ETL processes, and ensure data quality across diverse sources.

You will collaborate with data scientists and software engineers to implement automated, scalable solutions using GCP or Azure, BigQuery, Airflow, Kafka, and related tools.

Qualifications

• 3+ years of progressively complex related experience.
• Knowledge in Java, Python, PySpark, Pig, MySQL or NoSQL or similar.
• Strong collaboration and communication skills within and across teams.
• Ability to leverage multiple tools and programming languages to analyze and manipulate data sets from disparate data sources towards Data Integration and Data Quality processes.
• Ability to understand complex systems and solve challenging analytical problems.
• Experience with bash shell scripts, UNIX utilities & UNIX Commands.
• Strong knowledge of large-scale search applications and building high volume data pipelines.
• Ability to communicate technical ideas and results to non-technical clients in written and verbal form.
• Proven ability to create innovative solutions to highly complex technical problems. Advanced knowledge on Cloud Architecture.
• Work experience on GCP or Azure ecosystems.
• Work experience on building CICD enabled pipelines. Advanced knowledge in BigQuery, Cloud Storage, DataProc, Airflow, Kafka, Spanner, Cloud Shell, Composer, HDFS, Python, Hive, Spark, MySQL or NoSQL or similar.
• Experience in Implementing Machine Learning models, automation and Tableau is preferred.
• Experience in RT data ingestion and API/Microservice design & development

Responsibilities

• Design data marts and data models to support Data Science and internal customers
• Develop and implement large & complex cloud initiatives
• Write ETL processes and build real-time and offline analytics pipelines
• Collaborate with data scientists, analysts and engineers to ensure data quality
• Build data pipelines and optimize queries for scalable data processing
• Ingest data from multiple sources and ensure accessibility standards

Skills

3+ years experience

Java

Python

PySpark

BigQuery

Cloud Data Pipelines

GCP/Azure

SQL/NoSQL

Tools

Java

Python

PySpark

MySQL

NoSQL

BigQuery

Cloud Storage

DataProc

Airflow

Kafka

Spanner

Hive

Spark

Job description

Insight Global is seeking a Data Engineer to lead the design of data marts and data models that power Data Science and internal analytics teams. You will develop large-scale cloud-based data pipelines, write ETL processes, and ensure data quality across diverse sources.

You will collaborate with data scientists and software engineers to implement automated, scalable solutions using GCP or Azure, BigQuery, Airflow, Kafka, and related tools.