REMOTE Data Engineer

Insight Global; Woonsocket (RI); Remote; USD 76,000 - 83,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

HSA

FSA

DCFSA

401k

Job summary

Insight Global is seeking a Data Engineer to lead the design of data marts and data models that power Data Science and internal analytics teams. You will develop large-scale cloud-based data pipelines, write ETL processes, and ensure data quality across diverse sources.

You will collaborate with data scientists and software engineers to implement automated, scalable solutions using GCP or Azure, BigQuery, Airflow, Kafka, and related tools.

Qualifications

• 3+ years of progressively complex related experience.
• Knowledge in Java, Python, PySpark, Pig, MySQL or NoSQL or similar.
• Strong collaboration and communication skills within and across teams.
• Ability to leverage multiple tools and programming languages to analyze and manipulate data sets from disparate data sources towards Data Integration and Data Quality processes.
• Ability to understand complex systems and solve challenging analytical problems.
• Experience with bash shell scripts, UNIX utilities & UNIX Commands.
• Strong knowledge of large-scale search applications and building high volume data pipelines.
• Ability to communicate technical ideas and results to non-technical clients in written and verbal form.
• Proven ability to create innovative solutions to highly complex technical problems. Advanced knowledge on Cloud Architecture.
• Work experience on GCP or Azure ecosystems.
• Work experience on building CICD enabled pipelines. Advanced knowledge in BigQuery, Cloud Storage, DataProc, Airflow, Kafka, Spanner, Cloud Shell, Composer, HDFS, Python, Hive, Spark, MySQL or NoSQL or similar.
• Experience in Implementing Machine Learning models, automation and Tableau is preferred.
• Experience in RT data ingestion and API/Microservice design & development

Responsibilities

• Design data marts and data models to support Data Science and internal customers
• Develop and implement large & complex cloud initiatives
• Write ETL processes and build real-time and offline analytics pipelines
• Collaborate with data scientists, analysts and engineers to ensure data quality
• Build data pipelines and optimize queries for scalable data processing
• Ingest data from multiple sources and ensure accessibility standards

Skills

3+ years experience

Java

Python

PySpark

BigQuery

Cloud Data Pipelines

GCP/Azure

SQL/NoSQL

Tools

Java

Python

PySpark

MySQL

NoSQL

BigQuery

Cloud Storage

DataProc

Airflow

Kafka

Spanner

Hive

Spark

Job description

Job Description

We are in search of a Data Engineer who will be responsible for successful delivery of large-scale cloud based applications. You will design data marts and data models to support Data Science and other internal customers. Responsible for developing and implementing large & complex initiatives involving multiple projects. As a member of the Data and Analytics organization, you will be responsible for designing, building, and maintaining best-in-class data engineering pipelines aimed at driving best-in-class solutions primarily using Cloud platforms like GCP(or Azure). You will collaborate with cross-functional teams, including data scientists, analysts, business SME’s and software engineers, to ensure the efficient and reliable processing, storage, and retrieval of data.

Develops large scale data structures and pipelines to organize, collect and standardize data that helps generate insights and addresses reporting needs Writes ETL (Extract / Transform / Load) processes, designs database systems and develops tools for real-time and offline analytical processingCollaborates with data science team to transform data and integrate algorithms and models into automated processesUses knowledge in Data architecture and experience designing & optimizing queries to build data pipelinesUses strong programming skills in Python, Java or any of the major languages to build robust data pipelines and dynamic systemsBuilds data marts and data models to support Data Science and other internal customers. Integrates data from a variety of sources, assuring that they adhere to data quality and accessibility standardsAnalyzes current information technology environments to identify and assess critical capabilities and recommend solutionsExperiments with available tools and advice on new tools in order to determine optimal solution given the requirements dictated by the model/use case

Compensation:$55-60/hour

Exact compensation may vary based on several factors, including skills, experience, and education.

• Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching.
• Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.

Skills and Requirements

• 3+ years of progressively complex related experience
• Knowledge in Java, Python, PySpark Pig, MySQL or NoSQL or similar
• Strong collaboration and communication skills within and across teams.
• Ability to leverage multiple tools and programming languages to analyze and manipulate data sets from disparate data sources towards Data Integration and Data Quality processes
• Ability to understand complex systems and solve challenging analytical problems
• Experience with bash shell scripts, UNIX utilities & UNIX Commands
• Strong knowledge of large-scale search applications and building high volume data pipelines
• Ability to communicate technical ideas and results to non-technical clients in written and verbal form
• Proven ability to create innovative solutions to highly complex technical problems. Ability to understand and build complex systems and solve challenging analytical problems. Advanced knowledge on Cloud Architecture.
• Work experience on GCP or Azure ecosystems.
• Work experience on building CICD enabled pipelines. Advanced knowledge in BigQuery, Cloud Storage, DataProc, Airflow, Kafka, Spanner, Cloud Shell, Composer, HDFS, Python, Hive, Spark, MySQL or NoSQL or similar.
• Experience in Implementing Machine Learning models, automation and Tableau is preferred.3 or more years of progressively complex related experience.
• Experience in RT data ingestion and API/Microservice design & development

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [e-mail].