Data Engineer: Scalable ETL & Big Data Pipelines

Scale Marketing; Chicago (IL); On-site; USD 110,000 - 160,000; Full time

Job summary

Scale Marketing is hiring a Data Engineer to expand and optimize its data pipeline architecture, data flow, and ETL processes. The role will build infrastructure for ETL from varied data sources using SQL and big data technologies, and develop analytics tools to drive insights into customer acquisition and operations.

The ideal candidate has 5+ years in a Data Engineer role, advanced SQL skills, and experience with Hadoop, Spark, Hive, Pig, and at least one cloud service (AWS/Azure).

Qualifications

• Advanced SQL and relational database experience.
• Big data pipelines, architectures, and datasets expertise.
• Root-cause analysis on data and processes.
• Strong analytical skills with unstructured datasets.
• Data transformation, data structures, metadata, dependency/workload management experience.
• Proven history with large disconnected datasets.
• Knowledge of message queuing, stream processing, and scalable big data stores.
• Project management and organizational skills.
• Experience with cross-functional teams in a dynamic environment.
• 5+ years in a Data Engineer role.
• Graduate degree in CS/Statistics/Informatics/IS or similar.
• Experience with Hadoop/Hive/Spark/Pig.
• Experience with 3+ SQL/NoSQL databases.
• Experience with data pipeline/workflow tools like Kafka, Spark, Azure DevOps, AWS Data Pipeline.
• Experience with at least one cloud: AWS or Azure.
• Experience with at least one stream-processing system: Kinesis, Azure Stream Analytics, Flink.
• Proficiency in Python, Java, Scala, C++, Linux/Unix.

Responsibilities

• Create and maintain optimal data pipeline architecture.
• Assemble large, complex data sets to satisfy business requirements.
• Automate manual processes and optimize data delivery.
• Redesign infrastructure for scalability.
• Build ETL infrastructure using SQL and big data technologies.
• Develop analytics tools leveraging the data pipeline for insights.
• Collaborate with Partner, Client, and Data teams on data issues.
• Keep data separated and secure across national boundaries.
• Create data tools to support analytics and data scientists.
• Work with data and analytics experts to improve data systems.

Skills

Advanced SQL

Root cause analysis

Analytical skills

Cross-functional collaboration

Project management

5+ years Data Engineer experience

Education

Graduate degree in Computer Science or quantitative field

Tools

Apache Hadoop

Apache Hive

Apache Spark

Apache Pig

MongoDB

HBase

Oracle

SQL Server

Cassandra

Neo4j

Redis

Riak

Apache Kafka

Azure DevOps

AWS Data Pipeline

AWS

Microsoft Azure

AWS Kinesis

Azure Stream Analytics

Apache Flink

Python

Java

Scala

C++

Linux/Unix

Job description

Scale Marketing is hiring a Data Engineer to expand and optimize its data pipeline architecture, data flow, and ETL processes. The role will build infrastructure for ETL from varied data sources using SQL and big data technologies, and develop analytics tools to drive insights into customer acquisition and operations.

The ideal candidate has 5+ years in a Data Engineer role, advanced SQL skills, and experience with Hadoop, Spark, Hive, Pig, and at least one cloud service (AWS/Azure).