Data Pipeline Engineer – Big Data

TransUnion LLC; New York (NY); Hybrid; USD 72,000 - 105,000; Full time

Benefits offered by this job

Health insurance

Dental coverage

Vision coverage

HSA/FSA options

Life and AD&D insurance

Disability insurance

Legal plan

Pet insurance

Travel accident coverage

401(k) with employer match

ESPP

Tuition reimbursement

Paid holidays

Job summary

TransUnion LLC is seeking a Data Development Consultant to support Marketing Services products by building and maintaining data pipelines. The role involves fixing code, analyzing data-related issues, and developing end-to-end data ingestion for multiple clients.

You will work with Architecture, Product, and Engineering teams in a hybrid setting. The ideal candidate has 5+ years in Big Data, strong SQL and UNIX skills, and experience with Hadoop ecosystems.

Qualifications

• Bachelor's Degree with 5+ years related IT experience in Big Data.
• Excellent SQL skills and in-depth understanding of relational databases.
• Experience with distributed systems frameworks like Hadoop (HDFS/MapReduce) and cloud platforms (Google Cloud).
• Knowledge of UNIX environments and shell scripting.
• Experience with large-scale multi-node clusters (HDFS/BigQuery/Hive).
• Familiarity with OLAP/OLTP and ETL principles.
• Proficiency in Python and Java is a plus.

Responsibilities

• Manage day-to-day activities to facilitate project delivery.
• End-to-end data onboarding and create data pipelines for applications.
• Analyze source data to map business rules and insights.
• Develop, code, and implement the data pipeline.
• Ensure client-level data onboarding is successful.
• Collaborate with Product, Solution Architecture, Data Modeler, and Advisory Services.

Skills

SQL

Python

Java

UNIX

Shell scripting

Hadoop

Big Data

Google Cloud Platform

Education

Bachelor's Degree in Computer Science/IT

Tools

HDFS

BigQuery

Hive

Job description

TransUnion LLC is seeking a Data Development Consultant to support Marketing Services products by building and maintaining data pipelines. The role involves fixing code, analyzing data-related issues, and developing end-to-end data ingestion for multiple clients.

You will work with Architecture, Product, and Engineering teams in a hybrid setting. The ideal candidate has 5+ years in Big Data, strong SQL and UNIX skills, and experience with Hadoop ecosystems.