Sr Big Data Engineer - Oozie and Pig (GCP)

Rackspace Technology; United States; Remote; USD 120,000 - 160,000; Full time

Job summary

A leading company is seeking a Senior Big Data Engineer to design scalable batch processing systems and manage complex data workflows. This fully remote role requires strong expertise in the Hadoop ecosystem and cloud services, particularly GCP. The ideal candidate will have a proven track record in data engineering, effective communication skills, and the ability to thrive in complex environments. Join a diverse team committed to delivering innovative multicloud solutions.

Qualifications

• 5+ years of experience in customer-facing software/technology or consulting.
• 5+ years of experience building and operating solutions on GCP.

Responsibilities

• Design and develop scalable batch processing systems using Hadoop, Oozie, Pig.
• Implement DevOps best practices, including CI/CD pipelines and IaC.

Skills

Java

Python

Oozie

Pig

SQL

GCP

MapReduce

Education

Bachelor's degree in Computer Science

Bachelor's degree in Software Engineering

Tools

Terraform

Airflow

Hadoop

Hive

BigTable

Redis

Job description

2 weeks ago Be among the first 25 applicants

Get AI-powered advice on this job and more exclusive features.

About The Role

We are seeking a Senior Big Data Engineer with deep expertise in distributed systems, batch data processing, and large-scale data pipelines. The ideal candidate has strong hands-on experience with Oozie, Pig, the Apache Hadoop ecosystem, and programming proficiency in Java (preferred) or Python. This role requires a deep understanding of data structures and algorithms, along with a proven track record of writing production-grade code and building robust data workflows.

This is a fully remote position and requires an independent, self-driven engineer who thrives in complex technical environments and communicates effectively across teams.

Work Location: US-Remote, Canada-Remote

Key Responsibilities:

• Design and develop scalable batch processing systems using technologies like Hadoop, Oozie, Pig, Hive, MapReduce, and HBase, with hands-on coding in Java or Python.
• Write clean, efficient, and production-ready code with a strong focus on data structures and algorithmic problem-solving applied to real-world data engineering tasks.
• Develop, manage, and optimize complex data workflows within the Apache Hadoop ecosystem, with a focus on Oozie orchestration and job scheduling.
• Leverage Google Cloud Platform (GCP) tools such as Dataproc, GCS, and Composer to build scalable and cloud-native big data solutions.
• Implement DevOps and automation best practices, including CI/CD pipelines, infrastructure as code (IaC), and performance tuning across distributed systems.
• Collaborate with cross-functional teams to ensure data pipeline reliability, code quality, and operational excellence in a remote environment.

Qualifications:

• Bachelor's degree in Computer Science, Software Engineering, or related field.
• Experience with managed cloud services and understanding of cloud-based batch processing systems.
• Proficiency in Oozie, Airflow, MapReduce, Java, Python, Pig, and SQL.
• Expertise in public cloud services, particularly GCP.
• Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, MapReduce.
• Familiarity with BigTable and Redis.
• Experience with Infrastructure and Applied DevOps principles, including CI/CD and IaC tools like Terraform.
• Proven experience in engineering batch processing systems at scale.

Must Have:

• 5+ years of experience in customer-facing software/technology or consulting.
• 5+ years of experience with “on-premises to cloud” migrations or IT transformations.
• 5+ years of experience building and operating solutions on GCP.
• Proficiency in Oozie and Pig.
• Proficiency in Java or Python.

The pay transparency information for specific states is included in the original description.

About Rackspace Technology

We are the multicloud solutions experts, delivering end-to-end solutions across applications, data, and security. We are committed to diversity and equal employment opportunity. If you require accommodations, please let us know.

Additional Details

Seniority level: Not Applicable

Employment type: Full-time

Job function: Engineering and Information Technology

Industries: IT Services and IT Consulting

Referrals increase your chances of interviewing at Rackspace Technology.