Sr Big Data Engineer - Oozie and Pig (GCP)

Rackspace Technology; United States; Remote; USD 120,000 - 160,000; Full time

Job summary

A leading technology company is looking for a Senior Big Data Engineer to design scalable batch processing systems. The ideal candidate will have expertise in distributed systems and a strong background in programming with Java and Python, along with extensive experience in cloud environments. This fully remote position requires an independent engineer who can collaborate effectively with teams across the globe.

Qualifications

• 5+ years in customer-facing software/technology or consulting.
• 5+ years in 'on-premises to cloud' migrations.
• Experience building solutions on GCP.

Responsibilities

• Design and develop scalable batch processing systems using technologies like Hadoop.
• Lead Jira Epics to manage project workflows.
• Collaborate with cross-functional teams remotely.

Skills

Proficiency in Oozie

Proficiency in Pig

Strong programming skills in Java

Strong programming skills in Python

Experience with MapReduce

Proficiency in SQL

Experience with GCP

Education

Bachelor's degree in Computer Science

Software Engineering or related field

Tools

Terraform

GCP tools (Dataproc, GCS, Composer)

Job description

About the Role

We are seeking a Senior Big Data Engineer with expertise in distributed systems, batch data processing, and large-scale data pipelines. The ideal candidate has hands-on experience with Oozie, Pig, the Apache Hadoop ecosystem, and programming in Java (preferred) or Python. This role requires a strong understanding of data structures and algorithms, along with a proven record of writing production-grade code and building robust data workflows.

This is a fully remote position and requires an independent, self-driven engineer who communicates effectively across teams.

Work Location: US-Remote, Canada-Remote

Key Responsibilities:

1. Design and develop scalable batch processing systems using technologies like Hadoop, Oozie, Pig, Hive, MapReduce, and HBase, with coding in Java or Python (Java is essential).
2. Lead Jira Epics to manage project workflows.
3. Write clean, efficient, production-ready code focusing on data structures and algorithms for real-world data engineering tasks.
4. Develop, manage, and optimize complex data workflows within the Hadoop ecosystem, emphasizing Oozie orchestration and job scheduling.
5. Utilize GCP tools such as Dataproc, GCS, and Composer to build scalable, cloud-native big data solutions.
6. Implement DevOps and automation best practices, including CI/CD pipelines, IaC (e.g., Terraform), and performance tuning.
7. Collaborate remotely with cross-functional teams to ensure data pipeline reliability, code quality, and operational excellence.

Qualifications:

• Bachelor's degree in Computer Science, Software Engineering, or related field.
• Experience with managed cloud services and cloud-based batch processing systems.
• Proficiency in Oozie, Airflow, MapReduce, Java, and Python.
• Strong programming skills in Java (especially Spark), Python, Pig, and SQL.
• Expertise in GCP and familiarity with BigTable and Redis.
• Experience with Infrastructure and DevOps principles, CI/CD, and IaC tools like Terraform.
• Proven experience in engineering large-scale batch processing systems.

Must Have (Important):

• 5+ years in customer-facing software/technology or consulting.
• 5+ years in 'on-premises to cloud' migrations or IT transformations.
• 5+ years building and operating solutions on GCP.
• Proficiency in Oozie and Pig.
• Ability to lead Jira Epics.
• Proficiency in Java or Python.

This job posting is active and accepting applications.