Remote Data Engineer: Spark, Python & GCP Data Pipelines

Shell Infotech; United States; Remote; USD 110,000 - 140,000; Full time

Job summary

Shell Infotech is seeking a Data Engineer to design and implement scalable data pipelines and data platforms in a remote setting. The role focuses on building robust data frameworks, APIs, and data products to deliver actionable insights across the business.

You will drive PoC/POT evaluations, monitor production health, and align data solutions with stakeholder reporting needs while continuously refining data ingestion and processing patterns.

Qualifications

• Bachelor's degree in Computer Science or a related field.
• 5+ years of hands-on experience with Scala/Python (PySpark) and distributed databases.
• Strong understanding of CS fundamentals and data structures.
• Hands-on SQL and query optimization experience.
• Experience building data ingestion frameworks and patterns.
• Experience with GCP data processing tools and BigQuery.
• Familiarity with streaming data via Kafka or structured streaming.

Responsibilities

• Develop data pipelines and data frameworks, APIs, and data products.
• Lead PoC/PoT evaluations.
• Build data health monitoring and quality processes for large data systems.
• Review KPIs, monitoring, and alarms for production systems.
• Collaborate with stakeholders to design data solutions that provide insights and automate processes.
• Educate the team and share knowledge.
• Keep up-to-date with tools and document architecture.

Skills

Spark

Scala

Python

Big Data

Scripting

GCP Data Tech Stack

Airflow

CI/CD

Kafka

SQL/NoSQL

Education

Bachelor's degree in Computer Science or related field

Tools

Airflow

Kafka

GCP

Job description

Shell Infotech is seeking a Data Engineer to design and implement scalable data pipelines and data platforms in a remote setting. The role focuses on building robust data frameworks, APIs, and data products to deliver actionable insights across the business.

You will drive PoC/POT evaluations, monitor production health, and align data solutions with stakeholder reporting needs while continuously refining data ingestion and processing patterns.