Data Engineer

SFE; Sunnyvale (CA); Remote; USD 120,000 - 150,000; Full time

Job summary

SFE in Sunnyvale, CA is seeking a Senior Data Engineer to design, build, govern, and optimize data platforms, pipelines, and integrations ensuring secure, reliable, and scalable data for analytics, reporting, AI/ML, and business operations.

You will work with PySpark, Airflow, Hive, and Kafka to implement robust data solutions, support production incidents, and drive continuous improvement in data quality and governance.

Qualifications

• Proficient in designing, building, governing data platforms and pipelines.
• Hands-on experience with PySpark, Airflow, Hive, Kafka.
• Experience with large-scale GCP data engineering solutions.
• Strong ETL/ELT pipeline development experience.
• Knowledge of data quality, governance, and operational controls.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines.
• Automate data ingestion from multiple sources.
• Ensure reliable, high-performance data movement.
• Develop and optimize PySpark data processing frameworks.
• Build and manage Airflow workflows and orchestration pipelines.
• Work with Hive and Kafka-based batch and streaming data solutions.
• Implement data quality, monitoring, governance, and operational controls.
• Support production incidents, performance tuning, and continuous improvement initiatives

Skills

Retail domain knowledge

Analytics experience

Tools

GCP Data Engineering

PySpark

Python

Airflow

Hive

Kafka

Spark

Job description

Section

Details

Job Title

Senior Data Engineer

Job Type

Full Time

Client Location

Sunnyvale, CA

Work Arrangement

Remote

Duration

Full Time

Pay Rate / Salary

$120-$150k/yr.

Job Summary

• The Data Engineer will be responsible for designing, building, governing, and optimizing data platforms, pipelines, and integrations to ensure secure, reliable, and scalable data availability for analytics, reporting, AI/ML, and business operations.

Key Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines.
• Automate data ingestion from multiple data sources.
• Ensure reliable, high-performance, and scalable data movement.
• Develop and optimize PySpark-based data processing frameworks.
• Build and manage Airflow workflows and orchestration pipelines.
• Work with Hive and Kafka-based batch and streaming data solutions.
• Implement data quality, monitoring, governance, and operational controls.
• Support production incidents, performance tuning, and continuous improvement initiatives

Required Qualifications

• Strong Functional Knowledge in Retail (Search & Personalization) Added advantage.
• Strong experience with large-scale GCP Data Engineering solutions.
• Hands-on experience with Airflow DAG development and support
• Experience building and optimizing Spark/PySpark data pipelines.
• Exposure to Hive-based data processing and Kafka-driven ingestion frameworks.
• Experience supporting analytics, personalization, recommendation, or search platforms is highly desirable
• Google Cloud Platform (GCP)
• PySpark
• Python
• Airflow
• Hive
• Kafka