Senior Data Engineer

Talentify; Oregon; Hybrid; USD 120,000 - 150,000; Full time

Job summary

Diverse Lynx LLC is seeking a Senior Data Engineer in Sunnyvale, CA (Remote) to design, build, and optimize data pipelines for analytics, AI/ML, and reporting. You will implement data quality controls, governance, and reliable data movement across platforms, leveraging Kafka, PySpark, Airflow, and Hive.

The role emphasizes scalable ETL/ELT development, ingestion automation, and collaboration with data teams to support advanced analytics initiatives. Remote-friendly for the US market.

Qualifications

• Experience with big data processing landscapes and building reliable data pipelines.
• Proficiency with Python-based data tooling and Spark-based processing.
• Strong knowledge of data quality, governance, and monitoring practices.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines.
• Automate data ingestion from multiple data sources.
• Develop and optimize PySpark-based data processing frameworks.
• Build and manage Airflow workflows and orchestration pipelines.
• Work with Hive and Kafka-based batch and streaming data solutions.

Skills

Kafka

Python

PySpark

Airflow

Hive

ETL testing

Tools

Google Cloud Platform

Job description

About this job

Job Title: Senior Data Engineer

Location: Sunnyvale, CA (Remote)

Duration: Fulltime

Skills: Kafka | Python | ETL Testing | Hive

Salary: 120K-150K/Year

Required Skills & Qualifications

• Google Cloud Platform (Eligible to workP)
• PySpark
• Python
• Airflow
• Hive
• Kafka
• Design, build, and maintain scalable ETL/ELT pipelines.
• Automate data ingestion from multiple data sources.
• Ensure reliable, high-performance, and scalable data movement.
• Develop and Eligible to workimize PySpark-based data processing frameworks.
• Build and manage Airflow workflows and orchestration pipelines.
• Work with Hive and Kafka-based batch and streaming data solutions.
• Implement data quality, monitoring, governance, and operational controls.Support production incidents, performance tuning, and continuous improvement initiatives
• The Data Engineer will be responsible for designing, building, governing, and Eligible to workimizing data platforms, pipelines, and integrations to ensure secure, reliable, and scalable data availability for analytics, reporting, AI/ML, and business operations.

Preferred Experience

• Strong Functional Knowledge in Retail (Search & Personalization) Added advantage.
• Strong experience with large-scale Eligible to workP Data Engineering solutions.
• Hands-on experience with Airflow DAG development and support
• Experience building and Eligible to workimizing Spark/PySpark data pipelines.
• Exposure to Hive-based data processing and Kafka-driven ingestion frameworks.
• Experience supporting analytics, personalization, recommendation, or search platforms is highly desirable

Disclaimer: Diverse Lynx LLC is an Equal Opportunity Employer. All applicants and employees are evaluated without discrimination, based solely on their qualifications, ability, competence and performance. This email and its attachments may contain confidential or proprietary information and is intended only for the recipient(s). If you received this message in error, please disregard it and notify the sender. If you no longer wish to receive our communications, you may unsubscribe at any time.

Security Notice: Our official website is www.diverselynx.com We do not operate or authorize any other websites representing Diverse Lynx LLC.