Senior Data Engineer - Real-Time Pipelines (Spark/Kafka)

UST; Sunnyvale (CA); On-site; USD 72,000 - 108,000; Full time

Benefits offered by this job

Medical Insurance

401(k) with employer match

Paid time off

Job summary

UST is seeking a Data Engineer to design and maintain real-time data pipelines using Spark, Kafka, and cloud services (GCP). The role focuses on ingesting, transforming, and delivering data for analytics and ML, with emphasis on scalable architectures and data quality.

You will work across global teams to optimize pipelines, leverage BigQuery, Airflow, and Databricks, and ensure data integrity and governance in a fast-paced environment.

Qualifications

• Bachelor's degree in computer science or equivalent.
• 7–10 years of experience in data engineering.
• Strong Python, SQL, Spark (PySpark/Scala) and streaming experience.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Spark and streaming tech.
• Build scalable data architectures, data lakes, warehouses (BigQuery) and streaming platforms.
• Tune performance of Spark jobs, SQL queries, and data workflows for speed and cost.
• Implement data quality checks, monitoring, and governance.
• Collaborate with stakeholders to deliver analytics-ready datasets.
• Monitor and optimize data pipelines and database performance.

Skills

Python

SQL

Spark/Java/PySpark

Kafka

Airflow

Cloud (GCP/AWS/Azure)

Big Data

Data Modeling

Education

Bachelor's degree in Computer Science

Tools

Databricks

Docker

Kubernetes

BigQuery

Snowflake/Redshift

Job description

UST is seeking a Data Engineer to design and maintain real-time data pipelines using Spark, Kafka, and cloud services (GCP). The role focuses on ingesting, transforming, and delivering data for analytics and ML, with emphasis on scalable architectures and data quality.

You will work across global teams to optimize pipelines, leverage BigQuery, Airflow, and Databricks, and ensure data integrity and governance in a fast-paced environment.