Senior Data Engineer - Databricks & AWS Cloud Pipelines

CloudBigD

Sunnyvale (CA)

On-site

USD 76,000 - 108,000

Full time

4 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

CloudBigD in Sunnyvale, CA seeks a Senior Data Engineer with strong Databricks, Python, PySpark, SQL, and cloud data engineering experience to design scalable data pipelines and enterprise data platforms.

The ideal candidate will work with AWS services (S3, Glue, Redshift, EMR), Delta Lake, and Snowflake, building robust ETL/ELT workflows, optimizing Spark jobs, and implementing data governance and observability in modern lakehouse architectures.

Qualifications

  • 8+ years of experience in Data Engineering.
  • Strong experience with Python, PySpark, and SQL.
  • Hands-on experience with Databricks and Delta Lake.
  • Strong knowledge of ETL/ELT pipeline development.
  • Experience with AWS services such as S3, Glue, Lambda, Redshift, and EMR.
  • Experience with Snowflake or other cloud data warehouses.
  • Strong knowledge of data modeling and data warehousing.
  • Experience with Apache Spark and distributed data processing.
  • Knowledge of data integration and orchestration tools.
  • Experience with Git, Jenkins, and CI/CD pipelines.
  • Strong understanding of data quality, performance tuning, and optimization.
  • Recent / Modern Skills (2025–2026)
  • Databricks Unity Catalog and data governance.
  • Databricks Lakehouse architecture.
  • Databricks Workflows and Delta Live Tables.
  • Databricks Genie and Genie Agents.
  • Model Context Protocol (MCP) integration.
  • Conversational analytics and Text-to-SQL.
  • AI-powered data engineering and automation.
  • Vector search and RAG integration with enterprise data.
  • Data pipeline observability and monitoring.
  • Infrastructure as Code using Terraform.

Responsibilities

  • Design, develop, and maintain scalable data engineering pipelines.
  • Build ETL/ELT workflows using Python, PySpark, and SQL.
  • Develop data processing solutions using Databricks and AWS.
  • Implement data lakehouse and data warehouse architectures.
  • Optimize Spark jobs and large-scale data processing workloads.
  • Implement data quality checks and data governance practices.
  • Integrate enterprise data sources into centralized data platforms.
  • Collaborate with architects, analysts, and business stakeholders.
  • Support production deployments, troubleshooting, and performance tuning.

Skills

Python
PySpark
SQL
Databricks
Delta Lake
ETL/ELT
AWS
S3
Snowflake
Spark
CI/CD
Terraform
Data Governance

Tools

Git
Jenkins
Terraform

Job description

CloudBigD in Sunnyvale, CA seeks a Senior Data Engineer with strong Databricks, Python, PySpark, SQL, and cloud data engineering experience to design scalable data pipelines and enterprise data platforms.

The ideal candidate will work with AWS services (S3, Glue, Redshift, EMR), Delta Lake, and Snowflake, building robust ETL/ELT workflows, optimizing Spark jobs, and implementing data governance and observability in modern lakehouse architectures.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AWS Data Engineer - Databricks & DMS Pipelines
Senior AWS Data Engineer - Databricks & DMS Pipelines

CloudBigD • Wilmington (DE)

On-site
USD 120,000 - 180,000
Senior Data Engineer – Databricks & AWS Platform
Senior Data Engineer – Databricks & AWS Platform

Robotics Technologies • Sunnyvale (CA)

On-site
USD 76,000 - 108,000
Senior Data Engineer - Databricks, AWS & Lakehouse AI
Senior Data Engineer - Databricks, AWS & Lakehouse AI

Ethereum Technologies LLC • Sunnyvale (CA)

On-site
USD 76,000 - 108,000
Senior Data Engineer: Databricks Lakehouse & Pipelines
Senior Data Engineer: Databricks Lakehouse & Pipelines

Steampunk, Inc. • McLean (VA)

On-site
USD 140,000 - 180,000
Senior Data Engineer – Databricks
Senior Data Engineer – Databricks

Valiant Technologies LLC • Madison (WI)

On-site
USD 120,000 - 150,000
Senior Data Engineer
Senior Data Engineer

TechDigital Group • Chicago (IL)

On-site
USD 120,000 - 150,000
Senior Data Engineer: Databricks + AWS
Senior Data Engineer: Databricks + AWS

Robotic Process Automation LLC • Sunnyvale (CA)

On-site
USD 83,000 - 102,000
Senior Data Engineer – Databricks
Senior Data Engineer – Databricks

Quantum Technologies LLC • Madison (WI)

On-site
USD 110,000 - 140,000
Senior Data Engineer
Senior Data Engineer

Veriipro • Atlanta (GA)

On-site
USD 140,000 - 180,000
Senior Databricks & AWS Data Engineer
Senior Databricks & AWS Data Engineer

Quantum Technologies LLC • Sunnyvale (CA)

On-site
USD 76,000 - 108,000