Big Data Engineer - Databricks & Data Pipelines

Agilisium Onshore

Westlake Village (CA)

On-site

USD 140,000 - 210,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Agilisium Onshore in Westlake Village, CA, and across the U.S. seeks a Big Data Specialist to design, build and optimize enterprise data pipelines using Databricks and PySpark. The role requires strong experience in ETL/ELT, SQL tuning, and cloud-native architectures on AWS.

You will implement data quality, governance, and a layered Medallion architecture, work with multiple analytics tools, and contribute to scalable, high-performance solutions for diverse client environments.

Qualifications

  • Bachelor's degree in CS/CE or related field with 5 years of relevant experience.
  • Strong experience building and optimizing data pipelines using PySpark and Databricks.
  • Proficiency with AWS services (S3, EC2, Kinesis, Lambda) and ETL/ELT tooling.
  • Experience with data quality, governance, and Medallion architecture preferred.

Responsibilities

  • Develop Databricks notebooks to implement Snaplogic data cleansing and processing.
  • Optimize and manage data manipulation tasks.
  • Automate data flow orchestration and pipelines.
  • Design modular pipelines to meet client performance expectations.
  • Develop and execute validation processes to ensure data integrity.
  • Establish a standardized CDL framework (Bronze, Silver, Gold) for scalability.
  • Create reusable code modules and maintain version control practices.
  • Lead data integration across heterogeneous systems.
  • Ingest Google Analytics data for actionable insights.

Skills

PySpark
Databricks
SQL optimization
ETL/ELT automation
Airflow
Data governance
Delta Lake
Dimensional modeling
Python

Education

Bachelor's degree in Computer Science, Computer Engineering, or related field

Tools

GitLab
Amazon Redshift
AWS S3
AWS EC2
AWS Kinesis
AWS Lambda
SnapLogic
Informatica PowerCenter
Delta Lake
Parquet

Job description

Agilisium Onshore in Westlake Village, CA, and across the U.S. seeks a Big Data Specialist to design, build and optimize enterprise data pipelines using Databricks and PySpark. The role requires strong experience in ETL/ELT, SQL tuning, and cloud-native architectures on AWS.

You will implement data quality, governance, and a layered Medallion architecture, work with multiple analytics tools, and contribute to scalable, high-performance solutions for diverse client environments.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Big Data Engineer: Databricks, ETL & Cloud Pipelines
Big Data Engineer: Databricks, ETL & Cloud Pipelines

Agilisium • Westlake Village (CA)

On-site
USD 150,000 - 190,000
Remote Databricks Data Engineer — Pipelines & Data Lakes
Remote Databricks Data Engineer — Pipelines & Data Lakes

Eliassen Group • Olympia (WA)

Remote
USD 150,000 - 154,000
Medical insurance
Dental insurance
Vision insurance
+4
Databricks Data Engineer — Pipelines & Big Data
Databricks Data Engineer — Pipelines & Big Data

Cyber Space Technologies LLC • Columbus (OH)

On-site
USD 90,000 - 115,000
Data Engineer - Databricks + ADF Data Pipelines
Data Engineer - Databricks + ADF Data Pipelines

Ethereum Technologies LLC • Plano (TX)

On-site
USD 118,000 - 136,000
Databricks Data Engineer - Azure, CDC Pipelines, PySpark
Databricks Data Engineer - Azure, CDC Pipelines, PySpark

CloudBigD • Phoenix (AZ)

On-site
USD 81,000 - 115,000
Data Engineer – Databricks Pipelines & GenAI (Remote)
Data Engineer – Databricks Pipelines & GenAI (Remote)

100 Primoris Services Corporation • Dallas (TX)

Remote
USD 115,000 - 120,000
401k with employer match
Health/Dental/Vision insurance
Paid time off
+2
Senior AWS Data Engineer - Databricks & DMS Pipelines
Senior AWS Data Engineer - Databricks & DMS Pipelines

CloudBigD • Wilmington (DE)

On-site
USD 120,000 - 180,000
Remote Databricks Data Engineer for Scalable Pipelines
Remote Databricks Data Engineer for Scalable Pipelines

Eliassen Group • Madison (WI)

Remote
USD 150,000 - 154,000
Comprehensive benefits
Remote work
Senior Data Engineer: Databricks & Data Pipelines Lead
Senior Data Engineer: Databricks & Data Pipelines Lead

Veriipro • Atlanta (GA)

On-site
USD 140,000 - 180,000
Remote Databricks Data Engineer for Pipelines & Lakes
Remote Databricks Data Engineer for Pipelines & Lakes

Eliassen Group • Hartford (CT)

Remote
USD 150,000 - 154,000