Big Data Specialist (Multiple Positions) | ZR_925_JOB

Agilisium Onshore

Westlake Village (CA)

On-site

USD 140,000 - 210,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Agilisium Onshore in Westlake Village, CA, and across the U.S. seeks a Big Data Specialist to design, build and optimize enterprise data pipelines using Databricks and PySpark. The role requires strong experience in ETL/ELT, SQL tuning, and cloud-native architectures on AWS.

You will implement data quality, governance, and a layered Medallion architecture, work with multiple analytics tools, and contribute to scalable, high-performance solutions for diverse client environments.

Qualifications

  • Bachelor's degree in CS/CE or related field with 5 years of relevant experience.
  • Strong experience building and optimizing data pipelines using PySpark and Databricks.
  • Proficiency with AWS services (S3, EC2, Kinesis, Lambda) and ETL/ELT tooling.
  • Experience with data quality, governance, and Medallion architecture preferred.

Responsibilities

  • Develop Databricks notebooks to implement Snaplogic data cleansing and processing.
  • Optimize and manage data manipulation tasks.
  • Automate data flow orchestration and pipelines.
  • Design modular pipelines to meet client performance expectations.
  • Develop and execute validation processes to ensure data integrity.
  • Establish a standardized CDL framework (Bronze, Silver, Gold) for scalability.
  • Create reusable code modules and maintain version control practices.
  • Lead data integration across heterogeneous systems.
  • Ingest Google Analytics data for actionable insights.

Skills

PySpark
Databricks
SQL optimization
ETL/ELT automation
Airflow
Data governance
Delta Lake
Dimensional modeling
Python

Education

Bachelor's degree in Computer Science, Computer Engineering, or related field

Tools

GitLab
Amazon Redshift
AWS S3
AWS EC2
AWS Kinesis
AWS Lambda
SnapLogic
Informatica PowerCenter
Delta Lake
Parquet

Job description

Big Data Specialist (Multiple Positions) | ZR_925_JOB

Westlake Village, United States | Posted on 11/04/2026

TITLE: Big Data Specialist (Multiple Positions)

LOCATION: Westlake Village, CA and various and unanticipated locations throughout the U.S. (Must be willing to work anywhere in the U.S. as the position may involve relocation to various and unanticipated client site locations; any relocation to be paid by employer pursuant to internal policy.)

Duties
  • Develop Databricks notebooks to implement Snaplogic pipeline data cleansing and processing.
  • Optimize and manage data manipulation tasks.
  • Implement and automate data flow orchestration.
  • Design and implement modular, standardized pipelines to meet client performance expectations.
  • Develop and execute comprehensive validation processes to ensure data integrity throughout the pipeline.
  • Establish a standardized framework for multiple CDL layers, ensuring scalability and consistency.
  • Create reusable code modules and maintain version control practices.
  • Lead data integration, to drive system improvements.
  • Develop automated processes for ingesting Google Analytics data, enabling actionable insights.
  • EOE
Requirements
  • Must have a Bachelor’s degree or foreign equivalent in Computer Science, Computer Engineering, or a related field plus five (5) years of experience in the position offered, as a Software Engineer, or a related position.
  • Must have five (5) years of experience with all of the following: Developing and optimizing data pipelines in PySpark and Databricks for large-scale data ingestion and processing, integrating GitLab version control for code management and automated deployments; Implementing advanced SQL development and optimization techniques for complex data modeling, query performance tuning, and relational warehouse management in Amazon Redshift and SQL DW; Engineering and automating ETL/ELT workflows leveraging Apache Airflow, SnapLogic, and Informatica PowerCenter, ensuring reliability, reusability, and end-to-end data integration across heterogeneous systems; Architecting and operating cloud-native big-data ecosystems on AWS, integrating services including AWS S3 for storage, AWS EC2 for compute, AWS Kinesis for real-time data streaming, and AWS Lambda for event-driven processing to enable low-latency ingestion and analytics pipelines; Designing and implementing data quality and governance frameworks within the Medallion Architecture (Bronze, Silver, and Gold layers) to ensure data reliability, lineage, and compliance with enterprise governance standards; Implementing Delta Lake optimization strategies, including Z-Ordering, OPTIMIZE, and VACUUM, to improve query performance, data compaction, and storage efficiency, while designing partition pruning and columnar data formats (Parquet) to minimize I/O overhead and accelerate analytical workloads; and Architecting and optimizing dimensional and relational data models using Star and Snowflake schemas within Databricks and Amazon Redshift, ensuring scalable performance, schema consistency, and efficient analytical processing across large datasets.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Big Data Specialist (Multiple Positions) | ZR_925_JOB
Big Data Specialist (Multiple Positions) | ZR_925_JOB

Agilisium • Westlake Village (CA)

On-site
USD 150,000 - 190,000
Big Data Specialist
Big Data Specialist

Hyrhub • New York (NY)

On-site
USD 150,000 - 180,000
Databricks Data Engineer
Databricks Data Engineer

Compunnel Inc. • Spring (TX)

On-site
USD 110,000 - 140,000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Ethereum Technologies LLC • Sunnyvale (CA)

On-site
USD 76,000 - 108,000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Valiant Technologies LLC • Sunnyvale (CA)

On-site
USD 83,000 - 103,000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Quantum Technologies LLC • Sunnyvale (CA)

On-site
USD 76,000 - 108,000
Databricks Data Engineer
Databricks Data Engineer

Volto Consulting • Irving (TX)

On-site
USD 120,000 - 150,000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Robotic Process Automation LLC • Sunnyvale (CA)

On-site
USD 83,000 - 102,000
Databricks Data Engineer
Databricks Data Engineer

Valiant Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 190,000
Databricks Data Engineer
Databricks Data Engineer

Robotic Process Automation LLC • Dallas (TX)

On-site
USD 140,000 - 180,000