Senior Data Engineer — Databricks & Data Pipelines

VSG Business Solutions LLC; Glendale (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

VSG Business Solutions LLC in Glendale, CA is seeking a seasoned Data Engineer to design, develop, and operate large-scale data pipelines. You will work with PySpark, Scala, SQL, and Python to deliver scalable data platform solutions for stakeholders.

You will contribute to Databricks governance, orchestration with Airflow (MWAA), and cloud-native data services across AWS and Kubernetes. A collaborative, Agile environment offers strong growth opportunities.

Qualifications

• 5+ years of data engineering experience building and operating large-scale pipelines.
• Deep hands-on Databricks and Spark (batch and streaming) experience.
• Strong understanding of Spark architecture and performance tuning.
• Proficient with Databricks tooling (API/SDK/CLI) for automation and governance.
• Advanced SQL with performance tuning and optimization.
• Production experience with Airflow (MWAA) for orchestration.
• Experience governing Databricks: Unity Catalog, ACLs, lineage, data privacy.
• Proficiency in Python and at least one other language (Scala, Kotlin, or SQL).
• Experience designing scalable ETL/ELT pipelines across diverse data sources.
• AWS experience; cloud and containerized services (Docker/Kubernetes).
• CI/CD and DevOps practices; data quality, monitoring, and logging.
• Self-starter with strong communication and Scrum/Agile familiarity.
• Snowflake familiarity is a plus.

Responsibilities

• Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, Python.
• Meet with stakeholders to translate requirements into scalable data platform solutions.
• Understand Databricks tooling to diagnose errors and automate updates across pipelines.
• Explain Spark architecture and pipeline behavior to stakeholders for root-cause analysis.
• Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA).
• Manage Databricks governance: Unity Catalog, ACLs, lineage, data privacy tooling.
• Build and maintain Kubernetes containers and containerized utilities for data services.
• Troubleshoot connectivity and integration issues across platform components.
• Provision/remove access, monitor resources, and assess stakeholder requests.
• Collaborate across engineers, architects, and product managers in agile ceremonies.
• Document platform changes, standards, and pipeline configurations for data quality.

Skills

Databricks experience

Apache Airflow

Advanced SQL

Python

Spark / PySpark

Scala

Experience building and maintaining d

Education

Bachelor's Degree in Computer Science, Information Systems, or a related field, or equivalent work experience

Tools

Databricks

Apache Airflow

Spark

PySpark

SQL

Kubernetes

Docker

Job description

VSG Business Solutions LLC in Glendale, CA is seeking a seasoned Data Engineer to design, develop, and operate large-scale data pipelines. You will work with PySpark, Scala, SQL, and Python to deliver scalable data platform solutions for stakeholders.

You will contribute to Databricks governance, orchestration with Airflow (MWAA), and cloud-native data services across AWS and Kubernetes. A collaborative, Agile environment offers strong growth opportunities.