Data Engineer: Build Scalable, Config-Driven Pipelines

AspiringIT; Irvine (CA); On-site; USD 110,000 - 160,000; Full time

Job summary

AspiringIT in Irvine is seeking an experienced Data Engineer to design, build, and operate the pipelines powering our Prism platform on Databricks. You will move beyond bespoke scripts, enabling new sources via configuration files and scalable metadata-driven pipelines.

You will implement dbt transformations with testing and documentation, ensure governance with Unity Catalog, and build CI/CD workflows for high data quality and reliability.

Qualifications

• 3+ years of Data Engineering in production environments.
• Configuration-driven or metadata-driven data pipelines.
• Hands-on dbt development, testing, and docs.
• Strong SQL and Python skills.
• Experience with Databricks or Snowflake.
• Knowledge of orchestration tools (Airflow, Databricks Workflows, dbt Cloud).

Responsibilities

• Design and develop scalable, metadata-driven data pipelines.
• Build ingestion frameworks for batch, CDC, streaming, API and file sources.
• Develop dbt transformations with testing and documentation.
• Collaborate with data architects to translate requirements into data solutions.
• Implement governance and lineage using Unity Catalog.
• Build CI/CD pipelines and ensure data quality and observability.
• Troubleshoot and optimize end-to-end data workflows.

Skills

Data pipelines

dbt

SQL

Python

Databricks

Airflow

Cloud platforms

Tools

Unity Catalog

Terraform

Git

Kafka

dbt Cloud

Job description

AspiringIT in Irvine is seeking an experienced Data Engineer to design, build, and operate the pipelines powering our Prism platform on Databricks. You will move beyond bespoke scripts, enabling new sources via configuration files and scalable metadata-driven pipelines.

You will implement dbt transformations with testing and documentation, ensure governance with Unity Catalog, and build CI/CD workflows for high data quality and reliability.