Senior Databricks Engineer – Lakehouse Pipelines & Spark

Valiant Technologies LLC; Austin (TX); On-site; USD 130,000 - 190,000; Full time

Job summary

Valiant Technologies LLC in Austin, TX is seeking a Databricks Engineer to design, develop, and optimize scalable data engineering solutions on the Databricks Lakehouse Platform. This role focuses on building robust data pipelines, processing large datasets, implementing data transformation workflows, and supporting enterprise analytics initiatives.

You will work with Python, PySpark, SQL, Spark SQL, and Delta Lake; manage Delta Live Tables, Workflows, Unity Catalog, and Databricks SQL;

Qualifications

• 5+ years of experience in data engineering or big data development.
• 3+ years of hands-on experience with Databricks and Apache Spark.
• Strong programming experience with Python, PySpark, and SQL.
• Understanding of distributed computing, data architecture, and performance tuning.

Responsibilities

• Design, develop, and optimize scalable data pipelines on Databricks.
• Build ETL/ELT workflows for structured and unstructured data.
• Implement data quality checks, monitoring, and error handling.
• Collaborate with data architects, analysts, and business teams to deliver analytics solutions.

Skills

Databricks

Python

PySpark

SQL

Spark SQL

Delta Lake

ETL/ELT

Cloud Databricks

Data modeling

Data governance

Airflow

Git / CI/CD

Education

Bachelor's degree in CS/IT

Master's degree preferred

Databricks certifications (Data Engineer)

Tools

Databricks

Apache Spark

Airflow

Azure Data Factory

Unity Catalog

Delta Lake

Git

CI/CD tools

Cloud storage (ADLS,S3)

Job description

Valiant Technologies LLC in Austin, TX is seeking a Databricks Engineer to design, develop, and optimize scalable data engineering solutions on the Databricks Lakehouse Platform. This role focuses on building robust data pipelines, processing large datasets, implementing data transformation workflows, and supporting enterprise analytics initiatives.

You will work with Python, PySpark, SQL, Spark SQL, and Delta Lake; manage Delta Live Tables, Workflows, Unity Catalog, and Databricks SQL;