Senior Databricks Data Engineer: Spark, ETL & AWS

Valiant Technologies LLC; Dallas (TX); On-site; USD 120,000 - 190,000; Full time

Job summary

Valiant Technologies LLC is seeking a Senior Databricks Data Engineer to design, develop, and optimize large-scale data pipelines on the Databricks platform. You will work with PySpark, Spark SQL, and AWS to deliver enterprise analytics and regulatory reporting capabilities.

The role emphasizes building robust data ingestion, quality checks, and governance, with collaboration across stakeholders to ensure scalable, reliable data solutions for business units and compliance needs.

Qualifications

• 7+ years of hands-on Databricks data engineering experience.
• Strong PySpark, Spark SQL, and Spark API knowledge.
• Experience building ETL/ELT pipelines on cloud platforms (AWS).
• Familiarity with data governance, quality checks, and lineage.

Responsibilities

• Design and maintain scalable data pipelines on Databricks with PySpark and Spark SQL.
• Build ETL/ELT workflows linking S3, Snowflake, Airflow, and Glue.
• Transform large-scale structured and semi-structured data for enterprise reporting and compliance.
• Develop reusable data ingestion and transformation components with validation and governance checks.
• Tune Spark applications and Databricks workloads for performance and cost efficiency.
• Collaborate with stakeholders to translate data requirements into scalable solutions.
• Troubleshoot production pipelines and ensure high availability and reliability.
• Support advanced analytics, customer segmentation, and regulatory reporting initiatives.

Skills

Databricks

PySpark

Spark SQL

Apache Spark

AWS S3

Snowflake

Airflow

AWS Glue

Data modeling

Data warehousing

Structured data formats

Data quality

SQL

Regulatory reporting

CI/CD

Delta Lake

Databricks Workflows

Unity Catalog

Tools

Delta Lake

Databricks Workflows

Unity Catalog

Job description

Valiant Technologies LLC is seeking a Senior Databricks Data Engineer to design, develop, and optimize large-scale data pipelines on the Databricks platform. You will work with PySpark, Spark SQL, and AWS to deliver enterprise analytics and regulatory reporting capabilities.

The role emphasizes building robust data ingestion, quality checks, and governance, with collaboration across stakeholders to ensure scalable, reliable data solutions for business units and compliance needs.