Senior Databricks Engineer - Cloud Data Platform Leader

Valiant Technologies LLC; Pennsylvania; On-site; USD 92,000 - 131,000; Full time

Job summary

Valiant Technologies LLC is seeking a Senior Databricks Developer in Malvern, PA to lead data engineering initiatives and build scalable data pipelines on AWS using Delta Lake and Medallion architecture. 7+ years of experience required; this is a hands-on, production-focused role.

The position supports enterprise data platforms, with duties spanning design, development, optimization, and operational support, continuing for a 12-month contract with potential extension.

Qualifications

• 7+ years of data engineering building enterprise-scale data platforms.
• 5+ years of hands-on Databricks development in production.
• Strong Python, PySpark, and SQL for pipelines.

Responsibilities

• Design, develop, and maintain large-scale ETL/ELT pipelines with Databricks.
• Build scalable data ingestion, transformation, and orchestration in AWS.
• Develop Delta Lake implementations and Medallion data pipelines.
• Create and optimize batch and streaming pipelines for large data volumes.
• Leverage S3, Glue, Lambda, EMR, IAM, EventBridge, CloudWatch, Kinesis, Redshift.
• Optimize Spark, Delta tables, and Databricks workloads for performance and cost.
• Collaborate with engineers, architects, analytics teams, and stakeholders.
• Implement data quality controls, monitoring, and operations.
• Troubleshoot production issues across Databricks and AWS.
• Contribute to best practices, reviews, automation, and modernization.

Skills

Data engineering

Databricks

Python

PySpark

SQL

AWS

Delta Lake

Medallion Architecture

Tools

Spark

Delta Lake

Job description

Valiant Technologies LLC is seeking a Senior Databricks Developer in Malvern, PA to lead data engineering initiatives and build scalable data pipelines on AWS using Delta Lake and Medallion architecture. 7+ years of experience required; this is a hands-on, production-focused role.

The position supports enterprise data platforms, with duties spanning design, development, optimization, and operational support, continuing for a 12-month contract with potential extension.