Senior Data Engineer - Databricks on AWS

Valiant Technologies LLC; Tampa (FL); On-site; USD 89,000 - 126,000; Full time

Job summary

Valiant Technologies LLC is seeking a senior Data Engineer in Tampa, FL to architect and build scalable data pipelines on the Databricks lakehouse platform. You will leverage Spark, Delta Lake and SQL to model data, design secure multi-account AWS solutions, and plan migrations from legacy Hadoop/Hive systems.

Responsibilities include choosing batch vs. streaming, ensuring disaster recovery and high availability, and producing clear architecture documentation for stakeholders.

Qualifications

• 12+ years in data engineering or architecture with 5+ years on Databricks lakehouse.
• Strong Spark, Delta Lake and SQL skills with data modeling knowledge.
• AWS networking, IAM, encryption and multi-account design.
• Secure design for PII/NPI with Unity Catalog access models and lineage.
• Migration planning from Hadoop/Hive/warehouse environments.
• Ability to design for both batch and streaming (Kafka, Structured Streaming).
• Disaster recovery and high-availability design know-how.
• FinOps and Databricks cost and capacity modeling with business case.

Responsibilities

• Lead data engineering architecture and implementation on Databricks lakehouse.
• Design secure, scalable data pipelines with batch and streaming support.
• Model data using medallion architecture and ensure data quality.
• Collaborate with stakeholders; manage vendors and document architecture.

Skills

Databricks lakehouse

Apache Spark

SQL

Data modeling (medallion)

AWS architecture

Security design (PII/NPI)

Streaming (Kafka)

Disaster recovery

Cost modeling (FinOps)

Stakeholder communication

Tools

Databricks lakehouse

Spark

SQL

Job description

Valiant Technologies LLC is seeking a senior Data Engineer in Tampa, FL to architect and build scalable data pipelines on the Databricks lakehouse platform. You will leverage Spark, Delta Lake and SQL to model data, design secure multi-account AWS solutions, and plan migrations from legacy Hadoop/Hive systems.

Responsibilities include choosing batch vs. streaming, ensuring disaster recovery and high availability, and producing clear architecture documentation for stakeholders.