Senior Data Engineer - Databricks Tech Lead

V4C.ai; United States; Hybrid; USD 150,000 - 200,000; Full time

Benefits offered by this job

Unlimited PTO

Life Insurance

AD&D Insurance

Job summary

V4C.ai is seeking a senior Lead Data Engineer in the United States to guide production Lakehouse pipelines on client engagements. You will translate architectures into robust data flows, own design patterns, and drive quality in batch and streaming workloads with Databricks.

Expect to mentor engineers, collaborate with data architects, and communicate trade-offs to stakeholders while navigating a fast-paced consulting environment and travel as needed.

Qualifications

• 7+ years of data engineering and project/team leadership experience.
• Proficiency in Python, SQL, Spark, and Delta Lake.
• Production-grade batch and streaming pipelines on cloud platforms.

Responsibilities

• Lead engineering on customer engagements from design to production handover.
• Build and optimize batch and streaming pipelines with PySpark, SQL, and Delta Lake on Databricks.
• Implement Medallion architecture and set team patterns/standards.
• Own performance and cost: benchmark Spark jobs, tune SQL, and manage compute.
• Mentor mid-level and junior engineers through code review and design feedback.
• Collaborate with Data Architect on design decisions and client-facing explanations.

Skills

Python

SQL

Apache Spark

Delta Lake

Databricks

Leadership

Mentorship

Client communication

Tools

Unity Catalog

Lakeflow

Databricks Asset Bundles

dbt

Airflow

Job description

V4C.ai is seeking a senior Lead Data Engineer in the United States to guide production Lakehouse pipelines on client engagements. You will translate architectures into robust data flows, own design patterns, and drive quality in batch and streaming workloads with Databricks.

Expect to mentor engineers, collaborate with data architects, and communicate trade-offs to stakeholders while navigating a fast-paced consulting environment and travel as needed.