Lead Data Engineer - Databricks & Production Platforms

United States Digital Space LLC; United States; Remote; USD 175,000 - 230,000; Full time

Job summary

United States Digital Space LLC is seeking a Lead Data Engineer to serve as the hands-on technical leader on client-facing data engagements. You will own the architecture, build scalable data pipelines, and drive productionized data solutions across cloud platforms.

You will guide projects, mentor engineers, and promote DataOps practices to ensure reliable, production-ready data platforms. This role requires deep Spark and Delta Lake expertise and strong communication with stakeholders.

Qualifications

• 8+ years hands-on data engineering experience delivering production-grade platforms.
• Expert-level in Apache Spark with deep understanding of runtime internals and performance tuning.
• Python production-quality coding; Scala experience is a strong plus for Spark.
• Built and productionized Delta Lake architectures, Unity Catalog governance, and Databricks Workflows; Databricks certification is a plus.
• Experience across at least two major cloud platforms (AWS, Azure, GCP) with depth in at least one (e.g., Redshift/Glue/S3; Synapse/Data Factory/ADLS; BigQuery/Dataflow/GCS).
• Led data engineering projects end-to-end in client-facing/consulting contexts; managed scope and timelines.
• DataOps mindset: CI/CD for data pipelines, automated testing, observability, infrastructure-as-code.
• Experience across ETL/ELT design, data warehousing, lakehouse, and data modeling.
• Strong communication with both technical and non-technical stakeholders.

Responsibilities

• Lead Client Data Engagements: senior technical lead on client projects; own architecture and delivery.
• Build and Productionize Data Solutions: design scalable pipelines and lakehouse architectures on Databricks and cloud platforms; write and review code.
• Architect for Scale and Reliability: translate requirements into robust technical designs and data models for production.
• Drive Technical Delivery: manage scope and timelines; identify blockers and coordinate with PMs and stakeholders to stay on track.
• Mentor Data Engineers: pair programming, code reviews, and feedback to raise the team's skills.
• Promote Knowledge Sharing: contribute blogs, reference architectures, and internal guides from real client work.
• Champion DataOps Practices: enforce CI/CD, testing, observability, version control, and documentation across engagements.

Skills

Data engineering

Spark

Python

Scala

Databricks

Cloud platforms

DataOps

ETL/ELT

Communication

Tools

Databricks

Delta Lake

Unity Catalog

Databricks Workflows

AWS

Azure

GCP

BigQuery

Job description

United States Digital Space LLC is seeking a Lead Data Engineer to serve as the hands-on technical leader on client-facing data engagements. You will own the architecture, build scalable data pipelines, and drive productionized data solutions across cloud platforms.

You will guide projects, mentor engineers, and promote DataOps practices to ensure reliable, production-ready data platforms. This role requires deep Spark and Delta Lake expertise and strong communication with stakeholders.