Lead Data Engineer (Databricks)

HMG AMERICA LLC; United States; Remote; USD 140,000 - 180,000; Full time

Job summary

HMG AMERICA LLC is seeking a Lead Data Engineer (Databricks) to architect and deliver enterprise-scale data platforms on AWS using Databricks. You will guide architecture, governance, and cost optimization while engaging client stakeholders as a trusted advisor.

The role requires hands-on Databricks expertise, Delta Live Tables, Unity Catalog, and a strong background in IT consulting and modern data lakehouse patterns.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related field.
• 8+ years in Data Engineering with at least 3+ years in a Lead/Architect capacity.
• Deep experience with Databricks, PySpark/Scala, Delta Lake, Unity Catalog, and Delta Live Tables (DLT).
• Hands-on experience with AWS Cloud Data Ecosystem (S3, Glue, Lambda, EMR, Redshift, IAM).
• Proficiency in Python, SQL, and Spark SQL performance tuning.
• Strong experience with Infrastructure-as-Code (Terraform) and CI/CD tools (GitHub Actions, Azure DevOps, Jenkins).
• Consulting & Soft Skills: Exceptional verbal and written communication; client-facing advisory experience.

Responsibilities

• Architect, design, and implement scalable data platforms on AWS using Databricks Lakehouse architecture.
• Establish Bronze-Silver-Gold patterns for real-time and batch pipelines.
• Implement data governance, access control, and lineage using Unity Catalog and IAM.
• Optimize cluster configurations, Delta Lake performance, and costs.
• Integrate AWS services with Databricks and manage multi-environment CI/CD pipelines.

Skills

Databricks

PySpark/Scala

Delta Lake

Unity Catalog

Delta Live Tables

AWS

Python

SQL

Spark SQL

Consulting

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Terraform

GitHub Actions

Azure DevOps

Jenkins

Job description

Job Title: Lead Data Engineer (Databricks)

Location: [Remote]

Job Summary

We are seeking an experienced Lead Data Engineer (Databricks) with a strong background in IT consulting and professional services to lead end-to-end cloud data enterprise initiatives. In this role, you will act as a trusted technical advisor to client stakeholders while leading the architecture, design, and execution of scalable Lakehouse platforms on AWS using Databricks.

The ideal candidate possesses exceptional client engagement abilities, deep hands-on expertise with the Databricks Medallion Architecture, Unity Catalog, and Delta Live Tables (DLT), along with a proven track record of migrating legacy data warehouses to modern AWS data ecosystems.

Key Responsibilities

1. Solution Architecture & Technical Leadership

• Lakehouse Architecture: Architect, design, and implement scalable, enterprise-grade data platforms on AWS using the Databricks Lakehouse architecture.
• Medallion Architecture: Establish robust data architecture patterns across Bronze, Silver, and Gold layers for both real-time streaming and batch processing pipelines.
• Data Governance & Security: Implement fine-grained access control, governance, data lineage, and audit trails using Databricks Unity Catalog, AWS IAM, and KMS.
• Performance & Cost Optimization: Optimize cluster configurations, auto-scaling parameters, and Delta Lake performance techniques (Z-Ordering, Liquid Clustering, compaction).
• Ecosystem Integration: Seamlessly integrate AWS native services (S3, AWS Glue, Lambda, EventBridge, Redshift, Athena, Secrets Manager) with Databricks.

2. Client Engagement & Strategic Advisory (Consulting)

• Stakeholder Management: Lead discovery workshops, technical assessments, and architecture reviews with C-level executives, VPs of Engineering/Data, and Enterprise Architects.
• Technical Roadmaps: Translate complex business requirements into solution blueprints, high-level architecture diagrams, technical roadmaps, and Statements of Work (SOWs).
• Legacy Modernization: Drive client migration strategies from legacy data warehouses (e.g., Teradata, Netezza, Snowflake, On-Prem Hadoop) to AWS Databricks.
• Best Practices Guidance: Educate clients on cloud data best practices, FinOps/cost optimization strategies, and modern Lakehouse paradigms.
• Technical Mentorship: Guide, direct, and mentor cross-functional teams of Data Engineers, Analytics Engineers, and DevOps Engineers.
• Infrastructure as Code (IaC): Enforce IaC best practices using Terraform or CloudFormation for automated enterprise workspace and cluster deployments.
• DevOps & Automation: Build and maintain CI/CD pipelines for Databricks notebooks, Delta Live Tables (DLT), and Spark jobs across multi-environment setups (Dev, Test, Prod).

Qualifications & Requirements

• Education: Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related field.
• 8+ years in Data Engineering with at least 3+ years in a Lead/Architect capacity.
• Deep experience with Databricks, PySpark/Scala, Delta Lake, Unity Catalog, and Delta Live Tables (DLT).
• Hands-on experience with AWS Cloud Data Ecosystem (S3, Glue, Lambda, EMR, Redshift, IAM).
• Proficiency in Python, SQL, and Spark SQL performance tuning.
• Strong experience with Infrastructure-as-Code (Terraform) and CI/CD tools (GitHub Actions, Azure DevOps, Jenkins).
• Consulting & Soft Skills:
• Exceptional verbal and written communication skills with the ability to articulate complex technical ideas to executive audiences.
• Strong problem-solving skills and client-facing advisory experience.