Senior AWS Databricks Lakehouse Engineer

HMG AMERICA LLC; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

HMG AMERICA LLC is seeking an experienced AWS Databricks Engineer with 7–10 years of experience to design, develop, and implement scalable data engineering solutions on AWS and Databricks. The role focuses on building robust data pipelines, ETL/ELT workflows, and optimizing data processing in enterprise environments.

The ideal candidate will have hands-on expertise in Apache Spark, PySpark, Spark SQL, Python, and SQL, with practical experience in Delta Lake and Lakehouse architectures.

Qualifications

• 7–10 years of experience in Data Engineering.
• Hands-on experience with AWS and Databricks.
• Expertise in Apache Spark, PySpark, and Spark SQL.
• Strong programming experience in Python and SQL.
• Experience with Databricks Lakehouse architecture and Delta Lake.
• Hands-on experience with AWS services such as S3, Glue, Lambda, Redshift, and IAM.
• Understanding of data warehousing, data modeling, and distributed computing.
• Experience with Git, CI/CD, and deployment automation.
• Strong troubleshooting and performance optimization skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using AWS and Databricks.
• Build and optimize ETL/ELT workflows using Databricks, PySpark, and SQL.
• Develop data processing solutions using Apache Spark and Delta Lake.
• Implement and manage Databricks Lakehouse architecture on AWS.
• Work with AWS services such as S3, Glue, Lambda, Redshift, and IAM.
• Develop data ingestion and transformation pipelines from multiple data sources.
• Implement data quality, data validation, and monitoring processes.
• Optimize Spark jobs, SQL queries, and Databricks workloads for performance and cost efficiency.
• Work with Delta Lake features, including schema evolution, time travel, and ACID transactions.
• Collaborate with data architects, analysts, and cross-functional engineering teams.
• Implement CI/CD pipelines and follow data engineering best practices.

Skills

AWS

Databricks

Apache Spark

PySpark

SQL

Python

Data Pipelines

ETL/ELT

CI/CD

Delta Lake

Tools

Git

CI/CD

Terraform

Databricks Workflows

Unity Catalog

Job description

HMG AMERICA LLC is seeking an experienced AWS Databricks Engineer with 7–10 years of experience to design, develop, and implement scalable data engineering solutions on AWS and Databricks. The role focuses on building robust data pipelines, ETL/ELT workflows, and optimizing data processing in enterprise environments.

The ideal candidate will have hands-on expertise in Apache Spark, PySpark, Spark SQL, Python, and SQL, with practical experience in Delta Lake and Lakehouse architectures.