AWS Databricks Engineer

HMG AMERICA LLC; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

HMG AMERICA LLC is seeking an experienced AWS Databricks Engineer with 7–10 years of experience to design, develop, and implement scalable data engineering solutions on AWS and Databricks. The role focuses on building robust data pipelines, ETL/ELT workflows, and optimizing data processing in enterprise environments.

The ideal candidate will have hands-on expertise in Apache Spark, PySpark, Spark SQL, Python, and SQL, with practical experience in Delta Lake and Lakehouse architectures.

Qualifications

• 7–10 years of experience in Data Engineering.
• Hands-on experience with AWS and Databricks.
• Expertise in Apache Spark, PySpark, and Spark SQL.
• Strong programming experience in Python and SQL.
• Experience with Databricks Lakehouse architecture and Delta Lake.
• Hands-on experience with AWS services such as S3, Glue, Lambda, Redshift, and IAM.
• Understanding of data warehousing, data modeling, and distributed computing.
• Experience with Git, CI/CD, and deployment automation.
• Strong troubleshooting and performance optimization skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using AWS and Databricks.
• Build and optimize ETL/ELT workflows using Databricks, PySpark, and SQL.
• Develop data processing solutions using Apache Spark and Delta Lake.
• Implement and manage Databricks Lakehouse architecture on AWS.
• Work with AWS services such as S3, Glue, Lambda, Redshift, and IAM.
• Develop data ingestion and transformation pipelines from multiple data sources.
• Implement data quality, data validation, and monitoring processes.
• Optimize Spark jobs, SQL queries, and Databricks workloads for performance and cost efficiency.
• Work with Delta Lake features, including schema evolution, time travel, and ACID transactions.
• Collaborate with data architects, analysts, and cross-functional engineering teams.
• Implement CI/CD pipelines and follow data engineering best practices.

Skills

AWS

Databricks

Apache Spark

PySpark

SQL

Python

Data Pipelines

ETL/ELT

CI/CD

Delta Lake

Tools

Git

CI/CD

Terraform

Databricks Workflows

Unity Catalog

Job description

We are looking for an experienced AWS Databricks Engineer with 7–10 years of experience in designing, developing, and implementing scalable data engineering solutions using AWS and Databricks. The ideal candidate should have strong hands-on experience in building data pipelines, developing ETL/ELT workflows, optimizing data processing, and managing cloud-based data platforms.

The candidate should have a solid understanding of AWS services, Databricks Lakehouse architecture, Apache Spark, and Python/SQL, along with experience working in enterprise data environments.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using AWS and Databricks.
• Build and optimize ETL/ELT workflows using Databricks, PySpark, and SQL.
• Develop data processing solutions using Apache Spark and Delta Lake.
• Implement and manage Databricks Lakehouse architecture on AWS.
• Work with AWS services such as S3, Glue, Lambda, Redshift, and IAM.
• Develop data ingestion and transformation pipelines from multiple data sources.
• Implement data quality, data validation, and monitoring processes.
• Optimize Spark jobs, SQL queries, and Databricks workloads for performance and cost efficiency.
• Work with Delta Lake features, including schema evolution, time travel, and ACID transactions.
• Collaborate with data architects, analysts, and cross-functional engineering teams.
• Implement CI/CD pipelines and follow data engineering best practices.

Required Technical Skills

• 7–10 years of experience in Data Engineering.
• Strong hands-on experience with AWS and Databricks.
• Expertise in Apache Spark, PySpark, and Spark SQL.
• Strong programming experience in Python and SQL.
• Experience with Databricks Lakehouse architecture and Delta Lake.
• Hands-on experience with AWS services such as:
• Amazon S3
• AWS Glue
• AWS Lambda
• Amazon Redshift
• AWS IAM
• Understanding of data warehousing, data modeling, and distributed computing.
• Experience with Git, CI/CD, and deployment automation.
• Strong troubleshooting and performance optimization skills.

Preferred Qualifications

• Experience with Databricks Workflows and Unity Catalog.
• Knowledge of Delta Live Tables / Lakeflow Declarative Pipelines.
• Experience with Terraform or Infrastructure as Code.
• Familiarity with Apache Kafka or other streaming technologies.
• Knowledge of data governance, security, and access management.
• Experience working in Agile development environments.

Ideal Candidate Profile

The ideal candidate will be a hands-on AWS Databricks Engineer capable of developing and maintaining enterprise-scale data engineering solutions, optimizing large-scale data processing workloads, and delivering reliable cloud-based data platforms.