AWS Databricks Lead

Smart IT Frame LLC; Connecticut; On-site; USD 120,000 - 170,000; Full time

Job summary

Smart IT Frame LLC is seeking an experienced AWS Databricks Lead to design, develop, and lead cloud-based data engineering solutions on AWS. You will build scalable data pipelines, implement Delta Lake architectures, and provide technical leadership to the data engineering team.

The ideal candidate brings hands-on AWS and Databricks expertise, PySpark, Python, SQL, and strong knowledge of ETL/ELT, data security, and performance tuning.

Qualifications

• Strong hands-on experience with AWS and Databricks.
• Expertise in PySpark, Python, SQL, and Delta Lake.
• Experience leading data engineering teams/projects.
• Experience with AWS services (S3, Glue, IAM, Lambda).
• Excellent troubleshooting and communication skills.

Responsibilities

• Lead the design and development of data solutions using Databricks on AWS.
• Build and optimize ETL/ELT pipelines using PySpark and SQL.
• Develop Delta Lake and Databricks workflows for large-scale data processing.
• Integrate Databricks with AWS services such as S3, Glue, Lambda, and Redshift.
• Implement data security, governance, performance tuning, and monitoring.
• Lead technical discussions, code reviews, and mentor team members.
• Collaborate with architects, business teams, and other engineering teams.

Skills

AWS

Databricks

PySpark

Python

SQL

Delta Lake

ETL/ELT

Cloud architecture

Data engineering leadership

Troubleshooting & communication

Tools

AWS S3

AWS Glue

AWS IAM

AWS Lambda

Redshift

Job description

We are looking for an experienced AWS Databricks Lead to design, develop, and lead cloud-based data engineering solutions. The role involves building scalable data pipelines, implementing Databricks solutions on AWS, and providing technical leadership to the data engineering team.

Key Responsibilities:

• Lead the design and development of data solutions using Databricks on AWS.
• Build and optimize ETL/ELT pipelines using PySpark and SQL.
• Develop Delta Lake and Databricks workflows for large-scale data processing.
• Integrate Databricks with AWS services such as S3, Glue, Lambda, and Redshift.
• Implement data security, governance, performance tuning, and monitoring.
• Lead technical discussions, code reviews, and mentor team members.
• Collaborate with architects, business teams, and other engineering teams.

Required Skills:

• Strong hands-on experience with AWS and Databricks.
• Expertise in PySpark, Python, SQL, and Delta Lake.
• Experience with AWS S3, Glue, IAM, Lambda, and related services.
• Strong knowledge of data engineering, ETL/ELT, and cloud architecture.
• Experience leading data engineering teams/projects.
• Excellent troubleshooting and communication skills.