AWS Databricks Engineer

Shakti Solutions; United States; On-site; USD 140,000 - 170,000; Full time

Job summary

Shakti Solutions is seeking a seasoned Databricks Architect to design and implement the lifecycle of a Databricks platform on AWS, including Dev/SIT/UAT/PROD environments. You will drive data access controls, lineage, and auditing using Unity Catalog while guiding engineering teams on best practices.

You will also work across AWS/Azure/GCP, leveraging Apache Spark, Delta Lake, Python, PySpark, Genie, and Databricks Apps to enable AI/ML workflows and data science enablement.

Qualifications

• Experience designing Databricks platforms on AWS.
• Strong knowledge of Delta Lake, Spark, and SQL.
• Hands-on cloud architecture across AWS/Azure/GCP.
• Proven ability to translate business requirements into technical blueprints.
• Experience with AI/ML enablement and Genie for Databricks.

Responsibilities

• Design and implement end-to-end lifecycle of Databricks platform on AWS.
• Design and implement environments like Dev/SIT/UAT/PROD for Databricks.
• Implement data access controls, lineage, and auditing using Unity Catalog.
• Guide engineering teams on best practices and conduct POCs.
• Work on Delta Lakehouse concepts and Databricks tooling like Genie and DBX notebooks.

Skills

Databricks on AWS

AWS

Delta Lake

Apache Spark

SQL

Python

PySpark

Unity Catalog

Genie

DBX notebooks

Databricks Apps

Tools

Delta Lake

DBX notebooks

Genie

Unity Catalog

Databricks Apps

Job description

Full Time or Contract w2 with shakti Solutions - No C2c / H1B transfers.

• Design and implement end-to-end lifecycle of Databricks platform on AWS.
• Design and implement different environments like Dev/SIT/UAT/PROD for Databricks.
• Expertise in AWS
• Implement data access controls, Lineage and auditing using Unity catalog.
• Hands-on architecture experience within AWS, Azure, or GCP environments.
• Deep expertise in Apache Spark, Delta Lake, SQL, and cluster resource management.
• Translate business requirements into technical blueprints, run proof-of-concepts (POCs), and guide engineering teams on best practices
• AI/ML Enablement, experience in setting up AI and Genie for Databricks.
• Experience with Delta lake house, Unity catalog, DBX notebooks, Python, Pyspark, Genie, Databricks Apps