Databricks Engineer

Shakti Solutions; Chicago (IL); On-site; USD 140,000 - 180,000; Full time

Job summary

Shakti Solutions seeks a Databricks architect to design and implement the end-to-end Databricks platform lifecycle on AWS. You will define Dev/SIT/UAT/PROD environments, implement governance with Unity Catalog, and guide engineering teams on best practices.

Responsibilities include architecture across AWS/Azure/GCP, with Delta Lake, PySpark, Python, Genie, and Databricks Apps expertise to enable AI/ML workflows.

Qualifications

• Design and implement end-to-end Databricks platform lifecycle on AWS.
• Hands-on cloud architecture across AWS, Azure, or GCP.
• Translate business needs into technical blueprints and run POCs.
• Experience with Delta Lake, Unity Catalog, DBX notebooks, Python, PySpark, Genie.

Responsibilities

• Design and implement Dev/SIT/UAT/PROD environments for Databricks.
• Implement data access controls, lineage, and auditing via Unity Catalog.
• Lead engineering teams with best practices and architectural guidance.

Skills

AWS

Spark

Delta Lake

SQL

Cluster management

Python

PySpark

Unity Catalog

Genie

Databricks Apps

AI/ML Enablement

Tools

Databricks

DBX notebooks

Genie

Databricks Apps

Job description

• Design and implement end-to-end lifecycle of Databricks platform on AWS.
• Good experience in working with Healthcare Insurance.
• Design and implement different environments like Dev/SIT/UAT/PROD for Databricks.
• Expertise in AWS
• Implement data access controls, Lineage and auditing using Unity catalog.
• Hands-on architecture experience within AWS, Azure, or Google Cloud Platform environments.
• Deep expertise in Apache Spark, Delta Lake, SQL, and cluster resource management.
• Translate business requirements into technical blueprints, run proof-of-concepts (POCs), and guide engineering teams on best practices
• AI/ML Enablement, experience in setting up AI and Genie for Databricks.
• Experience with Delta lake house, Unity catalog, DBX notebooks, Python, Pyspark, Genie, Databricks Apps.