Senior Platform Engineer

Daman; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

Daman seeks an experienced Senior Platform Engineer to manage the Databricks environment on Azure, focusing on administration, security, performance, and cost optimization. You will oversee cluster lifecycle, jobs, and integrations with other Azure services to support robust data pipelines.

You will implement access controls, governance, and IaC automation while collaborating with data engineers and DevOps teams to ensure reliability and scalable analytics workloads.

Qualifications

• 5+ years of experience in data platform or cloud administration.
• Strong hands-on experience with Databricks Administration.
• Experience in cluster configuration, performance tuning, and cost optimization.
• Knowledge of security best practices, including identity management and data protection.
• Experience with CI/CD pipelines and infrastructure as code (Terraform preferred).
• Proficiency in Python, SQL, or Scala for troubleshooting and support.
• Familiarity with monitoring and logging tools.
• Strong problem-solving and analytical skills.

Responsibilities

• Administer and manage the Databricks platform on Azure, including workspace configuration and cluster lifecycle management.
• Monitor system performance, troubleshoot issues, and ensure high availability and reliability of the platform.
• Implement and manage access controls, RBAC, and data security policies.
• Optimize cluster performance and cost through effective configuration and auto-scaling strategies.
• Manage Databricks jobs, workflows, and scheduling.
• Integrate Databricks with other Azure services such as Data Lake, Azure Data Factory, and Synapse Analytics.
• Maintain and enforce data governance and compliance standards.
• Automate infrastructure provisioning using tools like Terraform or ARM templates.
• Collaborate with data engineers, architects, and DevOps teams to support data pipelines and analytics workloads.
• Perform upgrades, patching, and environment maintenance activities.
• Create and maintain documentation for configurations, processes, and best practices.

Skills

Databricks Administration

Cluster configuration

Performance tuning

Cost optimization

Security best practices

CI/CD pipelines / Infrastructure as代码

Python/SQL/Scala troubleshooting

Monitoring & logging tools

Analytical problem-solving

Tools

Terraform

ARM templates

Job description

We are seeking an experienced Senior Platform Engineer to manage, optimize, and secure our Databricks environment within the Azure ecosystem. The ideal candidate will have deep expertise in Databricks platform administration, Azure cloud services, and data platform governance. This role requires hands-on experience in cluster management, security, cost optimization, and performance tuning.

Key Responsibilities:

• Administer and manage the Databricks platform on Azure, including workspace configuration and cluster lifecycle management
• Monitor system performance, troubleshoot issues, and ensure high availability and reliability of the platform
• Implement and manage access controls, RBAC, and data security policies
• Optimize cluster performance and cost through effective configuration and auto-scaling strategies
• Manage Databricks jobs, workflows, and scheduling
• Integrate Databricks with other Azure services such as Data Lake, Azure Data Factory, and Synapse Analytics
• Maintain and enforce data governance and compliance standards
• Automate infrastructure provisioning using tools like Terraform or ARM templates
• Collaborate with data engineers, architects, and DevOps teams to support data pipelines and analytics workloads
• Perform upgrades, patching, and environment maintenance activities
• Create and maintain documentation for configurations, processes, and best practices

Required Skills & Qualifications:

• 5+ years of experience in data platform or cloud administration
• Strong hands-on experience with Databricks Administration
• Experience in cluster configuration, performance tuning, and cost optimization
• Knowledge of security best practices, including identity management and data protection
• Experience with CI/CD pipelines and infrastructure as code (Terraform preferred)
• Proficiency in Python, SQL, or Scala (for troubleshooting and support)
• Familiarity with monitoring and logging tools
• Strong problem-solving and analytical skills