Data Ops Technical Lead - Azure Databricks

Tredence Inc.; Bengaluru; On-site; INR 3,000,000 - 5,500,000; Full time

Job summary

Tredence Inc. Bengaluru is seeking a DataOps Lead to guide a technical team through end-to-end DataOps project delivery on Azure-Databricks. You will oversee platform administration, data pipelines, and data quality initiatives while mentoring engineers.

Candidates should have 8+ years of Azure-Databricks DataOps experience, be comfortable with UK/US shift patterns, and operate within an Agile/Sprint framework. A B.Tech/BE/MCA degree is required.

Qualifications

• 8+ years of Azure-Databricks DataOps experience.
• Team handling experience: able to manage a team of 5-7 engineers.
• Open to work in UK & US Shift.
• Working in agile methodology and Sprint framework.
• Must have education degree of B.Tech/B.E/MCA.

Responsibilities

• Lead the technical team and manage day-to-day technical deliverables.
• Manage end-to-end project deliverables for DataOps projects.
• Oversee DataOps support deliverables, KPIs and platform administration for Azure-Databricks.
• Lead Data pipelines: monitor, debug, and fix data pipelines; mentor juniors.
• Data Quality Tracking and Data Observability.
• Platform Administration and Databricks governance (Identity, Access & Governance).
• Incident Management: troubleshoot Databricks clusters and jobs; minimize downtime.
• Create Runbooks, Knowledge Base, SOPs; document configurations and procedures.
• Collaborate with data engineering, data science and IT teams; provide best-practice guidance.

Skills

Azure-Databricks

DataOps

Python PySpark

Team Leadership

Data Quality

KPI Monitoring

DevOps

Education

B.Tech/BE/MCA

Tools

Databricks

SQL Server

GitHub

ServiceNow

Cosmos

ADO

Job description

Experience : 6-10yrs

Job Description, Essential Duties and Responsibilities:

• DataOps Lead is responsible for leading technical team and managing end-to-end project deliverables.
• Should have strong experience and knowledge in DataOps support deliverables, KPIs and platform administration for Azure-Databricks DataOps project
• Data Ops & Support roles
• Data Anomaly Investigations & Root Cause Analysis
• Hotfixes, Enhancements & Release Coordination
• Creating Runbooks and Knowledge Base documents
• Data Quality Tracking
• Platform Administration
• Databricks Platform Administration
• Identity, Access & Governance
• Tools & Technologies
• Databricks
• SQL Server
• Python and PySpark scripting using DLTs
• Github
• AI based tools (Claude, Co-Pilotetc)
• ServiceNow[AG1]
• Cosmos
• ADO
• Data pipelines: Monitor, identify issue, Debugging and Bug fixing of data pipelines. Should understand complex data pipelines and data engineering concepts. Mentoring and guiding junior team members.
• Data Quality & Data Observability: Identify and fix the Data quality issues by performing RCA. Should have a good understanding of data observability tools like Unravel etc.
• Data orchestration: Orchestration of jobs using Airflow.
• DevOps: Should have a good understanding of the Dev Ops process.
• Collaboration: Work closely with data engineering, data science, and IT teams to support data pipeline development, troubleshoot issues, and provide guidance on best practices.
• Incident Management: Lead troubleshooting and resolution of issues related to Databricks clusters and jobs, ensuring minimal downtime and impact on business operations.
• KPI: Responsible for managing MTTA, MTTR, IRR, response & resolution SLA and other key delivery matrix.
• Upgrades & Patches: Manage upgrades, patches, and new releases of Databricks, ensuring minimal disruption to ongoing operations.
• Documentation: Create and maintain detailed documentation of the Databricks environment, configurations, and procedures.
• SOP: Able to build SOPs and set up DataOps best practices.
• Continuous improvement and Innovation: Bringing continuous improvement and innovation for cost optimization, effort reduction, incidents reduction, platform stability and discuss with client, Development team for implementation.
• Team Leading: Lead the technical team and manage day-to-day technical deliverables.
• Stakeholder management: Able to handle key stakeholders, including client leadership.

Skills, Education and Experience:

• Having 8+ years of Azure-Databricks DataOps experience.
• Team handling experience: Able to handle a team of 5-7 engineers
• Open to work in UK & US Shift.
• Working in agile methodology and Sprint framework
• Must have education degree of B. Tech/B.E/MCA