Senior DevOps Engineer

NXP Semiconductors; Bengaluru Urban; On-site; INR 1,500,000 - 2,000,000; Full time

Job summary

A technology leader in semiconductor solutions is seeking a skilled professional to design and maintain CI/CD pipelines specifically tailored for data workflows and machine learning. The ideal candidate will have over 9 years of experience, with extensive hands-on knowledge of Databricks, Python, and SQL. This role encompasses implementing automated deployment strategies, ensuring optimal performance and availability of data infrastructure, and establishing proactive monitoring solutions for effective data pipeline operations.

Qualifications

• 9+ years of experience in data engineering or related fields.
• Hands-on experience with Databricks for data workflows and ML model deployment.
• Proficiency in Python (PySpark) and SQL for data processing.

Responsibilities

• Design and maintain CI/CD pipelines for data workflows.
• Implement automated deployment of Databricks notebooks and jobs.
• Set up observability for data pipelines and integrate with monitoring tools.

Skills

Databricks Workflows

Delta Lake

MLflow

Python (PySpark)

SQL

Azure DevOps

Jenkins

Terraform

AWS

Airflow

Tools

GitHub Actions

Prometheus

Grafana

Oracle

MySQL

PostgreSQL

Job description

Overview

Design and maintain CI/CD pipelines for data workflows and machine learning jobs using tools like Azure DevOps, Jenkins, or GitHub Actions. For Databricks, implement automated deployment of notebooks, jobs, and Delta Live Tables, ensuring version control and environment consistency.

Implement monitoring solutions for data pipelines and Databricks jobs, track latency, throughput, and failures. Configure auto-scaling clusters and recovery strategies to guarantee high availability and resilience.

− Secrets & Environment Management:

Securely manage credentials, API keys, and Databricks Secret Scopes across development, staging, and production environments. Apply best practices for role-based access control (RBAC) and compliance.

Automate deployment of data infrastructure, Databricks clusters, and ML models using Infrastructure-as-Code (Terraform) and orchestration tools. Ensure reproducibility and reduce manual intervention.

− Observability & Alerting:

Set up end-to-end observability for pipelines using Databricks monitoring dashboards, integrate with Prometheus, Grafana, or cloud-native tools, and configure proactive alerting for SLA breaches and anomalies.

Work closely with Data Engineers, AI Engineers, and Platform teams to ensure smooth integration. Document Databricks workflows, cluster configurations, and CI/CD processes for transparency and operational excellence.

Qualifications

• Experience: 9+ years
• Hands-on experience with Databricks Workflows, Delta Lake, Unity Catalog, and MLflow for model tracking and deployment.
• Proficiency in Python (PySpark) and SQL for data processing and transformation. Ability to optimize queries for large-scale analytics.
• Familiarity with Oracle, MySQL, PostgreSQL, and integration with Databricks for ingestion and analytics.
• Experience with Airflow, Prefect, Dagster, or Databricks Workflows for scheduling and monitoring complex pipelines.
• Understanding of data modeling principles, Delta Lake architecture, and performance tuning for big data environments.
• Proficiency in Azure DevOps, GitHub Actions, Jenkins, and integration with Databricks for automated deployments.
• Experience with Terraform or CloudFormation for provisioning Databricks resources and managing cloud infrastructure.
• Strong knowledge of AWS and Databricks, including S3 integration, IAM roles, and secure data access patterns.