Data Engineer- DataBricks

Palo Alto Innovation Labs; Pune District; Remote; INR 1,200,000 - 2,400,000; Full time

Job summary

Palo Alto Innovation Labs is seeking a Data Engineer with 4–7 years of experience to design and operationalize scalable data pipelines. The role focuses on end-to-end Databricks workflows, ingestion, transformation, and quality checks, with a strong emphasis on automated testing and CI/CD pipelines.

The engineer will implement IT infrastructure practices, FinOps for cost visibility, and secure access controls, collaborating with cross-functional teams to ensure reliability and security across

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, or related field preferred, or equivalent work experience.
• 4–7+ years in DevOps, Cloud Engineering, Site Reliability Engineering, or Platform Engineering, with at least 2+ years supporting data/analytics platforms.
• Experience operating production data workloads, including monitoring, logging, performance tuning, and incident response.
• Scripting skills (e.g., Python, Bash, PowerShell) for automation and integration.
• Experience in CP/G/retail/manufacturing or distribution environments preferred.

Responsibilities

• Design and implement CI/CD pipelines for datapipelines and transformation projects (e.g., dbt onDatabricks, SQL, notebooks).
• Orchestrate Databricks jobs and workflows end-to-end (ingestion, transformation, quality checks).
• Integrate automated testing into CI/CD, including schema and contract checks for data models and tables.
• Implement FinOps practices to support cost monitoring and allocation across the EDP.
• Automate platform operations for Databricks and related services, including workspace and cluster provisioning, library and runtime mgmt, and job deployment/config.

Skills

Databricks

CI/CD pipelines

GitHub Actions

GitLab CI

Azure DevOps

IaC

Python

Bash

Data pipelines

Education

Bachelor’s degree in CS/IT

Tools

Databricks

Airflow

Terraform

Azure

Job description

Role : Data Engineer
Location: Remote
Employment Type: Full-Time
Experience: 4-7 years

KEYRESPONSIBILITIES

• Design and implement CI/CD pipelines for datapipelines (ingestion jobs) and transformation projects (e.g., dbt onDatabricks, SQL, notebooks)
• Orchestrate Databricks jobs and workflowsend-to-end (ingestion, transformation, quality checks)
• Integrate automated testing into CI/CD,including schema and contract checks for data models and tables
• Implement FinOps best practices to supportcost monitoring and allocation across the EDP
• Automate platform operations for Databricksand related services, such as workspace and cluster provisioning, library andruntime mgmt. and job deployment/config
• Implement and maintain identity and accessmanagement for Databricks and supporting cloud resources, including workspace-and cluster-level permissions, table- and view-level access controls (e.g.,Unity Catalog or equivalent), service principals, groups, and roles forautomated workloads, and RBAC & TBAC models in collaboration with EDPArchitect
• Provide patterns, templates, and reusablemodules (Terraform modules, Airflow DAG patterns, Databricks job templates) toaccelerate onboarding of new projects
• Continuously evaluate and improve tooling,pipelines, and platform architecture to increase reliability, security, anddeveloper productivity on Databricks
• Define the code promotion process to minimizeimpacts across domains as code is promoted to production
• Manage end-to-end orchestration using managedAirflow. Contribute to defining and tracking SLA/SLO/SLIs for the platform andparticipate in incident response (triage, root cause analysis)
• Practical knowledge of IT Infrastructuretechnologies, cloud computing Azure), cybersecurity, and disaster recovery.
• Working knowledge of Azure ecosystem,including hands-on experience designing, building, and optimizing scalable datapipelines within cloud-native environments.

QUALIFICATIONS

• Hands-on experience with Databricks inproduction environment, including workspace and cluster management,jobs/workflows and integrations with orchestration tools
• Strong experience with CI/CD pipelines (e.g.,GitHub Actions, GitLab CI, Azure DevOps, or similar) and Git-based workflows.
• Strong experience with Infrastructure as Code(IaC) and orchestration tools for provisioning and managing Databricks andcloud infrastructure.
• Experience implementing automated tests andquality gates in CI/CD pipelines.
• Ability to partner effectively with dataengineers, analytics engineers, architects, and security teams.

MINIMUMEXPERIENCE & EDUCATION

• Bachelor’s degree in Computer Science,Information Technology, or related field preferred, or equivalent workexperience.
• 4–7+ years in DevOps, Cloud Engineering, SiteReliability Engineering, or Platform Engineering, with at least 2+ yearssupporting data/analytics platforms.
• Experience operating production dataworkloads, including monitoring, logging, performance tuning, and incidentresponse.
• Scripting skills (e.g., Python, Bash,PowerShell) for automation and integration.
• Experience in CPG, retail, manufacturing, ordistribution environments preferred.