Data Engineer- DataBricks

Palo Alto Labs; Pune District; Remote; INR 1,200,000 - 2,500,000; Full time

Job summary

Palo Alto Labs is hiring a Data Engineer to join remotely. You will design and implement CI/CD pipelines for datapipelines and data transformations, orchestrate Databricks jobs end-to-end, and embed automated testing for quality checks.

You will also implement FinOps, automate platform ops, and manage access controls in a cloud-centric Databricks environment. You will collaborate with data engineers, analytics engineers, architects, and security teams to improve reliability, security, and

Qualifications

• Hands-on Databricks in production environments with workspace and cluster management.
• Strong experience building CI/CD pipelines and Git-based workflows.
• IaC and orchestration tools for provisioning cloud infrastructure.
• Experience with automated tests and quality gates in CI/CD pipelines.
• Ability to partner with data engineers, analytics engineers, architects and security teams.

Responsibilities

• Design and implement CI/CD pipelines for datapipelines and transformation projects.
• Orchestrate Databricks jobs and workflows end-to-end, including ingestion and quality checks.
• Integrate automated testing, including schema and contract checks for data models.
• Implement FinOps practices to monitor and allocate costs across the platform.
• Automate platform operations for Databricks, workspace provisioning, and job deployment.
• Manage identity and access controls for Databricks and cloud resources with RBAC/TBAC.
• Provide reusable patterns and templates to accelerate onboarding of new projects.
• Continuously evaluate tooling and architecture to improve reliability and security.

Skills

Databricks

CI/CD pipelines

IaC

Automated tests

Team collaboration

Education

Bachelor's degree in CS/IT

Tools

GitHub Actions

GitLab CI

Azure DevOps

Terraform

Airflow

Job description

Role : Data Engineer
Location: Remote
Employment Type: Full-Time
Experience: 4-7 years

KEYRESPONSIBILITIES

• Design and implement CI/CD pipelines for datapipelines (ingestion jobs) and transformation projects (e.g., dbt onDatabricks, SQL, notebooks)
• Orchestrate Databricks jobs and workflowsend-to-end (ingestion, transformation, quality checks)
• Integrate automated testing into CI/CD,including schema and contract checks for data models and tables
• Implement FinOps best practices to supportcost monitoring and allocation across the EDP
• Automate platform operations for Databricksand related services, such as workspace and cluster provisioning, library andruntime mgmt. and job deployment/config
• Implement and maintain identity and accessmanagement for Databricks and supporting cloud resources, including workspace-and cluster-level permissions, table- and view-level access controls (e.g.,Unity Catalog or equivalent), service principals, groups, and roles forautomated workloads, and RBAC & TBAC models in collaboration with EDPArchitect
• Provide patterns, templates, and reusablemodules (Terraform modules, Airflow DAG patterns, Databricks job templates) toaccelerate onboarding of new projects
• Continuously evaluate and improve tooling,pipelines, and platform architecture to increase reliability, security, anddeveloper productivity on Databricks
• Define the code promotion process to minimizeimpacts across domains as code is promoted to production
• Manage end-to-end orchestration using managedAirflow. Contribute to defining and tracking SLA/SLO/SLIs for the platform andparticipate in incident response (triage, root cause analysis)
• Practical knowledge of IT Infrastructuretechnologies, cloud computing Azure), cybersecurity, and disaster recovery.
• Working knowledge of Azure ecosystem,including hands-on experience designing, building, and optimizing scalable datapipelines within cloud-native environments.

QUALIFICATIONS

• Hands-on experience with Databricks inproduction environment, including workspace and cluster management,jobs/workflows and integrations with orchestration tools
• Strong experience with CI/CD pipelines (e.g.,GitHub Actions, GitLab CI, Azure DevOps, or similar) and Git-based workflows.
• Strong experience with Infrastructure as Code(IaC) and orchestration tools for provisioning and managing Databricks andcloud infrastructure.
• Experience implementing automated tests andquality gates in CI/CD pipelines.
• Ability to partner effectively with dataengineers, analytics engineers, architects, and security teams.

MINIMUMEXPERIENCE & EDUCATION

• Bachelor’s degree in Computer Science,Information Technology, or related field preferred, or equivalent workexperience.
• 4–7+ years in DevOps, Cloud Engineering, SiteReliability Engineering, or Platform Engineering, with at least 2+ yearssupporting data/analytics platforms.
• Experience operating production dataworkloads, including monitoring, logging, performance tuning, and incidentresponse.
• Scripting skills (e.g., Python, Bash,PowerShell) for automation and integration.
• Experience in CPG, retail, manufacturing, ordistribution environments preferred.