Databricks Engineer - USA

Cogniify, Inc.; Mountain View, Northern (CA, KY); Hybrid; USD 120,000 - 130,000; Full time

Benefits offered by this job

Unlimited PTO

Generous parental leave

Entrepreneurial culture

Open communication

Small teams with big impact

Medical, Dental and Vision coverage

Disability & Life insurance

Mental health support

Employee Stock Purchase Program

Yearly team building

Mentorship opportunities

Manager resources

Job summary

Cogniify, Inc. in Mountain View is seeking an experienced Databricks Data Engineer to support, maintain, and enhance existing Databricks-based data applications and pipelines.

The role focuses on reliability, performance, and scalability of production Databricks workloads rather than building new platforms from scratch. You will work closely with data, analytics, and engineering teams to keep critical data applications stable, optimized, and aligned with business needs, and participate in

Qualifications

• 6–9 years in data engineering with Databricks experience.
• Strong proficiency in Spark (PySpark/Scala) and SQL.
• Experience supporting and optimizing production Databricks workloads (jobs, notebooks, Delta Lake).
• Understanding of Delta Lake concepts (ACID, time travel, optimization).
• Experience with Databricks Job clusters, Interactive clusters, and performance tuning.

Responsibilities

• Support and maintain existing Databricks apps, notebooks, jobs, and Delta Lake pipelines in production.
• Monitor, troubleshoot, and resolve issues related to job failures, performance, data quality, and cluster utilization.
• Optimize Spark jobs, SQL queries, and Delta tables for cost, performance, and reliability.
• Manage Databricks workspace configurations, clusters, job scheduling, access controls, and Unity Catalog.
• Implement and maintain data quality checks, logging, alerting, and basic observability for Databricks workloads.
• Collaborate with stakeholders to understand requirements for enhancements or bug fixes on existing applications.
• Perform incremental improvements, refactoring, and technical debt reduction on current Databricks solutions.
• Ensure adherence to best practices around security, governance, and cost management within the Databricks environment.
• Document existing pipelines, dependencies, and operational runbooks.
• Participate in on-call or support rotations as needed to maintain production stability.

Skills

Databricks

Apache Spark

SQL

Delta Lake

Production workloads

Cloud platforms

Troubleshooting

Communication

Tools

Databricks

Unity Catalog

Databricks SQL

Airflow

Git integration

Terraform

Job description

Engineering · Data Science / Data Engineering

work Full-time distance On-site location_on Mountain View

Role Overview

We are looking for an experienced Databricks Data Engineer to support, maintain, and enhance existing Databricks-based data applications and pipelines. The role focuses on ensuring reliability, performance, and scalability of production Databricks workloads rather than building net-new platforms from scratch. You will work closely with data, analytics, and engineering teams to keep critical data applications stable, optimized, and aligned with business needs.

Key Responsibilities

• Support and maintain existing Databricks applications, notebooks, jobs, and Delta Lake pipelines in production.
• Monitor, troubleshoot, and resolve issues related to job failures, performance degradation, data quality, and cluster utilization.
• Optimize existing Spark jobs, SQL queries, and Delta tables for cost, performance, and reliability.
• Manage and improve Databricks workspace configurations, including clusters, job scheduling, access controls, and Unity Catalog (where applicable).
• Implement and maintain data quality checks, logging, alerting, and basic observability for Databricks workloads.
• Collaborate with stakeholders to understand requirements for enhancements or bug fixes on existing applications.
• Perform incremental improvements, refactoring, and technical debt reduction on current Databricks solutions.
• Ensure adherence to best practices around security, governance, and cost management within the Databricks environment.
• Document existing pipelines, dependencies, and operational runbooks.
• Participate in on-call or support rotations as needed to maintain production stability

Required Qualifications

• 6–9 years of overall experience in data engineering, with strong hands-on experience in Databricks.
• Solid proficiency in Apache Spark (PySpark and/or Scala) and SQL.
• Proven experience supporting and optimizing production Databricks workloads (jobs, notebooks, Delta Lake, workflows).
• Strong understanding of Delta Lake concepts (ACID transactions, time travel, optimization techniques such as Z-ordering, vacuum, optimize).
• Experience with Databricks Job clusters, Interactive clusters, and performance tuning (partitioning, caching, shuffle optimization, autoscaling).
• Familiarity with data modeling, ETL/ELT patterns, and production data pipeline support.
• Experience working with cloud platforms (preferably Azure, AWS, or GCP) in the context of Databricks.
• Ability to troubleshoot complex Spark and Databricks issues independently.
• Strong communication skills and ability to work effectively with EST-aligned team.

Preferred Qualifications

• Experience with Unity Catalog, Databricks SQL, or Lakehouse architecture.
• Knowledge of CI/CD practices for Databricks (e.g., Databricks Asset Bundles, Git integration, Terraform/ARM templates).
• Familiarity with orchestration tools (Airflow, Azure Data Factory, or Databricks Workflows).
• Exposure to data quality frameworks, monitoring tools, or cost optimization initiatives on Databricks.
• Experience supporting analytics or BI teams consuming Databricks data products.

Salary Range

$120,000 - $130,000

Disclaimer: The base salary range is a guideline and may vary based on factors such as candidate experience, specialized skills, and geographical location. Actual compensation may include additional benefits and bonuses.

Perks And Benefits Of Working With Us

• Unlimited PTO.
• Please ask us about our very generous parental leave, much above industry standards!.
• Entrepreneurial culture where pushing limits and taking risks is everyday business.
• Open communication with management and company leadership.
• Small, dynamic teams = massive impact.
• Medical, Dental and Vision coverage for employees.
• Access to Disability & Life insurance.
• Mental health and wellbeing support
• Employer Stock Purchase Program (ESPP)
• Yearly Team building experiences
• Mentorship and sponsorship opportunities
• Manager resources and support

We are an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other protected characteristic