Staff Software Engineer (Data Platform)

Databricks; Bengaluru; On-site; INR 2,500,000 - 4,000,000; Full time

Benefits offered by this job

Comprehensive benefits

Job summary

Databricks is looking for a Staff Software Engineer to join their Data Platform Team in Bengaluru, India. In this role, you will be responsible for building and running the Data Intelligence Platform, automating decision-making across the company. The ideal candidate has over 12 years of industry experience and strong expertise in large-scale distributed systems. Responsibilities include designing the metrics store, developing tooling, and providing technical leadership on major projects. Comprehensive benefits and a commitment to diversity and inclusion are offered.

Qualifications

• 12+ years of industry experience.
• 4+ years of experience with large-scale distributed systems.
• 5+ years of technical leadership on large projects.

Responsibilities

• Design and run the Databricks metrics store.
• Develop tooling and infrastructure for Databricks.
• Partner with engineering teams for long-term vision.

Skills

Technical leadership

Cross-functional communication

Data infrastructure passion

Experience with orchestration frameworks

Large-scale data pipeline management

Tools

Airflow

DBT

Kafka

RabbitMQ

Job description

Staff Software Engineer – Data Platform Team

Databricks builds and runs a large‑scale data and AI infrastructure platform. In this role you will help build the Data Intelligence Platform to automate decision‑making across the company.

Responsibilities

• Design and run the Databricks metrics store that enables all business units and engineering teams to bring their detailed metrics into a common platform for sharing and aggregation.
• Design and run the cross‑company Data Intelligence Platform, balancing data protections and shareability as we transition to a public company.
• Develop tooling and infrastructure to efficiently manage and run Databricks at scale across multiple clouds, geographies and deployment types, including CI/CD processes, test frameworks for pipelines and data quality, and infrastructure‑as‑code tooling.
• Design the base ETL framework used by all pipelines developed at the company.
• Partner with engineering teams to provide leadership in developing the long‑term vision and requirements for the Databricks product.
• Build reliable data pipelines and solve data problems using Databricks, partner products, and other OSS tools, providing early feedback on design and operations.
• Establish conventions and create new APIs for telemetry, debug, feature and audit event log data, and evolve them as the product and services change.
• Represent Databricks at academic and industrial conferences and events.

Qualifications

• 12+ years of industry experience.
• 4+ years of experience building large‑scale distributed systems.
• 5+ years of providing technical leadership on large projects such as ETL frameworks, metrics stores, infrastructure management and data security.
• Experience building, shipping and operating reliable multi‑geo data pipelines at scale.
• Experience working with workflow or orchestration frameworks, including open source tools like Airflow and DBT or commercial enterprise tools.
• Experience with large‑scale messaging systems such as Kafka or RabbitMQ or commercial systems.
• Excellent cross‑functional and communication skills, consensus builder.
• Passion for data infrastructure and for enabling others by making their data easier to access.

Benefits

We strive to provide comprehensive benefits and perks that meet the needs of all of our employees.

Our Commitment to Diversity and Inclusion

At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great care to ensure that our hiring practices are inclusive and meet equal employment opportunity standards. Individuals looking for employment at Databricks are considered without regard to age, color, disability, ethnicity, family or marital status, gender identity or expression, language, national origin, physical and mental ability, political affiliation, race, religion, sexual orientation, socio‑economic status, veteran status, and other protected characteristics.