Data Engineer (Databricks)

K1 Investment Management, LLC

Pune District

On-site

INR 900,000 - 1,500,000

Full time

13 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health coverage
Flexible PTO
Family benefits
Life and disability insurance
Voluntary benefits
Tax-advantaged accounts
Volunteer day

Job summary

Onit is redefining the future of legal operations with an AI-driven platform. We seek a Data Engineer to build and operate Databricks-based data pipelines powering a new benchmarking capability, integrating data across enterprise systems with strong governance and privacy baked in.

You will partner with Data Scientists, Data Architects, and Product teams to deliver reliable ingestion, entity resolution, and scalable data quality metrics for multi-tenant analytics.

Qualifications

  • 5+ years in data engineering or analytics engineering with production data pipelines.
  • Experience with Databricks, PySpark, Spark SQL, and Delta Lake.
  • Strong SQL and Python and data modelling skills.
  • Experience with cloud platforms (AWS preferred) and CI/CD for data pipelines.
  • Familiarity with governance, privacy, and multi-tenant architectures.

Responsibilities

  • Pipeline Development: design, build, and maintain scalable pipelines in Databricks (PySpark, Spark SQL, Delta Lake).
  • Data Integration & Modelling: implement canonical data model for consistent reconciliation.
  • Entity Resolution: build pipelines to resolve duplicates and conflicts across systems.
  • Data Quality Engineering: profile data, implement quality checks, and monitor quality metrics.
  • Orchestration & Reliability: manage jobs with Databricks Workflows and optimize performance/cost.
  • Governance & Privacy: apply Unity Catalog governance and anonymisation rules.
  • Collaboration: work with Data Scientist and Engineering to expose curated data.

Skills

Data Eng
SQL
Python
Data Modeling
Cloud Platforms

Tools

Databricks
PySpark
Spark SQL
Delta Lake
Unity Catalog

Job description

About Onit

We're redefining the future of legal operations through the power of AI. Our cutting-edge platform streamlines enterprise legal management, matter management, spend management and contract lifecycle processes, transforming manual workflows into intelligent, automated solutions.


We’re a team of innovators using AI at the core to help legal departments become faster, smarter, and more strategic. As we continue to grow and expand the capabilities of our new AI-centric platform, we’re looking for bold thinkers and builders who are excited to shape the next chapter of legal tech.


If you're energized by meaningful work, love solving complex problems, and want to help modernize how legal teams operate, we’d love to meet you.


Position Summary

The Data Engineer builds and operates the Databricks-based data pipelines that power a new customer-facing benchmarking capability in Onit's AI-driven legal operations platform. Integrating data from multiple source systems across enterprise legal management, matter management, and spend management, this role turns inconsistent, duplicated, and incomplete source data into a clean, reconciled, and trustworthy foundation for analytics. Working closely with the Data Architect, Data Scientist, and Product, the Data Engineer delivers reliable ingestion, entity resolution, and data quality measurement at scale, with multi-tenant data isolation and privacy built in.


Key Responsibilities


  • Pipeline Development: Design, build, and maintain scalable batch and incremental pipelines in Databricks (PySpark, Spark SQL, Delta Lake) that ingest data from multiple source systems into a medallion (bronze/silver/gold) architecture.

  • Data Integration & Modelling: Implement the canonical data model defined with the Data Architect, so figures reconcile consistently across every product surface.

  • Entity Resolution: Build production pipelines that resolve duplicate and conflicting records across systems that were never designed to reconcile with one another.

  • Data Quality Engineering: Profile source data, implement automated data quality checks and expectations, and build monitoring and reporting so data quality stays continuously visible against defined targets.

  • Orchestration & Reliability: Orchestrate and monitor jobs with Databricks Workflows, optimise Spark performance and cost, and troubleshoot pipeline failures.

  • Governance & Privacy: Apply Unity Catalog governance, multi-tenant data isolation, and anonymisation and aggregation rules for pooled cross-customer analytics.

  • Collaboration: Partner with the Data Scientist to productionise comparability models and data quality metrics, and with Engineering to expose curated data to the product.


Required Skills


  • 5+ years in data engineering or analytics engineering, building production data pipelines in a cloud environment.

  • Hands-on Databricks experience: PySpark and Spark SQL, Delta Lake, Databricks Workflows/Jobs, and Unity Catalog.

  • Strong SQL and Python, with solid data modelling skills (dimensional and/or canonical models) and experience with the lakehouse/medallion architecture.

  • Practical experience with deduplication, record linkage, or entity resolution, and with diagnosing and remediating data quality problems at scale.

  • Experience implementing automated data quality checks, pipeline monitoring, and Spark performance and cost optimisation.

  • Experience with cloud platforms (AWS preferred), Git-based version control, and CI/CD for data pipelines.

  • Clear communication with data scientists, architects, and product stakeholders, including flagging data limitations early.

  • Working proficiency with AI tools (e.g., Claude, Databricks Assistant, GitHub Copilot) to accelerate pipeline development, debugging, and documentation, with the judgment to validate AI-generated code before production use.


Preferred Qualifications


  • Databricks certification (e.g., Data Engineer Associate or Professional).

  • Experience with streaming or incremental ingestion (e.g., Auto Loader, Structured Streaming, Delta Live Tables).

  • Experience building data for benchmarks, scoring models, or analytics intended for external consumption.

  • Experience with anonymisation, aggregation thresholds, or privacy-preserving analytics.

  • Background in enterprise B2B SaaS or the legal technology domain.


Benefits & Perks That Support You:


  • Health Coverage: Employee and immediate family members.

  • Time Away:Flexible paid time off and 10 company paid holidays annually.

  • Family Support:Exceptional paid leave for birth parents, non-birth parents, and caregivers. Onit also offers surrogacy and adoption reimbursement.

  • Income Protection:100% employer-paid life and disability insurance.

  • Additional Coverage Options:Voluntary benefits including hospital indemnity, critical illness, accident.

  • Tax-Advantaged Accounts:Flexi, NPS.

  • Community Engagement:One paid volunteer day each year to give back to the community.


Our Commitment to Applicants

We know that not everyone will check every box in a job description. At Onit, we value diversity, inclusion, and authenticity. If you’re excited about this role but your experience doesn’t align perfectly with every qualification, we encourage you to apply. You may be exactly who we’re looking for.


Onit Values

Customer First -Customer success is our success. We deliver value, listen, and act on customer needs.


Purposeful Innovation -Innovation fuels our growth. We harness creativity to solve problems and lead with the intentions and expertise.


Win as One -Teamwork is how we win. We are accountable, act with integrity, and communicate openly.


Intentional Growth -Our people are the difference. We create an environment with compelling work, impactful contributions, and career growth.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer (Databricks)
Data Engineer (Databricks)

Onit • Pune District

On-site
INR 2,400,000 - 3,600,000
Health coverage for employee and 1st 2
Flexible PTO
Family leave & adoption support
+4
Data Engineer (Databricks)
Data Engineer (Databricks)

Onit, Inc. • Pune District

On-site
INR 900,000 - 1,300,000
Health Coverage
Flexible time off
Birth/adoption leave
+4
Data Scientist
Data Scientist

Onit, Inc. • Pune District

On-site
INR 2,500,000 - 3,500,000
Health coverage
Time away: Flexible paid time off & 10
Family leave & caregiver support
+4
Data Architect
Data Architect

Onit, Inc. • Pune District

On-site
INR 3,500,000 - 6,000,000
Health Coverage
Flexible paid time off
Volunteer day
+1
Data Scientist
Data Scientist

K1 Investment Management, LLC • Pune District

On-site
INR 2,500,000 - 4,000,000
Health coverage for employee + family
Flexible paid time off
10 company holidays
+4
Software Engineer
Software Engineer

OnIT India • Pune District

On-site
INR 1,200,000 - 2,400,000
Health Coverage
Time Off & Holidays
Exceptional family leave including sur
+4
Data Engineer (Databricks)
Data Engineer (Databricks)

Onit India • Pune District

On-site
INR 1,500,000 - 2,800,000
Health Coverage
Flexible paid time off
Family support and leaves
+4
Senior Software Engineer - Data
Senior Software Engineer - Data

Onit, Inc. • Pune District

On-site
INR 3,500,000 - 6,500,000
Health coverage
Flexible PTO
Family leave reimbursement
+4
Software Engineer - Data
Software Engineer - Data

Onit, Inc. • Pune District

Hybrid
INR 1,200,000 - 2,400,000
Health Coverage
Time Away
Family Support
+4
Data Architect
Data Architect

K1 Investment Management, LLC • Pune District

On-site
INR 4,000,000 - 8,000,000
Health coverage for employee and fam
Flexible paid time off
Generous family leave and caregiving
+4