Principal Data Engineer

Mployee.me

Pune District

Hybrid

INR 3,500,000 - 5,200,000

Full time

9 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Healthcare coverage
Wellness reimbursement
Stock options
Annual learning reimbursement
Culture Club & career development

Job summary

HG Insights is hiring for a senior data engineering leader based in Pune to own and operate our data platform end-to-end. You will drive the transformation stack, improve entity matching, and ensure scalable, cost-aware pipelines that deliver trusted datasets to customers across the globe.

You will collaborate with engineers in Pune, the US and Brazil, shaping product-grade data attributes and the governance practices that support AI-first data solutions.

Qualifications

  • 15+ years building production data engineering systems with at least 5+ years in staff/principal scope.
  • Deep Spark and Databricks expertise at multi-terabyte scale and lakehouse table management.
  • Strong SQL and dimensional modeling with prudent rule-breaking judgement.

Responsibilities

  • Own the data platform architecture end-to-end, decide build vs. buy and batch vs. incremental.
  • Build quality layer with contracts, freshness metrics, and data quarantine pipelines.
  • Improve entity matching and attribution with explainable decisions.
  • Make releases boring with automated recurring deliveries.
  • Treat compute cost as a first-class metric across Spark, orchestration, and serving.

Skills

Data platform architecture
Entity matching / data quality
SQL & dimensional modeling
Spark/Databricks
Python/Scala/Java
AWS EC2/S3/EMR

Tools

Airflow
Spark
Databricks
Docker
Kubernetes
Terraform

Job description

About HG

HG Insights is the pioneer of Revenue Growth Intelligence. For more than a decade, we have delivered comprehensive, AI- driven datasets on B2B buyers, technology adoption, IT spend, and buyer intent, sourced from billions of data points.
Today, we are a trusted partner to Fortune 500 technology companies, hyperscalers, and innovative B2B vendors seeking precise go-to-market analytics and decision-making. Through an evolving suite of AI agents that incorporate first-party data and buyer signals, HG Insights enables AI-powered GTM automation across sales, marketing, RevOps, and data analytics teams, modernizing GTM execution from strategy through activation.

About the Role

This is a senior individual contributor role on the team that builds and runs HG's data platform: everything between a raw vendor file landing in our lake and a finished dataset arriving in a customer's hands. It includes the transformation stack that turns messy external signals into product-grade company attributes, the matching systems that decide which company a piece of evidence belongs to and the internal applications our analysts use to curate and correct the result.

You will be based in Pune, working with engineers at our Pune, US & Brazil locations.

What you’ll do
  • Own the architecture of the data platform end to end, and make the calls on build vs. buy, batch vs. incremental, and where each workload belongs.
  • Build the quality layer that catches silent failure: contracts at every handoff, freshness and completeness monitoring, and pipelines that quarantine bad data rather than publish it.
  • Improve entity matching and attribution — and make match decisions explainable to the customers who depend on them.
  • Make our release cycle boring. Recurring deliveries should not depend on people watching them.
  • Treat compute cost as a first-class engineering metric across Spark, orchestration, and the serving tier.
What we are looking for
  • 15+ years building production grade data engineering systems, with a minimum of 5 years in a staff or principal scope.
  • Deep Spark and Databricks expertise at multi-terabyte scale, including the operational side of lakehouse table formats - merges, compaction, small files, schema evolution.
  • Strong SQL and dimensional modelling, and the judgment to know when to break the rules.
  • Experience with performant and scalable OLTP setups (MySQL/Postgres).
  • Airflow orchestration (DAGs, operators, sensors) and integration with Spark/Databricks.
  • Proven experience in AWS ecosystems (EC2, S3, EMR).
  • Hands-on entity resolution, fuzzy matching, or record linkage at scale. This is central to what we do.
  • Python/Scala/Java, with real software discipline: testing, CI/CD, code review, infrastructure as code is a plus.
  • Experience in Docker - Kubernetes, Terraform is a plus.
  • Experience in integrating AI first implementations in traditional data engineering setups will greatly help in shaping our future designs.
  • Experience with machine learning pipelines (Spark MLlib, Databricks ML) for predictive analytics.
  • Knowledge of data governance frameworks and compliance standards (GDPR, CCPA).
Preferred Qualifications

B2B firmographic, technographic or intent data; streaming and CDC; warehouse-to-lakehouse migration experience.

What we offer/Benefits

We take care of our people fully and genuinely. Every HG Insights employee receives comprehensive healthcare coverage for themselves and their families, a monthly wellness reimbursement and access to Able To professional mental health coaching for stress, anxiety and depression support as per the regions. For your financial future, US employees benefit from a 401(k) with up to 5% company matching and all UK employees are enrolled in a pension scheme. Every employee also receives stock options because the people who build this company should share in its success.
We invest in your growth through an annual learning reimbursement and a genuine commitment to developing your career at every stage. And through our Culture Club, you will always have a real voice in shaping the kind of workplace we build together.

Equal Opportunity Employer

HG Insights is proud to be an Equal Employment Opportunity employer. We do not accept unsolicited resumes from recruiters or employment agencies. In the event of a recruiter or agency submitting a resume or candidate without a signed agreement being in place, we explicitly reserve the right to pursue and hire such candidates without any financial obligation to the recruiter or agency. Any unsolicited resumes, including those submitted directly to hiring managers, are deemed to be the property of HG Insights.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principal Data Engineer
Principal Data Engineer

TrustRadius • Pune District

On-site
INR 4,000,000 - 6,000,000
Healthcare coverage
Stock options
Learning reimbursement
+1
Principal/Staff Data Engineer
Principal/Staff Data Engineer

TrustRadius • Pune District

On-site
INR 3,500,000 - 7,000,000
Healthcare coverage
Annual learning reimbursement
Stock options
Principal Data Engineer
Principal Data Engineer

HG Insights • Maharashtra

On-site
INR 4,000,000 - 6,500,000
Healthcare coverage
Wellness reimbursement
Learning reimbursement
+2
People Operations Specialist
People Operations Specialist

TigerHire • Pune District

Remote
INR 8,499,000 - 14,164,000
Stock options
Remote-first role US/India
401(k) with company matching
+1
Hiring GCP Data Engineers
Hiring GCP Data Engineers

Hinduja Global Solutions (HGS) • Hyderabad

On-site
INR 1,400,000 - 2,000,000
Competitive renumeration
Shift allowance
Health, Accidental and Life coverage
+4
Customer Success Manager II
Customer Success Manager II

TrustRadius • Pune District

On-site
INR 1,000,000 - 1,800,000
Healthcare coverage
Wellness reimbursement
Mental health coaching
+5
Senior Data Engineer
Senior Data Engineer

BIG HAPPY • Mohali

On-site
INR 2,800,000 - 4,600,000
Flexible Time-Off
Health & Family Support
Performance Bonuses
+2
Senior Data Engineer
Senior Data Engineer

Big Happy • Panchkula

On-site
INR 1,800,000 - 3,600,000
Flexible time-off
Health & family support
Performance bonuses
+2
Lead/Staff - Data Engineer
Lead/Staff - Data Engineer

usemultiplier.com • Chennai District

Hybrid
INR 4,000,000 - 7,000,000
Hybrid environment
Competitive benefits
ESOPs
+3
Staff Software Engineer (Hands-on Architect)
Staff Software Engineer (Hands-on Architect)

Datahash • Bengaluru

On-site
INR 3,000,000 - 5,000,000