Principal Data Engineer

HG Insights; Pune District; On-site; INR 4,000,000 - 6,400,000; Full time

Job summary

HG Insights, based in Pune, seeks a senior data engineer to lead the data platform from raw vendor data to customer-ready datasets. You will architect end-to-end pipelines, optimize Spark/Databricks on AWS, and ensure data quality and governance. You’ll work with engineers in Pune, the US, and Brazil to deliver scalable, explainable data products.

15+ years in production data systems and strong entity resolution, SQL modelling, and CI/CD discipline are essential.

Qualifications

• 15+ years building production-grade data engineering systems.
• 5+ years in staff or principal scope.
• Deep Spark and Databricks expertise at multi-terabyte scale.
• Strong SQL and dimensional modelling.
• Experience with MySQL/Postgres OLTP.
• Airflow orchestration and Spark/Databricks integration.
• AWS ecosystem experience (EC2, S3, EMR).
• Hands-on entity resolution / record linkage at scale.
• Proficiency in Python/Scala/Java with CI/CD mindset.
• Experience with AI-first implementations in data engineering.

Responsibilities

• Own the architecture of the data platform end to end, decide build vs buy and batch vs streaming.
• Build the quality layer with monitoring, data freshness, and quarantine pipelines.
• Improve entity matching and attribution and explainability to customers.
• Make our release cycle boring with reliable, automated deliveries.
• Treat compute cost as a first-class metric across Spark, orchestration, and serving.

Skills

Spark & Databricks

SQL & dimensional modelling

Python/Scala/Java

Airflow

AWS (EC2, S3, EMR)

Entity resolution

CI/CD

Docker/Kubernetes

Terraform

Data governance

Tools

Databricks

Spark

MySQL

PostgreSQL

Docker

Kubernetes

Terraform

Airflow

Job description

HG Insights is the pioneer of Revenue Growth Intelligence. For more than a decade, we have delivered comprehensive, AI- driven datasets on B2B buyers, technology adoption, IT spend, and buyer intent, sourced from billions of data points.

Today, we are a trusted partner to Fortune 500 technology companies, hyperscalers, and innovative B2B vendors seeking precise go-to-market analytics and decision-making. Through an evolving suite of AI agents that incorporate first-party data and buyer signals, HG Insights enables AI-powered GTM automation across sales, marketing, RevOps, and data analytics teams, modernizing GTM execution from strategy through activation.

About the Role

This is a senior individual contributor role on the team that builds and runs HG's data platform: everything between a raw vendor file landing in our lake and a finished dataset arriving in a customer's hands. It includes the transformation stack that turns messy external signals into product-grade company attributes, the matching systems that decide which company a piece of evidence belongs to and the internal applications our analysts use to curate and correct the result.

You will be based in Pune, working with engineers at our Pune, US & Brazil locations.

What you’ll do

• Own the architecture of the data platform end to end, and make the calls on build vs. buy, batch vs. incremental, and where each workload belongs.
• Build the quality layer that catches silent failure: contracts at every handoff, freshness and completeness monitoring, and pipelines that quarantine bad data rather than publish it.
• Improve entity matching and attribution — and make match decisions explainable to the customers who depend on them.
• Make our release cycle boring. Recurring deliveries should not depend on people watching them.
• Treat compute cost as a first-class engineering metric across Spark, orchestration, and the serving tier.

What we are looking for

• 15+ years building production grade data engineering systems, with a minimum of 5 years in a staff or principal scope.
• Deep Spark and Databricks expertise at multi-terabyte scale, including the operational side of lakehouse table formats — merges, compaction, small files, schema evolution.
• Strong SQL and dimensional modelling, and the judgment to know when to break the rules.
• Experience with performant and scalable OLTP setups (MySQL/Postgres).
• Airflow orchestration (DAGs, operators, sensors) and integration with Spark/Databricks.
• Proven experience in AWS ecosystems (EC2, S3, EMR).
• Hands-on entity resolution, fuzzy matching, or record linkage at scale. This is central to what we do.
• Python/Scala/Java, with real software discipline: testing, CI/CD, code review, infrastructure as code is a plus.
• Experience in Docker - Kubernetes, Terraform is a plus.
• Experience in integrating AI first implementations in traditional data engineering setups will greatly help in shaping our future designs.
• Experience with machine learning pipelines (Spark MLlib, Databricks ML) for predictive analytics.
• Knowledge of data governance frameworks and compliance standards (GDPR, CCPA).

Preferred Qualifications

Helpful, not required: B2B firmographic, technographic or intent data; streaming and CDC; warehouse-to-lakehouse migration experience.