Lead Data Enineer - INTL - India

Insight Global; Waltham (MA); On-site; USD 140,000 - 180,000; Full time

Job summary

Insight Global in the United States seeks a Lead Data Engineer to provide hands-on development and technical leadership while guiding a team of data engineers.

You will build and optimize Databricks pipelines, ingest data from ERP sources such as Salesforce, SAP, Oracle, and SQL Server, and ensure data quality across Redshift and Athena.

This role requires mentoring engineers, DevOps/DataOps practices, and collaboration in an Agile, multi-vendor environment using Jira and GitHub.

Qualifications

• 6–8+ years in Data Engineering, with 2+ years in lead/technical leadership.
• Deep, hands-on Databricks on AWS experience designing and operating production pipelines.
• Advanced proficiency with PySpark and Apache Spark.
• Experience integrating Databricks with S3, IAM, Redshift, Athena, Glue.
• Strong programming and data modeling with SQL and Python.
• DevOps/DataOps mindset with GitHub, PRs, and code reviews.
• Experience leading delivery in Agile using Jira.

Responsibilities

• Lead data engineering deliverables and mentor junior engineers.
• Own end-to-end data pipelines from ingestion to deployment on Databricks.
• Oversee production deployments and ensure data quality.
• Collaborate in a multi-vendor Agile environment and participate in sprint planning.

Skills

Leadership

Databricks

PySpark

SQL

Python

AWS

GitHub

Jira

DataOps

Agile

Tools

Databricks Lakehouse Platform

Job description

Job Description

The Lead Data Engineer will provide both hands‑on development and technical leadership while guiding a team of data engineers while building, reviewing, and optimizing end‑to‑end data pipelines on Databricks (PySpark) within the AWS Enterprise Data Platform (EDP). This person will build and optimize data pipelines in Databricks, and ingest and transform data from ERP sources such as Salesforce, SAP, Oracle, and SQL Server. They will handle production deployments, ensuring data quality across AWS DWH environments like Redshift and Athena. They will collaborate within a multi‑vendor Agile environment and participat3 in sprint planning through Jira. They will help to enforce DevOps/Data Ops standards using GitHub, ensuring that deliverables meet timelines and enterprise architecture frameworks. The Lead will be responsible for ensuring junior and mid‑level engineers deliver effectively within each sprint cycle.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [e-mail] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

6–8+ years of experience in Data Engineering, including 2+ years in a Lead or Technical Leadership role, owning delivery while mentoring engineers and setting engineering best practices.

Deep, hands‑on expertise with Databricks on AWS, with demonstrated experience designing and operating production‑grade data pipelines on the Databricks Lakehouse Platform, while owning Databricks job orchestration, cluster configuration, and performance optimization.

Advanced proficiency with PySpark and Apache Spark to build scalable, distributed data transformations for large‑volume enterprise datasets

Strong experience integrating Databricks with AWS services such as S3, IAM, Redshift, Athena, and Glue

Proven ability to own end‑to‑end data pipelines, which includes ingesting and transforming data from complex ERP and operational sources (Salesforce, SAP, Oracle, SQL Server).

Strong programming and data modeling skills, using SQL and Python for data transformation and analytics use cases.

DevOps / DataOps mindset, with experience using GitHub for version control, pull requests, and code reviews.

Experience leading delivery in Agile environments, including active participation in sprint planning, estimation, and backlog refinement using Jira.