Databricks Integration Lead

New York Technology Partners

United States

Sur place

USD 170 000 - 210 000

Plein temps

Il y a 30 heures
Soyez parmi les premiers à postuler

Obtenez une réponse de cet employeur

Votre CV et votre lettre de motivation, adaptés exactement à ce que l’employeur recherche.

Passez les filtres ATS

Résumé du poste

New York Technology Partners seeks a senior, hands-on Databricks Integration Lead to own the integration architecture for a large migration from Veeva CRM to Vault CRM across 5 Veeva CRM orgs and 46 integration points. You will guide a Databricks engineering team, set technical direction, and ensure airtight data transfer with no loss.

Deep Databricks experience is essential, along with pharma data knowledge and strong leadership of distributed teams.

Qualifications

  • 10+ years in data integration/architecture; 5+ years hands-on Databricks in production.
  • Expert in PySpark/Spark SQL, Delta Lake, Unity Catalog, workflows and cost tuning.
  • Delivered large multi-source integration programs with dozens of interfaces.
  • Hands-on Veeva CRM data model and integration experience; Vault CRM knowledge preferred.
  • Life sciences/pharma commercial data experience (HCP/HCO data, consent, MDM).
  • Strong REST/SOAP API, bulk load, and file/event-based integration skills.
  • Proven technical leadership of engineering teams, including offshore resources.
  • Familiarity with GxP validation and regulated-environment delivery.

Responsabilités

  • Own end-to-end integration architecture across Databricks, 5 Veeva CRM orgs, Vault CRM, and connected systems
  • Catalog and analyze all 46 integration points; consolidate, retire, or standardize rather than lift-and-shift
  • Define standards: medallion architecture, Delta Lake, Unity Catalog governance, workflow orchestration, CI/CD
  • Design coexistence patterns while Veeva CRM and Vault CRM run in parallel
  • Review designs, notebooks, and pipelines; act as technical escalation point for the team
  • Establish validation/reconciliation frameworks and support GxP, privacy, and audit requirements
  • Own technical cutover plan, rollback strategy, and monitoring/operational support model
  • This is not an oversight-only role.

Connaissances

Databricks
PySpark/Spark SQL
Delta Lake
Unity Catalog
GxP validation
REST/SOAP API
Data integration
Technical leadership
Offshore teams
Pharma data experience

Outils

Veeva CRM
Vault CRM
Databricks

Description du poste

  • Build, not just lead: write, review and deploy code personally, and own each integration from design specification through build, testing, deployment across Dev, Sandbox, UAT and Prod, runbooks, and handover.
  • Ability to Analyze current Integrations: Should have the technical ability to analyze the legacy integrations (Informatica, IICS & AWS) and understand the current Transformation logic.
  • Move data from source systems into Vault CRM: take customer, account and product master data from Databricks (Delta tables in Unity Catalog) into Vault CRM, staying within the agreed schema, refresh schedule, data-quality and access rules.
  • Pick the right transfer method for each integration: Iceberg, Spark/PySpark or API, chosen by volume, latency and direction. This is Pattern 1 (Databricks to Vault CRM) in the new target patterns.
  • Map the data to Vault CRM: build the transformation and mapping logic in PySpark / Spark SQL, including incremental changes, data change requests (DCRs) and reference data.
  • Move existing integrations to the new patterns with matching results, reconciliation and no data loss.
  • Make integrations reliable: monitoring, data-quality checks, error handling, gap detection and reprocessing.
  • Represent integrations with Lilly: lead design reviews, working sessions and governance forums, and explain decisions, risks and dependencies clearly.
  • Work with Accenture and Veeva: collaborate with the other SI and with Veeva within Lilly’s dual-SI model.
Veeva CRM to Vault CRM Migration | Staffing Partner Summary
Role Overview

We need a senior, hands-on Databricks Integration Lead to own the integration architecture for a large migration from

Veeva CRM to Vault CRM. The scope covers 5 Veeva CRM orgs and 46 unique integration points that must be

analyzed, rationalized, and optimized before connecting to the new Vault CRM platform. The lead sets technical

direction, makes architecture decisions, and guides a skilled Databricks engineering team (including offshore

resources). Deep, current, hands-on Databricks expertise is the top screening criterion.

This is not an

oversight-only role.

Key Responsibilities
  • Own end-to-end integration architecture across Databricks, 5 Veeva CRM orgs, Vault CRM, and connected systems
  • Catalog and analyze all 46 integration points; consolidate, retire, or standardize rather than lift-and-shift
  • Define standards: medallion architecture, Delta Lake, Unity Catalog governance, workflow orchestration, CI/CD
  • Design coexistence patterns while Veeva CRM and Vault CRM run in parallel
  • Review designs, notebooks, and pipelines; act as technical escalation point for the team
  • Establish validation/reconciliation frameworks and support GxP, privacy, and audit requirements
  • Own technical cutover plan, rollback strategy, and monitoring/operational support model
  • This is not an oversight-only role.
Required Qualifications
  • 10+ years in data integration/architecture; 5+ years deep, hands-on Databricks in enterprise production
  • Expert in PySpark/Spark SQL, Delta Lake, Workflows/Jobs, Delta Live Tables, Auto Loader, Unity Catalog, performance and cost tuning
  • Delivered large multi-source integration programs (dozens of interfaces)
  • Hands-on Veeva CRM data model and integration experience; Vault CRM knowledge strongly preferred
  • Life sciences / pharma commercial experience (HCP/HCO data, call activity, samples, consent, MDM)
  • Strong REST/SOAP API, bulk load, and file/event-based integration skills
  • Proven technical leadership of engineering teams, including distributed/offshore teams
  • Familiarity with GxP validation and regulated-environment delivery
Preferred Qualifications
  • Databricks certifications (Data Engineer Professional, Solution Architect, or equivalent)
  • Prior Veeva CRM to Vault CRM or other CRM-to-CRM migration experience
  • Experience with Veeva Network, OpenData, or multi-org integration rationalization
Screening Guidance for Recruiters
  • Verify recent, hands-on Databricks work (last 2-3 years), not just architecture oversight or team management
  • Ask for specifics: Unity Catalog, DLT, Auto Loader, performance tuning, and cost management on real projects
  • Look for examples of rationalizing or consolidating many integrations, and of leading engineers who knew the tooling well
  • Prioritize candidates with both Veeva CRM and pharma commercial data backgrounds
  • Expect a technical panel interview with members of our Databricks team
Obtenez votre examen gratuit et confidentiel de votre CV.

ou faites glisser et déposez votre fichier ici.

Similar jobs

Postes similaires à comparer

Databricks Integration Lead
Databricks Integration Lead

New York Technology Partners • Indianapolis (IN)

Sur place
USD 180 000 - 240 000
Databricks Integration Lead (Veeva CRM)
Databricks Integration Lead (Veeva CRM)

New York Technology Partners • Indianapolis (IN)

Sur place
USD 180 000 - 230 000
Databricks Integration Lead: CRM Migration Architect
Databricks Integration Lead: CRM Migration Architect

New York Technology Partners • Indianapolis (IN)

Sur place
USD 180 000 - 230 000
Senior Databricks Integration Architect: Veeva to Vault Migration
Senior Databricks Integration Architect: Veeva to Vault Migration

New York Technology Partners • Indianapolis (IN)

Sur place
USD 180 000 - 240 000
Databricks Integration Architect: End-to-End Migration
Databricks Integration Architect: End-to-End Migration

New York Technology Partners • États-Unis

Sur place
USD 170 000 - 210 000
Veeva Vault Consultant
Veeva Vault Consultant

RADcube • Indianapolis (IN)

Sur place
USD 110 000 - 160 000
Veeva Vault Data Migration Specialist
Veeva Vault Data Migration Specialist

TechClub Inc • Summit (NJ)

Sur place
USD 110 000 - 130 000
Senior Migration Consultant - Clinical Data Programming
Senior Migration Consultant - Clinical Data Programming

Veeva Systems • États-Unis

Hybride
USD 100 000 - 160 000
Medical insurance
Dental insurance
Vision insurance
+5
Senior Migration Consultant - Clinical Data Programming
Senior Migration Consultant - Clinical Data Programming

Veeva • Boston (MA)

À distance
USD 100 000 - 160 000
Medical, dental, vision insurance
Flexible PTO and holidays
Retirement programs
+1
Senior Technical Lead – Veeva Vault RIM
Senior Technical Lead – Veeva Vault RIM

Cpl Life Sciences • États-Unis

Sur place
USD 140 000 - 190 000