Principal Data Engineer

Insight Global; San Diego (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Insight Global seeks a Principal Data Engineer in San Diego to design and implement scalable data solutions for research data initiatives. You will lead architectural decisions, mentor engineers, and collaborate with analysts and platform teams to deliver reliable, governed data assets using Databricks, Spark, and lakehouse architecture.

The role emphasizes strong Python/SQL skills, production ETL/ELT pipelines, and cloud experience (Azure/AWS) within a regulated Biopharma environment.

Qualifications

• 7+ years as Data Engineer in Biopharma/Life Science.
• Expert in building scalable data solutions with Databricks, Spark, Delta Lake.
• Strong Python and SQL, Spark-based processing, performance tuning.
• Experience designing/operating production ETL/ELT data pipelines.
• Hands-on in Azure and/or AWS cloud data environments.
• Proven technical leadership, mentoring engineers, and partnering with analytics teams.
• Experience with LIMS, ELN, or scientific data sources.
• Knowledge of data governance, security, privacy in regulated environments.
• Experience with Snowflake or other cloud data warehouses with Databricks.

Responsibilities

• Lead architecture decisions for research data platforms and guide engineers.
• Design, build, and optimize scalable data solutions for research initiatives.
• Mentor teams and set standards for data engineering practices across the org.
• Collaborate with business analysts, platform engineers, and researchers to translate data needs into governed assets.

Skills

Databricks

Spark

Delta Lake

Unity Catalog

lakehouse architecture

data quality

version control

performance optimization

Python

SQL

ETL/ELT pipelines

cloud data environments

Azure

AWS

LIMS/ELN data sources

data governance & security

Tools

Snowflake

Job description

Job Description

a pharmaceutical company based in San Diego is seeking a Principal level Data Engineer to join their team in support of their Research Data Solutions. The Principal Data Engineer is a senior technical leader responsible for designing, building, and optimizing scalable data solutions that support research and scientific data initiatives. This role requires deep expertise in modern data engineering practices, with a strong emphasis on Databricks, Spark, Delta Lake, lakehouse architecture, data quality, version control, performance optimization, and engineering best practices. The Principal Data Engineer partners primarily with business analysts, platform engineers, and other data engineers to translate research data needs into reliable, governed, and reusable data assets. This individual serves as a hands-on technical expert, architectural advisor, and mentor, helping establish standards and patterns for research-focused data engineering across the organization.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [e-mail]. To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

• 7+ years of experience as Data Engineer, with proven knowledge or experience in the Biopharma/Life Science industry.

• Expert-level experience architecting and building scalable data solutions using Databricks, including Spark, Delta Lake, Unity Catalog, and lakehouse best practices.

• Advanced proficiency in Python and SQL, with deep experience in Spark-based data processing, performance tuning, and production analytics workloads.

• Strong experience designing and operating production ETL/ELT data pipelines, including orchestration, incremental processing, monitoring, and reliability.

• Hands-on experience working in cloud data environments (Azure and/or AWS) supporting enterprise-scale data platforms.

• Proven technical leadership experience, including leading architecture decisions, mentoring engineers, and partnering with analytics, platform, and research teams. - Hands-on experience with LIMS, ELN, or scientific/discovery data sources.

• Strong understanding of data governance, security, privacy, and compliance practices in regulated environments.

• Experience working with Snowflake or other cloud data warehouses alongside Databricks.