Data Engineer (Databricks, AWS)

CEI; North Carolina; On-site; USD 170,492,000 - 230,665,000; Full time

Benefits offered by this job

Medical, dental, vision insurance

401(k) match

Job summary

6AM City, LLC in Raleigh, NC is seeking a Data Engineer to design and optimize large-scale data pipelines onsite. The role emphasizes Databricks, AWS, PySpark, and pharma datasets, delivering scalable products for sales insights and forecasting.

You will collaborate with Commercial Analytics, Data Science, IT, and business teams to translate requirements into robust data solutions while ensuring governance and security standards.

Qualifications

• 5–10+ years of experience in Data Engineering or Big Data Analytics.
• Strong hands-on experience with Databricks, Genie, Delta Lake, and MLflow.
• Proficiency in PySpark, Python, SQL, and distributed computing.
• Deep experience with AWS services (S3, Glue, Lambda, EMR, Redshift, IAM).
• Experience with Life Sciences commercial datasets.
• Knowledge of data modeling, ETL/ELT, and cloud architecture.
• Familiarity with CI/CD, Git, and DevOps.
• Ability to build scalable, high-performance data pipelines.

Responsibilities

• Build, optimize, and maintain data pipelines using Databricks, Delta Lake, and Genie-powered workflows.
• Implement LLM-based automation and query generation with Databricks Genie for data exploration and self-service.
• Develop notebooks, jobs, workflows, and ML-ready datasets within Databricks.
• Design and manage data ingestion, storage, and processing using AWS services.
• Ensure data quality, lineage, governance, and compliance with industry standards.

Skills

Databricks

Genie

Delta Lake

MLflow

PySpark

Python

SQL

AWS

Life Sciences data

Data modeling

ETL/ELT

CI/CD

Git

DevOps

Automation

Big data analytics

Education

No Education Requirements

Tools

Databricks Unity Catalog

Tableau

Power BI

Qlik

Lakehouse architecture

Job description

Job Description

Title: Data Engineer (onsite)
Location: Raleigh, NC
Contract: W2 only, 12-month contract with potential for extension or conversion to full time with the client.
Pay: $70/hour + optional medical, dental, vision, 401(k) match

Overview

The Data Engineer will be responsible for designing, developing, and optimizing large-scale data pipelines and analytical solutions supporting Life Sciences Commercial Operations. This role requires deep expertise in Databricks (including Genie), AWS cloud services, and PySpark, with strong domain knowledge of pharmaceutical/biotech commercial datasets such as IQVIA, Symphony, DDD, NPA, Xponent, Claims, Specialty Pharmacy, and CRM/Sales data. The candidate will partner closely with Commercial Analytics, Data Science, and Business Stakeholders to deliver high-quality, scalable data products that enable sales insights, targeting, forecasting, incentive compensation, and field performance reporting.

Key Responsibilities

• Build, optimize, and maintain data pipelines using Databricks, Delta Lake, and Genie-powered workflows.
• Implement LLM-based automation and query generation using Databricks Genie for data exploration and business-user self-service.
• Develop notebooks, jobs, workflows, and ML-ready datasets within the Databricks environment.
• Design and manage data ingestion, storage, and processing using AWS services such as S3, Glue, Lambda, EMR, Redshift, and Athena.
• Implement secure, scalable architectures following AWS best practices (IAM, VPC, encryption, monitoring).
• Build distributed data transformation pipelines using PySpark for high-volume commercial datasets.
• Optimize PySpark jobs for performance, cost efficiency, and reliability.
• Implement unit testing, CI/CD, and code versioning using Git-based workflows.
• Ingest, harmonize, and model datasets including IQVIA (Xponent, DDD, NPA, LAAD, NSP), Claims (medical, pharmacy), Specialty Pharmacy data feeds, Sales & CRM (Veeva, Salesforce), Roster, Territory, and Alignment datasets.
• Build commercial data marts supporting sales reporting and dashboards, targeting and segmentation, incentive compensation, forecasting, and field performance insights.
• Work with Commercial Analytics, Data Science, IT, and Business teams to translate requirements into scalable data solutions.
• Ensure data quality, lineage, governance, and compliance with industry standards (HIPAA, GxP, SOC2).

Required Skills

• 5–10+ years of experience in Data Engineering or Big Data Analytics.
• Strong hands-on experience with Databricks, Genie, Delta Lake, and MLflow.
• Advanced proficiency in PySpark, Python, SQL, and distributed computing.
• Deep experience with AWS (S3, Glue, Lambda, EMR, Redshift, IAM).
• Proven experience working with Life Sciences Commercial datasets.
• Strong understanding of data modeling, ETL/ELT, and cloud architecture.
• Experience with CI/CD, Git, DevOps, and automated workflow orchestration.
• Demonstrated knowledge of ingesting, harmonizing, and modeling pharmaceutical/biotech datasets.
• Ability to build scalable, high-performance data pipelines.
• Strong analytical and problem-solving skills.
• Excellent communication and documentation abilities.
• Ability to work in fast-paced, cross-functional environments with high attention to data accuracy.

Required Education

• No Education Requirements

Preferred Skills

• Experience with Databricks Unity Catalog and Lakehouse architecture.
• Familiarity with LLM-based automation or AI-assisted analytics.
• Experience supporting Commercial Operations, Sales Leadership, and Field Teams.
• Knowledge of incentive compensation methodologies and targeting algorithms.
• Experience with BI tools (Tableau, Power BI, Qlik).

Why Should I Apply?

This role offers the opportunity to work on cutting-edge data solutions supporting Life Sciences Commercial Operations, with significant impact on sales insights and forecasting. Join a dynamic team leveraging advanced cloud and big data technologies to transform pharmaceutical data analytics.

About CEI

As a trusted technology partner, CEI delivers solutions that help our customers transform their business and achieve meaningful results. From strategy and custom application development through application management - our technology and digital experience services are tailored to meet each unique need of our customers. Our staffing solutions bring specialized skills to complement our customers' workforce and project requirements.

#ZR
#INDGEN