Senior Data Engineer, Customer Data Platform (Remote)

National Board of Medical Examiners; Philadelphia (Philadelphia County); Hybrid; USD 127,000 - 160,000; Full time

Benefits offered by this job

Healthcare

Dental

Prescription

Vision plans

401(k) w/match

Tuition Reimbursement Plan

Commuter Benefit: Public Transit or...

Remote Friendly Workplace

Job summary

National Board of Medical Examiners (NBME) is seeking a Data Engineer to join a team of data engineers and BI developers. You will build data lakes, pipelines, and scalable data solutions on AWS to support analytics, AI/ML, and decision-making for internal staff and external customers.

You will collaborate with data scientists, engineers, and stakeholders to prepare structured and unstructured data for model training and feature extraction, while advancing data engineering practices and

Qualifications

• Bachelor's Degree required.
• 7+ years of application development experience.
• 4+ years in big data technologies.
• 4+ years with AWS cloud.
• 4+ years in application development (Python/SQL/Scala/Java).
• 4+ years with distributed data tools (MapReduce/Hadoop/Hive/EMR/Kafka/Spark/MySQL).
• 4+ years in real-time data and streaming apps.
• 4+ years with NoSQL (Mongo/Cassandra).
• 4+ years data warehousing (Redshift).
• 6+ years UNIX/Linux experience.
• 7+ years Agile engineering practices.
• 7+ years SQL optimization.
• 4+ years PySpark.
• 3+ years orchestration (AirFlow/KubeFlow/AWS Step Functions/Luigi).
• Preferred: Generative AI, LLM data prep pipelines, vector databases, feature stores, APIs, and DaaS.

Responsibilities

• Code, test, deploy, orchestrate, monitor, document, and troubleshoot cloud-based data engineering processes, feature stores, and vector databases.
• Collaborate with data scientists, AI researchers, data and enterprise architects, and stakeholders to prepare data for AI/ML model training and features.
• Lead evaluation and experimentation with batch/streaming data techs and MLOps to stay aligned with industry innovation.
• Act as a technical lead to showcase AI and data tech capabilities across NBME.
• Refine processes for the data engineering practice.
• Educate ETL developers on cloud-based data engineering initiatives.
• Ensure data integrity and security across corporate data.
• Guarantee high data quality for Data Services, Analytics and MDM.
• Coordinate solutions and own designs, development, testing, delivery.
• Build automated, scalable, test-driven data pipelines.
• Utilize Git, CI/CD, and release management to enhance AWS pipelines.
• Collaborate on DataOps tool/framework improvements.

Skills

Python

SQL

Scala

Java

PySpark

Spark

Hadoop

Kafka

Redshift

MongoDB

Cassandra

Unix/Linux

AirFlow

KubeFlow

AWS

Education

Bachelor's Degree

Tools

Git

CI/CD

EMR

Linux scripting

Job description

National Board of Medical Examiners (NBME) is seeking a Data Engineer to join a team of data engineers and BI developers. You will build data lakes, pipelines, and scalable data solutions on AWS to support analytics, AI/ML, and decision-making for internal staff and external customers.

You will collaborate with data scientists, engineers, and stakeholders to prepare structured and unstructured data for model training and feature extraction, while advancing data engineering practices and