Senior Data Engineer, Customer Data Platform (Remote)

NBME; Philadelphia (Philadelphia County); On-site; USD 127,337 - 160,000; Full time

Benefits offered by this job

Healthcare benefits

Dental and Vision plans

401(k) with match

Tuition Reimbursement Program

Commuter benefits

Job summary

NBME is seeking a Data Engineer to join a growing team building cloud-based data solutions on AWS for analytics, AI/ML, and decision support. You will develop data lakes, integration pipelines, feature stores, and scalable pipelines while partnering with data scientists and architects.

The role supports remote, hybrid, or onsite work within 50 miles of Philadelphia, with potential onsite interviews at the NBME Philadelphia office.

Qualifications

• Bachelor's Degree required.
• 7+ years of application development experience (Internship experience does not apply).
• 4+ years of experience in big data technologies.
• 4+ years' experience with cloud computing using AWS.
• 4+ years of experience in application development including Python, SQL, Scala, or Java.
• 4+ years' experience with distributed data/computing tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, MySQL).
• 4+ years of experience working on real‑time data and streaming applications.
• 4+ years of experience with NoSQL implementation (Mongo, Cassandra).
• 4+ years of data warehousing experience (Redshift).
• 6+ years of experience with UNIX/Linux including basic commands and shell scripting.
• 7+ years of experience with Agile engineering practices
• 7+ years of experience with SQL optimization
• 4+ years of experience with PySpark
• 3+ year of experience with process orchestration including AirFlow, KubeFlow, AWS step functions, or Luigi

Responsibilities

• Code, test, deploy, orchestrate, monitor, document, and troubleshoot cloud-based data engineering processes, feature stores, and vector databases in accordance with best practices and security standards throughout the development lifecycle.
• Partner closely with data scientists, AI researchers, data and enterprise architects, and business stakeholders to identify, extract, clean, and format structured and unstructured data for AI/ML model training, fine-tuning, and feature extraction.
• Lead evaluation, research, and experimentation efforts with batch and streaming data technologies, LLM data preparation frameworks, and MLOps tools to keep pace with industry innovation.
• Act as a technical lead to showcase the capabilities of emerging AI and data technologies, enabling the widespread adoption of modern data techniques across the organization.
• Significantly contribute to the definition and refinement of processes and procedures for the data engineering practice.
• Educate and develop ETL developers on data engineering cloud‑based initiatives to enable transition to data engineer and practice.
• Assures the integrity and accuracy of the corporate data, with particular attention to data security.
• Responsible for ensuring high data quality for Data Services, Analytics and Master Data Management.
• Helps coordinate technical solutions, takes responsibility for designs, development, testing and delivery of solutions.
• Build automated, scalable, test-driven data pipelines.
• Utilize software development practices such as version control via Git, CI/CD, and release management to enhance existing CI/CD pipelines in AWS.
• Collaborate with Data Engineers, DevOps engineers and architects on improvement opportunities for DataOps tools and frameworks.

Skills

Python

SQL

Scala/Java

AWS

PySpark

Unix/Linux

SQL optimization

Airflow

Distributed computing

Education

Bachelor's Degree

Tools

Hadoop

Spark

Kafka

EMR

Redshift

MySQL

Job description

NBME is seeking a Data Engineer to join a growing team building cloud-based data solutions on AWS for analytics, AI/ML, and decision support. You will develop data lakes, integration pipelines, feature stores, and scalable pipelines while partnering with data scientists and architects.

The role supports remote, hybrid, or onsite work within 50 miles of Philadelphia, with potential onsite interviews at the NBME Philadelphia office.