Senior Data Engineer - Big Data Pipelines & Insights

Quantum Technologies. LLC; Raleigh (NC); On-site; USD 140,000 - 190,000; Full time

Job summary

Quantum Technologies. LLC in Raleigh, NC is seeking a seasoned Data Engineer to architect and optimize data pipelines, build scalable data stores, and design dashboards for executive audiences.

You will collaborate with cross-functional teams, ensure data quality and governance, and implement cloud-based solutions using AWS services. Strong SQL, big data tech, Airflow, and education/workforce data experience are highly valued.

Qualifications

• Data engineer with 12+ years of experience in Advanced SQL and relational databases.
• Experience building and optimizing big data pipelines and datasets.
• Skilled in designing dashboards, reports, and analytical models for executive and non-technical users.
• Experience with student-level higher education data and linking sources to measure outcomes.
• Proficient in root-cause analysis and data-driven problem solving.
• Strong skills with unstructured data.
• Experience building data transformations, metadata, dependencies and workload management.
• Experience with messaging, streaming, and scalable big data stores.
• Strong project management and ability to work with cross-functional teams.
• Experience supporting cross-functional teams in dynamic environments.

Responsibilities

• Create and maintain optimal data pipeline architecture.
• Design and implement cloud-based data solutions using AWS services (S3, Glue, Redshift, EMR, Athena, Kinesis, Lambda).
• Build and maintain data orchestration workflows with Apache Airflow or AWS Step Functions.
• Assemble large, complex data sets meeting business requirements.
• Engineer analytics environments supporting statistical analysis, predictive modeling, ML, and evidence-building activities.
• Partner with data scientists and policy analysts to ensure reproducible research.
• Identify and implement internal process improvements for scalability.
• Build data extraction, transformation, loading pipelines from diverse sources with SQL and big data tech.
• Create analytics tools using the data pipeline for insights on education and workforce outcomes.
• Develop dashboards, reports, models, and decision-support products.
• Collaborate with stakeholders to address data infrastructure needs.
• Ensure compliance with data governance, privacy, security requirements.
• Create data tools for analytics and data science teams.
• Improve data system functionality with analytics teams.

Skills

Advanced SQL

Relational databases

SQL query authoring

Big data pipelines

Dashboard design

Higher education data

Root-cause analysis

Unstructured data

Data transformation

Metadata management

Workload management

Message queuing

Stream processing

Big data stores

Project management

Cross-functional collaboration

Tools

Apache Airflow

AWS Step Functions

Job description

Quantum Technologies. LLC in Raleigh, NC is seeking a seasoned Data Engineer to architect and optimize data pipelines, build scalable data stores, and design dashboards for executive audiences.

You will collaborate with cross-functional teams, ensure data quality and governance, and implement cloud-based solutions using AWS services. Strong SQL, big data tech, Airflow, and education/workforce data experience are highly valued.