Senior Data Engineer - Cloud Data Pipelines & Dashboards

Valiant Technologies LLC; Raleigh (NC); In loco; USD 140.000 - 180.000; Tempo pieno

Descrizione del lavoro

Valiant Technologies LLC seeks an experienced Data Engineer to design and optimize enterprise data pipelines and cloud-based solutions. You will build scalable architectures, integrate diverse data sources, and enable executive analytics with dashboards and models.

Work closely with data scientists, policy analysts, and cross-functional teams to drive data-informed decisions while ensuring governance, privacy, and security compliance.

Competenze

• Data engineer with 12+ years’ experience in SQL and relational databases.
• Experience building and optimizing big data pipelines and datasets.
• Designing dashboards, reports, models, and decision-support for executives and non-technical users.
• Experience with student-level higher education data and linking sources to measure outcomes.
• Root-cause analysis on data and processes to drive improvements.
• Strong analytics with unstructured data.
• Experience building data transformation processes and metadata management.
• History of handling large, disconnected datasets.
• Knowledge of message queues, stream processing, and scalable data stores.
• Strong project management and ability to work cross-functionally.

Mansioni

• Create and maintain optimal data pipeline architecture.
• Design and implement cloud-based data solutions using AWS services.
• Build and maintain data orchestration workflows (Airflow, Step Functions).
• Assemble large, complex data sets meeting business requirements.
• Engineer environments for advanced statistical analysis, predictive modeling, and ML.
• Collaborate with data scientists and policy analysts to support data assets.
• Identify and implement internal process improvements for scalability.
• Develop ETL/ELT infrastructure for diverse data sources and big data tech.
• Build analytics tools to provide actionable insights on outcomes and metrics.
• Develop data pipeline monitoring, alerts, and logging frameworks.
• Implement data quality checks and validation across pipelines.
• Create dashboards and reports to support decision making.
• Support stakeholders with data infrastructure needs and issue resolution.
• Ensure compliance with data governance, privacy, and security standards.
• Create tools to aid analytics and data science teams.

Conoscenze

Advanced SQL

Relational databases

Big data pipelines

Dashboards/reports/analytical models

Education data linkage

Root-cause analysis

Unstructured data analysis

Data transformation processes

Large dataset handling

Message queueing/stream processing

Cross-functional collaboration

Project management

Strumenti

AWS

Apache Airflow

AWS Glue

Redshift

EMR

Athena

Kinesis

Lambda

Descrizione del lavoro

Valiant Technologies LLC seeks an experienced Data Engineer to design and optimize enterprise data pipelines and cloud-based solutions. You will build scalable architectures, integrate diverse data sources, and enable executive analytics with dashboards and models.

Work closely with data scientists, policy analysts, and cross-functional teams to drive data-informed decisions while ensuring governance, privacy, and security compliance.