Data Engineer: Scalable Pipelines for AI & Analytics

SMX; Town of Hanover (NY); On-site; USD 103,000 - 172,000; Full time

Benefits offered by this job

Health insurance

Retirement

Paid leave

Job summary

SMX is seeking a Data Engineer to design, build, and maintain scalable data architectures powering AI, analytics, and reporting. You will transform raw, unstructured data into high-quality pipelines, collaborating with Analysts, AI Engineers, and Software Engineers across cloud and on-prem environments.

The role emphasizes data reliability, performance, and secure, governed datasets for enterprise-wide use, involving cross-team partnerships and production-ready data delivery.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Data Engineering.

Responsibilities

• Design, construct, install, test, and maintain scalable data management systems and robust ELT/ETL pipelines across cloud and on-prem systems.
• Build infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using cloud technologies.
• Implement monitoring systems to ensure data integrity, quality, and security across all storage and processing layers. Implement data quality checks, validation, lineage tracking, and metadata management.
• Prepare and optimize data for AI/ML models, semantic search, analytics, and mission applications.
• Identify, design, and implement internal process improvements, such as automating manual processes and optimizing data delivery for greater scalability. Automate ingestion and transformation processes using industry-standard tools and patterns. Troubleshoot pipeline issues and optimize for performance and cost.
• Work with analytics and business teams to understand their data requirements and deliver production-ready datasets. Collaborate with AI engineers and analysts to ensure datasets meet requirements.

Skills

Python

SQL

Java

Data modeling

Security clearance

Education

Bachelor's degree in Computer Science, Information Technology, Data Engineering

Tools

Snowflake

AWS Redshift

Spark

Databricks

Kafka

Azure Data Factory

AWS Glue

Apache Airflow

AWS

GCP

Azure

Job description

SMX is seeking a Data Engineer to design, build, and maintain scalable data architectures powering AI, analytics, and reporting. You will transform raw, unstructured data into high-quality pipelines, collaborating with Analysts, AI Engineers, and Software Engineers across cloud and on-prem environments.

The role emphasizes data reliability, performance, and secure, governed datasets for enterprise-wide use, involving cross-team partnerships and production-ready data delivery.