Senior Data Engineer: AI-Driven Data Pipelines

Quantum Technologies. LLC; Dallas (TX); On-site; USD 130,000 - 190,000; Full time

Job summary

Quantum Technologies LLC is seeking a Senior Data Engineer to design, build, and optimize scalable data ingestion and processing pipelines across SQL, Spark, and big data ecosystems. You will collaborate with backend, BI, and ML teams while leveraging AI tooling to accelerate delivery and code quality.

The role emphasizes ownership, data quality, and on-call reliability, with exposure to event-driven architectures and modern data warehouse platforms.

Qualifications

• Experience designing scalable data pipelines and ETL/ELT workflows.

Responsibilities

• Design, build, and maintain scalable data ingestion and processing pipelines using SQL , Spark, Hadoop, Airflow, SnapLogic, or similar tools
• Optimize existing ETL/ELT workflows and resolve bottlenecks in production jobs
• Integrate multiple structured and unstructured data sources from internal and external systems
• Design and implement event-driven data pipelines using event bus architectures (Kafka, AWS EventBridge, SNS/SQS, or similar)
• Build and maintain event producers, consumers, and stream processors for real-time and near-real-time data flows
• Manage event schema design, schema registry, and versioning to ensure backward compatibility across consumers
• Monitor event bus health, lag, throughput, and reliability; respond to failures before SLA breach
• Implement and manage CI/CD pipelines for data workflows using Git, Jenkins, or GitHub Actions
• Ensure robust job scheduling, monitoring, logging, and end-to-end testing of data pipelines
• Debug and resolve production issues before SLA breach; participate in on-call rotation
• Collaborate with backend and BI teams for API and data integration
• Design and develop schema definitions; support data warehouse/mart to integrate disparate data sources and make them available for analysis
• Support large data volumes and flexible provisioning of new data sources
• Contribute to data architecture design across the data landscape
• Write clean, reusable, well-tested Python and Shell code across the full software development lifecycle
• Gather functional requirements, develop technical specifications, and own project and test planning
• Work in Agile/Scrum environments; help align team priorities across the organization
• Leverage AI-assisted dev tools (Claude, Cursor .. etc) to accelerate delivery and code quality

Skills

Python

Advanced SQL

PySpark

Scala

Unix scripting

Airflow

SnapLogic

DBT

Kafka

EventBridge

SQS

Kinesis

Snowflake

Delta Lake

Redshift

BigQuery

Git

Jenkins

GitHub Actions

Docker

APIs

Data quality

NoSQL

Neo4j

GraphQL

AI tooling

Tools

Snowflake

Delta Lake

Redshift

BigQuery

Airflow

Job description

Quantum Technologies LLC is seeking a Senior Data Engineer to design, build, and optimize scalable data ingestion and processing pipelines across SQL, Spark, and big data ecosystems. You will collaborate with backend, BI, and ML teams while leveraging AI tooling to accelerate delivery and code quality.

The role emphasizes ownership, data quality, and on-call reliability, with exposure to event-driven architectures and modern data warehouse platforms.