Experience: 3–8 years in Data Engineering or ETL Development
Core Areas: Data Pipelines, ETL/ELT, Python & SQL, Databricks & Spark, Data Modeling & Lakehouse Architecture, Workflow Orchestration, Data Quality & Governance, AI/ML Data Infrastructure
Compensation: $160,000–$190,000 USD annually
About the Role
This Data Engineer position focuses on designing, building, and maintaining reliable, scalable data pipelines that support deep learning, video processing, large language models (LLMs), analytics, and AI-powered applications. The role spans data ingestion, transformation, orchestration, and delivery, including real-time data feeds, batch processing, and analytics-ready datasets. Responsibilities include ensuring data accuracy, availability, discoverability, and performance across production data systems.
The position involves developing ETL/ELT workflows using Python, SQL, Databricks, Spark, and modern orchestration technologies. Working closely with MLOps, Sports Data, ML/AI, product, and analytics teams, this role supports model training, retrieval-augmented generation (RAG), vector search, and AI-driven products across multiple sports. The work includes structured sports statistics, scouting information, video metadata, and modern lakehouse data architectures.
What You'll Do
- Design, build, deploy, and maintain reliable data pipelines for ingestion, cleaning, transformation, and processing using Databricks, Airflow, Kubernetes, and related technologies.
- Develop and optimize ETL/ELT workflows with Python and SQL to support batch and real-time streaming data workloads.
- Build and maintain scalable data infrastructure that supplies reliable, discoverable datasets for deep learning, video systems, LLM applications, model training, and analytics.
- Partner with ML/AI teams to make datasets and data tools accessible and safe for autonomous AI agents, including evaluation processes and guardrails for AI-generated queries.
- Develop retrieval pipelines using retrieval-augmented generation (RAG) and vector search across structured sports statistics and unstructured sources, including scouting notes and video metadata.
- Design, model, and maintain structured data assets using Delta, Parquet, and Iceberg, supporting data reliability, versioning, and lineage tracking.
- Implement workflow orchestration and monitoring solutions to schedule data processing jobs, manage dependencies, detect failures, and automate recovery.
- Establish and maintain data quality and compliance controls through validation frameworks, schema enforcement, and audit logging.
- Collaborate with MLOps, Sports Data, product, and analytics teams to integrate data pipelines and datasets into AI applications and analytical workflows.
- Evaluate data engineering tools and technologies, establish platform best practices, and improve the developer experience.
- Optimize performance, scalability, and infrastructure costs across compute, storage, and workflow orchestration systems.
Qualifications
Required Experience
- 3–8 years of professional experience as a Data Engineer or ETL Developer working with production data systems.
- Experience developing, maintaining, and supporting data engineering workflows in a production environment.
Required Skills
- Strong proficiency in Python and SQL for data engineering, data transformation, and ETL/ELT development.
- Practical familiarity with Databricks, Apache Spark, or equivalent big-data processing frameworks.
- Experience with workflow orchestration technologies such as Apache Airflow, Dagster, Luigi, or Prefect.
- Deep understanding of data modeling, data warehousing, and distributed data processing.
- Knowledge of modern data lakehouse architectures and associated data engineering practices.
- Familiarity with CI/CD practices, GitHub Actions, Infrastructure as Code (IaC), and data pipeline testing frameworks.
- Experience working with cloud infrastructure such as AWS, Google Cloud Platform (GCP), or Microsoft Azure.
- Familiarity with containerization and orchestration technologies, including Docker and Kubernetes.
- Exposure to LLM-powered data tools and techniques, including text-to-SQL, retrieval-augmented generation (RAG), agent/tool interfaces such as Model Context Protocol (MCP), or natural-language analytics.
- Ability to work across functional teams involving machine learning, product development, and analytics.
Preferred Qualifications
- Previous experience developing data pipelines for sports data, telemetry, or sensor-generated information.
- Familiarity with streaming frameworks and event-driven data processing technologies such as Apache Kafka, Spark Structured Streaming, or Apache Flink.
- General knowledge of American football, the NFL, and college football.
- Background in data governance, data lineage, and observability tools such as Monte Carlo, Great Expectations, Unity Catalog, or OpenLineage.
- Experience designing semantic layers or defining metrics consumed by artificial intelligence (AI) and business intelligence (BI) tools.
- Exposure to machine learning model management practices and MLOps methodologies.
- 401(k) retirement savings plan with company matching contributions.
- Remote working environment.
- Flexible, unlimited paid time off policy.
- 13 paid holidays, including the Monday following the Super Bowl.