BI Data Engineer

Agile; Orlando (FL); Remoto; USD 83.000 - 96.000; Tempo pieno

Vantaggi offerti da questo lavoro

Medical

Dental

Vision

Descrizione del lavoro

Agile Resources, Inc. seeks a Sr. Data Engineer to lead the evolution of a global data ecosystem. You will own production-grade data pipelines, enable LLM-ready data, and drive cloud-scale architectures in a fully remote role.

You will design secure data platforms, work across multi-cloud environments, and partner with cross-functional teams to deliver scalable, enterprise-ready solutions. Strong Python/SQL, Spark, and MLOps experience are required.

Competenze

• 7+ years of sophisticated data engineering experience.
• Proficient in Python and SQL with ML lifecycle management.
• Expertise in Spark tuning, partitioning, and cloud infrastructure.
• Experience designing secure, ACID-compliant storage layers and unified cataloging tools.
• Familiarity with event-driven architectures (e.g., Kafka) and CI/CD pipelines.

Mansioni

• Architect and optimize high-throughput distributed computing frameworks and Lakehouse structures.
• Build production-grade features and pipelines for LLMs and ML models.
• Create automated ingestion across multi-cloud with streaming and batch processing.
• Ensure data quality, cataloging, and access controls while optimizing costs.
• Collaborate with leadership and operations to translate goals into scalable technical solutions.

Conoscenze

Spark tuning

Partitioning

Cloud infrastructure

Python

SQL

ML lifecycle

LLM integrations

Data governance

Kafka

CI/CD

IaC

Strumenti

Unity Catalog

Delta Lake

Lakehouse

Kafka

Apache Spark

Databricks

MLflow

Descrizione del lavoro

Sr. Data Engineer

Location: Fully Remote
Type: 12-month Contract with Potential to Extend
Compensation: $60/hr - $70/hr

We are partnering with an industry leader in the construction supplies and equipment space to find a Sr. Data Engineer to spearhead the evolution of their global, next-generation data ecosystem. If you thrive on solving intricate, massive-scale data puzzles, optimizing bleeding-edge distributed computing environments, and laying the groundwork for sophisticated machine learning and LLM operations, this is your next definitive career move. In this role, you will act as the architect behind a mission-critical platform, transforming raw datasets into powerful, secure, and production-ready intelligence that fuels executive-level decisions. You will enjoy a high degree of technical ownership, bridging the gap between advanced cloud engineering and real-world AI applications.

Responsibilities

• Architect and optimize high-throughput, distributed computing frameworks and modern Lakehouse structures to handle massive data velocity and volume.
• Construct robust, production-grade features and pipelines that seamlessly operationalize large language models (LLMs) and predictive machine learning models.
• Build automated ingestion frameworks across multi-cloud environments, utilizing both real-time streaming and scheduled batch processing.
• Establish solid data quality, unified cataloging, and access controls while aggressively optimizing cluster performance and cloud infrastructure costs.
• Collaborate with cross-functional leadership, finance, and operations teams to convert complex strategic goals into highly scalable technical solutions.

Ideal Candidate

• 7+ years of sophisticated data engineering experience, highlighted by mastery-level knowledge of Spark tuning, partitioning, and cloud infrastructure.
• Extensive hands-on experience designing secure, ACID-compliant storage layers, ideally utilizing modern unified cataloging tools.
• Expertise in Python and SQL, coupled with practical experience supporting ML lifecycle management (such as tracking, feature stores, or LLM integrations).
• A proven track record of integrating disparate, complex data sources and maintaining high-availability production environments at enterprise scale.
• Familiarity with event-driven architectures (like Kafka), continuous integration/continuous deployment (CI/CD) pipelines, and infrastructure-as-code will be a plus.

Benefits

• Medical
• Dental
• Vision

Keywords: Lead Data Engineer, Senior Data Engineer, Databricks, Apache Spark, Delta Lake, Lakehouse Architecture, Python, SQL, Cloud Data Architect, MLOps, Data Pipelines, Spark Tuning, Unity Catalog, Kafka, Event Hubs, MLflow, LLM Infrastructure, Data Governance #LI-LS1