Senior Data Engineer, AI & BI Pipelines

Lever, Inc.; Puerto Rico; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Remote work

B2B employment (gross)

Long-term growth

Quality hardware

Learning & growth opportunities

Training & conferences

Referral program

English classes

Job summary

IDT is seeking a Senior Data Engineer to join our BI team and own end-to-end data pipelines powering our data warehouse, LLM-driven applications, and AI-based BI. You will design, build, and maintain scalable pipelines, optimize extraction and storage of semantic data, and support RAG-focused workflows.

The role emphasizes SQL/PLSQL, Spark, Kafka, and cloud-based data services like Snowflake/Redshift, with a focus on NLQ experiences and AI insights.

Qualifications

• 8+ years of experience as a Data Engineer.
• Experience with Python for data engineering tasks and large-scale processing.
• Hands-on with Spark, Hadoop, and Kafka for distributed processing and real-time ingestion.
• Experience designing pipelines from RDBMS, JSON, API, and flat files.
• Strong SQL/PLSQL skills and BI/data warehousing knowledge; experience with Snowflake/Redshift.

Responsibilities

• Design, build, and maintain scalable data pipelines for ingestion, transformation, and delivery to feature stores, model-training workflows, and real-time services.
• Develop workflows to index and retrieve unstructured data for advanced search and retrieval patterns.
• Architect lightweight analytics and dashboards providing NLQ experiences and AI-powered insights.
• Define processes for prompt engineering, orchestration, and model fine-tuning for conversational interfaces.
• Manage vector data stores and indexing for retrieval-augmented generation workflows.
• Collaborate with data stakeholders to translate requirements into scalable solutions.
• Create documentation for data processes, workflows, and model deployments.
• Stay informed on emerging data engineering, MLOps, and LLM operations.

Skills

Python

Apache Spark

Kafka

SQL/PLSQL

Data pipelines

Snowflake

Redshift

Tools

Snowflake

Redshift

Job description

IDT is seeking a Senior Data Engineer to join our BI team and own end-to-end data pipelines powering our data warehouse, LLM-driven applications, and AI-based BI. You will design, build, and maintain scalable pipelines, optimize extraction and storage of semantic data, and support RAG-focused workflows.

The role emphasizes SQL/PLSQL, Spark, Kafka, and cloud-based data services like Snowflake/Redshift, with a focus on NLQ experiences and AI insights.