Sr. Data & AI / LLM Engineer

Beacon Technologies, Inc

United States

Remote

USD 150,000 - 230,000

Full time

8 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Beacon Technologies, Inc. seeks a Senior Data & AI / LLM Engineer to design, build, and operate data, ML, and Generative AI solutions for the insurance domain.

You will develop scalable data pipelines, ML/LLM models, and enterprise AI applications using Azure, Databricks, and modern DevOps practices. You will collaborate with data scientists and engineers to productionize models, implement RAG workflows, and ensure data governance and security.

Qualifications

  • Strong professional experience in data engineering, analytics, ML, AI, or software engineering.
  • Advanced programming in Python and SQL.
  • Experience with Databricks, Spark/PySpark, and Delta Lake.
  • Design and support production-scale ETL/ELT pipelines.
  • Data analysis on large, complex datasets.
  • Knowledge of statistical analysis and ML techniques.
  • Hands-on experience with LLMs and Generative AI.
  • Experience with Azure cloud and AI services.
  • Familiarity with Git, CI/CD, and DevOps practices.

Responsibilities

  • Design, build, test, and maintain scalable data pipelines and data products using Databricks, Spark/PySpark, Delta Lake, SQL, and Python.
  • Perform exploratory and advanced data analysis to identify patterns and opportunities in insurance data.
  • Build reliable ingestion, transformation, integration, and data-quality processes across structured and unstructured data.
  • Work with large-scale datasets across cloud data platforms, APIs, files, databases, and enterprise apps.
  • Apply data quality, lineage, observability, metadata, security, and governance.
  • Optimize pipelines for reliability, performance, scalability, and cost.
  • Develop, evaluate, deploy, and monitor ML models using Python and ML frameworks.
  • Translate business problems into analytical or ML approaches and productionize models with data scientists.
  • Design and implement enterprise LLM applications (RAG, retrieval, embeddings, vector search, grounding).
  • Use AI development tools (GitHub Copilot, Claude) to accelerate engineering delivery.
  • Adhere to CI/CD, DevOps, testing, and production deployment practices.

Skills

Python
SQL
Databricks
Spark/PySpark
DL/Delta Lake
LLMs/Generative AI
Azure
Git/CI-CD/DevOps
Production deployment

Education

Bachelors/Masters in CS/DS/AI/Engineering

Tools

Azure Databricks
Delta Lake
GitHub Copilot
Claude
Vector search

Job description

Beacon is seeking an experienced Senior Data & AI / LLM Engineer to design, build, and operationalize modern data, machine learning, and Generative AI solutions within the insurance domain. This is a hands‑on engineering role suited to someone who can work across the full lifecycle, from understanding insurance business problems and analyzing data, through building scalable data pipelines and ML/LLM solutions, to deploying production-quality code using Azure and DevOps practices. The successful candidate will also be expected to embrace modern AI-assisted software engineering, using tools such as GitHub Copilot and Claude to accelerate development, refactoring, testing, documentation, and maintenance while maintaining appropriate engineering, security, and code-quality standards.

Key Responsibilities:
  • Data Engineering & Analytics
  • Design, build, test, and maintain scalable data pipelines and data products using Databricks, Apache Spark/PySpark, Delta Lake, SQL, and Python.
  • Perform exploratory and advanced data analysis to identify patterns, anomalies, trends, relationships, and opportunities within complex insurance datasets.
  • Build reliable ingestion, transformation, integration, and data-quality processes across structured and unstructured data.
  • Work with large‑scale datasets across cloud data platforms, APIs, files, databases, and enterprise applications.
  • Apply sound practices around data quality, lineage, observability, metadata, security, and governance.
  • Optimize pipelines and workloads for reliability, performance, scalability, and cost.
  • Machine Learning & AI
  • Develop, evaluate, deploy, and monitor machine learning models using Python and appropriate ML frameworks.
  • Understand core ML techniques including supervised and unsupervised learning, classification, regression, clustering, feature engineering, model evaluation, and model monitoring.
  • Translate business problems into analytical or machine‑learning approaches and determine when traditional analytics, ML, or Generative AI is most appropriate.
  • Work with data scientists, business SMEs, and engineering teams to productionize models and analytical solutions.
  • Generative AI & LLM Engineering
  • Design and implement enterprise LLM applications, including Retrieval‑Augmented Generation (RAG), document processing, summarization, extraction, classification, conversational assistants, and agentic workflows.
  • Develop solutions using LLM APIs and frameworks, with experience in areas such as prompt engineering, embeddings, vector search, chunking, grounding, tool/function calling, and structured outputs.
  • Understand evaluation techniques for LLM solutions, including accuracy, relevance, hallucination risk, latency, cost, security, and responsible AI considerations.
  • Integrate structured enterprise data and unstructured content into secure and governed AI solutions.
  • Build reusable AI components and APIs that can be incorporated into broader enterprise applications.
  • AI‑Assisted Software Engineering
  • Effectively use modern AI development tools such as GitHub Copilot, Claude, and similar coding assistants to accelerate engineering delivery.
  • Use AI tools to help create, refactor, debug, test, document, review, and maintain code.
  • Apply engineering judgment when using AI-generated code, including validating functionality, security, performance, maintainability, and compliance with coding standards.
  • Demonstrate an AI‑first engineering mindset, identifying opportunities to automate repetitive development, analysis, testing, and documentation activities rather than simply using AI as an autocomplete capability.
  • Azure, DevOps & Production Engineering
  • Build and deploy solutions using the Microsoft Azure technology stack, with relevant experience across services such as Azure Databricks, Azure Storage/Data Lake, Microsoft Entra ID, Key Vault, Azure Functions, APIs, Azure AI services, and related cloud services.
  • Work within modern CI/CD and DevOps practices using tools such as Azure DevOps and/or GitHub.
  • Use Git‑based source control, branching, pull requests, automated testing, build/release pipelines, infrastructure configuration, and environment management.
  • Design solutions with appropriate consideration for security, identity, secrets management, logging, monitoring, resilience, scalability, and production support.
  • Partner with architecture, cybersecurity, data governance, infrastructure, and application teams to ensure solutions meet enterprise technology standards.
  • Insurance Domain Knowledge
  • The ideal candidate should have experience working within insurance, financial services, or another highly regulated data‑intensive industry, with insurance experience strongly preferred.
  • Relevant insurance knowledge may include:
  • Policy, customer, broker, submission, quote, bind, premium, exposure, risk, claims, loss, and underwriting data.
  • Commercial and/or specialty insurance processes.
  • Underwriting and pricing workflows.
  • Claims analytics and automation.
  • Customer and broker analytics.
  • Submission ingestion, document extraction, triage, and risk assessment.
  • Insurance data quality, lineage, privacy, regulatory, and governance considerations.
  • The engineer should be able to engage with underwriters, actuaries, claims professionals, operations teams, data analysts, and technology teams and translate business needs into appropriate data and AI solutions.
Required Skills & Experience:
  • Strong professional experience in data engineering, analytics, machine learning, AI, or software engineering.
  • Advanced programming skills in Python and SQL.
  • Strong experience with Databricks, Spark/PySpark, and Delta Lake.
  • Experience designing and supporting production‑scale ETL/ELT and data pipelines.
  • Practical experience performing data analysis on large and complex datasets.
  • Working knowledge of statistical analysis and machine‑learning techniques.
  • Hands‑on experience developing solutions using LLMs and Generative AI.
  • Experience with RAG, embeddings, vector search, prompt engineering, evaluation, or related LLM patterns.
  • Experience with the Microsoft Azure cloud ecosystem.
  • Strong understanding of Git, CI/CD, DevOps, testing, code reviews, and production deployment practices.
  • Experience using AI coding assistants such as GitHub Copilot, Claude, or comparable tools as part of day‑to‑day software engineering.
  • Ability to work independently from an ambiguous business problem through analysis, technical design, implementation, testing, deployment, and production support.
Preferred Qualifications:
  • Prior experience in P&C, commercial, specialty, or reinsurance environments.
  • Experience with Azure Databricks and Azure AI services.
  • Experience operationalizing ML models using MLflow/MLOps practices.
  • Experience with vector databases/vector search and enterprise RAG architectures.
  • Experience working with insurance documents and unstructured datasets.
  • Experience building APIs and microservices supporting data and AI applications.
  • Familiarity with data governance, data cataloging, lineage, model governance, and Responsible AI practices.
  • Experience with Infrastructure as Code and automated cloud deployments.
  • Bachelor's or Master's degree in Computer Science, Data Science, Artificial Intelligence, Engineering, Mathematics, or a related discipline, or equivalent professional experience.
About Beacon Technologies

Beacon Technologies offers career advancement opportunities, extensive training, and excellent benefits including paying for health and dental premiums for salaried employees. In addition to providing interesting opportunities, Beacon Technologies provides that old fashioned, personal touch, so you feel like a part of the Beacon team.

Beacon Technologies, Inc. is an equal employment opportunity employer with a functioning Affendant Plan. It is the policy of Beacon Technologies, Inc. to ensure equal employment opportunity without discrimination or harassment on the basis of race, color, religion, sex, sexual orientation, gender identity or expression, age, disability, marital status, citizenship, national origin, genetic information, ethnicity, ancestry, disability, medical condition, military and veteran status, or any other characteristic protected by law. Beacon Technologies, Inc. prohibits any such discrimination or harassment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer, Artificial Intelligence/LLM
Senior Software Engineer, Artificial Intelligence/LLM

Beacon AI • San Carlos (CA)

Hybrid
USD 170,000 - 210,000
Healthcare premiums covered
3 weeks PTO + holidays
401(k) with future enhancements
Software Engineer, Artificial Intelligence/LLM (Multiple Seniority Levels)
Software Engineer, Artificial Intelligence/LLM (Multiple Seniority Levels)

Beacon AI • San Carlos (CA)

Hybrid
USD 100,000 - 140,000
Healthcare premiums covered
3 weeks PTO and 13+ paid holidays
Monthly stipends for phone and wellness
+1
Staff Software Engineer, Cloud Infrastructure Software
Staff Software Engineer, Cloud Infrastructure Software

Frontdoor Defense • San Carlos (CA)

Hybrid
USD 224,000 - 260,000
Healthcare
3 weeks PTO + 13+ holidays
401(k)
Lead Backend Software Engineer
Lead Backend Software Engineer

Beacon AI • San Carlos (CA)

Hybrid
USD 120,000 - 160,000
Comprehensive Healthcare Coverage
Paid Time Off
Connectivity Stipend
+2
Senior Data Engineer
Senior Data Engineer

Beaconcure Inc. • Needham (MA)

On-site
USD 100,000 - 130,000
Growth mindset culture
Recognition for impactful work
Use of modern tools and technologies
Data & AI / LLM Engineer
Data & AI / LLM Engineer

SANS Consulting Services, Inc. • United States

On-site
USD 150,000 - 230,000
Portfolio Software Engineer
Portfolio Software Engineer

Beacon Software • New York (NY)

On-site
USD 140,000 - 200,000
Lead Software Engineer
Lead Software Engineer

Beacon Holdings • United States

On-site
USD 180,000 - 240,000
Lead Software Engineer - US (Remote)
Lead Software Engineer - US (Remote)

Beacon Holdings • United States

On-site
USD 140,000 - 190,000
AI Strategist
AI Strategist

Beacon Holdings • San Francisco (CA)

On-site
USD 180,000 - 260,000