Senior Data Engineer | Remote AI Data Pipelines

Brooksource; United States; On-site; USD 140,000 - 180,000; Full time

Benefits offered by this job

401(k) Matching Plan

6 Paid Holidays

Regular Meetings

Job summary

Brooksource is seeking a Senior Data Engineer to join our AI Data Products Scrum Team for a Fortune 50 healthcare insurance organization. This fully remote role focuses on building scalable data pipelines and reusable data assets powering analytics, ML, and AI initiatives.

You will design, implement, and operate production data pipelines in a modern distributed environment, collaborating with cross‑functional teams to deliver reliable healthcare data solutions and maintain data governance and

Qualifications

• Senior-level production data pipeline experience in a modern distributed environment.
• Proficiency with Databricks, Spark, or Hadoop stack for scalable data work.
• Strong SQL and Python/PySpark for production data engineering.
• Hands-on DevOps experience with CI/CD and Git-based workflows.
• Experience with ETL/ELT, batch/streaming processing, data quality validation.
• Familiarity with Parquet/Delta formats and data warehousing concepts.
• Exposure to data governance, lineage, and discoverability practices (preferred).
• Experience with AI-enabled data assets (preferred).
• Experience in regulated enterprises (healthcare/financial services) (preferred).

Responsibilities

• Design, build, and maintain scalable batch and streaming data pipelines.
• Develop production data solutions using SQL, Python, and PySpark.
• Troubleshoot, optimize, and enhance pipelines to meet data needs.
• Create reusable ingestion, transformation, and data-product patterns.
• Implement data-quality checks, monitoring, and lineage for production pipelines.
• Use Git and CI/CD to test, review, and deploy assets across environments.
• Collaborate with engineers, data scientists, and platform teams on healthcare solutions.
• Support AI-enabling data assets like feature tables and embeddings.
• Ensure compliance with data governance and discovery practices.

Skills

Databricks Data Engineering

Spark

SQL

Python

PySpark

DevOps CI/CD

Git workflows

Data governance

Regulated environment experience

Tools

Databricks

Spark

Hadoop

Parquet/Delta

Git

CI/CD tooling

Job description

Brooksource is seeking a Senior Data Engineer to join our AI Data Products Scrum Team for a Fortune 50 healthcare insurance organization. This fully remote role focuses on building scalable data pipelines and reusable data assets powering analytics, ML, and AI initiatives.

You will design, implement, and operate production data pipelines in a modern distributed environment, collaborating with cross‑functional teams to deliver reliable healthcare data solutions and maintain data governance and