Senior Data Engineer

Brooksource; United States; On-site; USD 140,000 - 180,000; Full time

Benefits offered by this job

401(k) Matching Plan

6 Paid Holidays

Regular Meetings

Job summary

Brooksource is seeking a Senior Data Engineer to join our AI Data Products Scrum Team for a Fortune 50 healthcare insurance organization. This fully remote role focuses on building scalable data pipelines and reusable data assets powering analytics, ML, and AI initiatives.

You will design, implement, and operate production data pipelines in a modern distributed environment, collaborating with cross‑functional teams to deliver reliable healthcare data solutions and maintain data governance and

Qualifications

• Senior-level production data pipeline experience in a modern distributed environment.
• Proficiency with Databricks, Spark, or Hadoop stack for scalable data work.
• Strong SQL and Python/PySpark for production data engineering.
• Hands-on DevOps experience with CI/CD and Git-based workflows.
• Experience with ETL/ELT, batch/streaming processing, data quality validation.
• Familiarity with Parquet/Delta formats and data warehousing concepts.
• Exposure to data governance, lineage, and discoverability practices (preferred).
• Experience with AI-enabled data assets (preferred).
• Experience in regulated enterprises (healthcare/financial services) (preferred).

Responsibilities

• Design, build, and maintain scalable batch and streaming data pipelines.
• Develop production data solutions using SQL, Python, and PySpark.
• Troubleshoot, optimize, and enhance pipelines to meet data needs.
• Create reusable ingestion, transformation, and data-product patterns.
• Implement data-quality checks, monitoring, and lineage for production pipelines.
• Use Git and CI/CD to test, review, and deploy assets across environments.
• Collaborate with engineers, data scientists, and platform teams on healthcare solutions.
• Support AI-enabling data assets like feature tables and embeddings.
• Ensure compliance with data governance and discovery practices.

Skills

Databricks Data Engineering

Spark

SQL

Python

PySpark

DevOps CI/CD

Git workflows

Data governance

Regulated environment experience

Tools

Databricks

Spark

Hadoop

Parquet/Delta

Git

CI/CD tooling

Job description

Join our AI Data Products Scrum Team as a Senior Data Engineer, supporting a Fortune 50 healthcare insurance organization in a fully remote capacity. In this hands‑on role, you will be instrumental in building, optimizing, and operationalizing scalable data pipelines and reusable data assets that power advanced analytics, machine learning, and generative AI initiatives. You will work within a modern distributed data environment, collaborating with cross‑functional teams to deliver reliable, production‑ready healthcare data solutions. This is an opportunity to make a significant impact in a dynamic, innovative environment while enjoying comprehensive benefits and competitive compensation.

WHAT YOU'LL DO

• Design, build, and maintain scalable batch and streaming data pipelines using modern distributed data technologies
• Develop production data solutions using SQL, Python, and PySpark, selecting the appropriate tool for each workload
• Troubleshoot, optimize, and enhance existing data pipelines to meet new data requirements and improve performance
• Create reusable ingestion, transformation, reconciliation, and data-product patterns for analytics and AI use cases
• Implement data-quality checks, monitoring, lineage, documentation, and provide reliable operational support for production pipelines
• Utilize Git and CI/CD practices to test, review, version, deploy, and promote data-engineering assets across environments
• Collaborate with lead engineers, application teams, data scientists, and platform teams on production healthcare solutions, market expansion, and AI-assisted workflows
• Support AI-enabling data assets such as feature tables, vector indices, embeddings, or semantic context
• Ensure compliance with data governance, cataloging, lineage, and discoverability practices, especially in regulated environments

WHAT YOU BRING

• Senior-level, hands‑on experience designing, building, and supporting production data pipelines in a modern distributed data environment
• Strong production experience with Databricks Data Engineering, Spark distributed computing, or the Hadoop framework
• Strong SQL skills and practical experience using Python and/or PySpark for production data‑engineering work
• Hands‑on experience with DevOps, CI/CD pipelines, and Git-based source‑control workflows
• Experience with ETL/ELT, batch and/or streaming processing, production debugging, performance tuning, and data‑quality validation
• Familiarity with columnar formats such as Parquet and Delta, plus data warehousing and data‑analysis concepts
• Exposure to data governance, cataloging, lineage, and discoverability practices (preferred)
• Experience supporting AI‑enabling data assets such as feature tables, vector indices, embeddings, or semantic context (preferred)
• Experience working in a regulated enterprise environment, preferably healthcare or financial services (preferred)

Why Join Us?

• 401(k) Matching Plan
• Relationship Driven Process to Find Your Best Fit
• 6 Paid Holidays
• Regular Meetings to Ensure Quality in Your Engagement

About Us:

At Brooksource, relationships are the foundation of everything we do. Since 2000, we’ve built lasting partnerships with clients, consultants, and internal teams to deliver an exceptional experience across every engagement. As a trusted IT and Engineering services provider, Brooksource supports Fortune 500 organizations through Experience-Driven Staffing, Professional Services, and Elevate, our proprietary Workforce Transformation program. Whether you're hiring for software development, cloud computing, cybersecurity, data analytics, or enterprise IT, our customized staffing solutions are designed to align with your company’s unique goals, culture, and technology stack. We offer flexible hiring models, including contract, contract-to-hire, and direct placement to meet your evolving business needs.

We are a certified partner of leading platforms, including Salesforce, AWS, Microsoft, and Google Cloud, enabling us to deliver scalable, end-to-end technology solutions. With a growing national footprint, Brooksource is redefining expectations in IT consulting, engineering services, and technology workforce solutions.

Brooksource is a top consulting and professional services provider specializing in IT and Engineering solutions for Fortune 500 companies and the public sector. It utilizes a national network of over 25,000 consultants, local and national delivery engines, as well as strategic partnerships with industry-leading technology providers to support its customers’ high‑impact initiatives.

EEO Statement:

Brooksource is an equal opportunity employer that does not discriminate on the basis of actual or perceived race, color, creed, religion, national origin, ancestry, citizenship status, age, sex or gender (including pregnancy, childbirth, lactation and related medical conditions), gender identity or gender expression, sexual orientation, marital status, military service and veteran status, physical or mental disability, protected medical condition as defined by applicable state or local law, genetic information, or any other characteristic protected by applicable federal, state, or local laws and ordinances.

Benefits & Perks: Brooksource offers competitive medical, dental, vision, Health Savings Account, Dependent Care FSA, and supplemental coverage with plans that can fit each employee’s needs. We offer a 401k plan that includes a company match and is fully vested after you become eligible, paid time off, sick time, and paid company holidays. We also offer an Employee Assistance Program (EAP) that provides services like virtual counseling, financial services, legal services, life coaching, etc.

Pay Disclaimer:

The pay range for this job level is a general guideline only and not a guarantee of compensation or salary. Additional factors considered in extending an offer include (but are not limited to) responsibilities of the job, education, experience, knowledge, skills, and abilities, as well as internal equity, alignment with market data, applicable bargaining agreement (if any), or other law.