Sr Data Engineer

Baker Tilly Advisory Group, LP

Santa Clara (CA)

On-site

USD 185,000 - 230,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Baker Tilly Advisory Group, LP in Santa Clara, CA, seeks a Sr Data Engineer to join the AI Product Group and build scalable, secure data infrastructure for AI applications and agents. You will own production pipelines, lakehouse architecture, governance, and retrieval systems, focusing on data quality and reliability while collaborating with global teams across platforms and domains.

This role emphasizes automated testing, versioning, monitoring, and practical delivery to enable enterprise AI

Qualifications

  • Bachelor’s degree in Computer Science, Engineering, or a related field.
  • 3 years of hands-on data engineering or software engineering experience.
  • Experience with modern lakehouse platforms and production pipelines.
  • Strong Python and SQL; production-grade data engineering.
  • Experience with data governance, lineage, and automated testing.
  • Familiarity with data modeling principles and scalable structures.
  • Experience with cloud-based data platforms, preferably Azure.
  • Knowledge graphs, vector/search retrieval, and entity resolution.

Responsibilities

  • Design, develop, implement, and maintain scalable data pipelines in a lakehouse environment.
  • Build lakehouse architecture with medallion data layers and data modeling.
  • Implement validation, lineage tracking, metadata management, and governance.
  • Develop data models, contracts, and entity resolution across sources.
  • Collaborate with AI/ Platform teams and external partners; ensure reliability.
  • Maintain data freshness, monitoring, and production-readiness of pipelines.

Tools

Databricks
Apache Spark
Delta Lake
Medallion architecture
Microsoft Azure
Unity Catalog
Knowledge graphs
Graph databases
Vector search
Hybrid search
Retrieval-based applications
Document extraction
Entity resolution
Master data management
Change data capture

Job description

The Sr Data Engineer will join the founding team of Baker Tilly’s AI Product Group (APG) to build scalable, secure, and reliable data infrastructure that supports AI applications and intelligent agents.

Role Overview

This position centers on governed data foundations and production-grade data pipelines designed for AI use cases. You will work across lakehouse architecture, knowledge graph and retrieval capabilities, and evaluation-driven data engineering, with an emphasis on data quality, lineage, and operational reliability.

Responsibilities
  • Design, develop, implement, and maintain scalable batch and incremental data pipelines that transform structured and unstructured information into clean, validated, business-ready datasets within a modern lakehouse environment.
  • Build and optimize lakehouse architecture, including data processing and transformation layers that apply modern principles such as medallion architecture, data modeling, and integration of enterprise and legacy data sources.
  • Implement automated validation, reconciliation, lineage tracking, and metadata management to support data accuracy, consistency, traceability, and reliability across the data lifecycle.
  • Develop data models, data contracts, and entity-resolution capabilities to establish consistent client, engagement, and document identities across enterprise data sources.
  • Support data governance and access controls through data cataloging, classification, permissions, and access management for sensitive information used by authorized users and AI agents.
  • Build and maintain knowledge graph structures, data relationships, search indexes, and retrieval pipelines that enable AI applications to access relevant, accurate, and properly governed enterprise information.
  • Create pipelines that capture AI agent activity, practitioner feedback, reviews, and corrections to support ground-truth datasets, AI evaluation, and continuous improvements in data and retrieval quality.
  • Establish evaluation-based data engineering practices, including quality standards, testing frameworks, benchmarks, and validation processes for extraction, retrieval effectiveness, and grounding accuracy.
  • Maintain data freshness and reliability using incremental processing, change data capture, versioning, and monitoring so data products, knowledge graphs, and search indexes remain current as sources evolve.
  • Deploy, monitor, troubleshoot, and maintain production data pipelines using engineering best practices such as automated testing, code reviews, and operational documentation.
  • Collaborate with external engineering partners to transition data solutions into internally owned environments, including reusable code, tests, and runbooks.
  • Partner with AI Engineering, Platform & Production Engineering, enterprise data teams, subject matter experts, and engineering teams in India to align solutions with enterprise standards and business objectives.
  • Participate in special projects and other assigned duties to support the firm’s AI and data platform capabilities.
Requirements
  • Bachelor’s degree in Computer Science, Engineering, or a related field; advanced degree preferred.
  • 3 years of hands-on data engineering or software engineering experience, including demonstrated success developing, deploying, and maintaining production data pipelines.
  • Experience independently building and operating production pipelines on a modern lakehouse platform, with ownership of data processing, transformation, and operational reliability.
  • Experience implementing data quality, lineage, or data contracts in production pipelines, including automated testing, reconciliation, validation, or traceability practices.
  • Strong proficiency in Python and SQL, including the ability to develop, optimize, test, and maintain production-quality data engineering solutions.
  • Strong understanding of data modeling principles and the ability to design scalable, maintainable data structures for evolving requirements and source systems.
  • Experience designing and developing batch and incremental data pipelines for structured and unstructured enterprise data.
  • Solid software engineering fundamentals, including version control, automated testing, code reviews, debugging, and reliable, repeatable data processing workflows.
  • Ability to validate technical solutions with measurable outcomes such as automated tests, benchmarks, experiments, performance metrics, or comparative evaluations.
  • Strong preference for experience with modern lakehouse technologies such as Databricks, Apache Spark, Delta Lake, or medallion architecture.
  • Preference for experience with cloud-based data platforms, preferably Microsoft Azure.
  • Preference for familiarity with enterprise data governance capabilities including data catalogs, classification, access controls, lineage, and technologies such as Unity Catalog or comparable solutions.
  • Preference for knowledge graphs and related tooling such as graph databases, graph data modeling, vector search, hybrid search, or retrieval-based applications.
  • Preference for familiarity with unstructured data processing, including document extraction, metadata management, content chunking, and search indexing.
  • Preference for experience with entity resolution, master data management, change data capture, or incremental data processing.
  • Understanding of data provenance, traceability, and governance requirements in complex enterprise or regulated environments.
  • Excellent written and verbal communication skills, including documentation and collaboration with technical and nontechnical stakeholders.
  • Demonstrated ability to work with business subject matter experts and globally distributed stakeholders to translate requirements into reliable data solutions.
  • Ability to manage competing priorities and adapt quickly in a fast-paced, collaborative environment.
  • Experience using AI-assisted and agentic development tools to improve engineering productivity, or willingness to incorporate them into daily development practices.
  • No professional certifications or licenses required; prior experience with AI, machine learning, LLM systems, or knowledge graphs is not required.
Preferred Skills & Tools
  • Python, SQL
  • Databricks, Apache Spark, Delta Lake
  • Medallion architecture
  • Microsoft Azure
  • Unity Catalog
  • Knowledge graphs, graph databases, graph data modeling
  • Vector search, hybrid search, retrieval-based applications
  • Document extraction, content chunking, search indexing
  • Entity resolution, master data management, change data capture, incremental data processing
  • LLM systems
Location

Santa Clara, CA (onsite)

Compensation

USD 185,000 - 230,000 per year

National pay rate range: $185,000 – $230,000. In California: $185,000 – $230,000.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Sr Data Engineer
Sr Data Engineer

Baker Tilly Advisory Group, LP • San Jose (CA)

On-site
USD 185,000 - 230,000
Sr Data Engineer
Sr Data Engineer

Bakertilly • United States

On-site
USD 185,000 - 230,000
Senior Data Engineer
Senior Data Engineer

Baker Tilly (US) • Santa Monica (CA)

On-site
USD 185,000 - 230,000
Senior Data Engineer
Senior Data Engineer

Baker Tilly (US) • Pasadena (CA)

On-site
USD 185,000 - 230,000
Senior Data Engineer
Senior Data Engineer

Baker Tilly (US) • Santa Clara (CA)

On-site
USD 185,000 - 230,000
Senior Data Engineer
Senior Data Engineer

Baker Tilly (US) • Portland (OR)

On-site
USD 185,000 - 230,000
Senior Data Engineer
Senior Data Engineer

Baker Tilly (US) • Irvine (CA)

On-site
USD 185,000 - 230,000
Senior Data Engineer
Senior Data Engineer

Baker Tilly (US) • California

On-site
USD 185,000 - 230,000
Senior Data Engineer
Senior Data Engineer

Baker Tilly (US) • San Jose (CA)

On-site
USD 185,000 - 230,000
Senior Data Engineer
Senior Data Engineer

Baker Tilly International • California

On-site
USD 185,000 - 230,000