Senior Managing Consultant Data Engineer

Berkeley Research Group, LLC; New York (NY); On-site; USD 158,000 - 210,000; Full time

Job summary

Berkeley Research Group, LLC in New York, NY is seeking a Senior Managing Consultant Data Engineer to lead hands-on data engineering workstreams supporting AI-enabled financial services solutions. The role focuses on building reliable data products and scalable pipelines for ingestion, transformation, and retrieval.

You will design warehouse and lakehouse models, implement data quality controls, and collaborate with AI engineers and client teams to deliver trusted data assets and actionable

Qualifications

• Minimum 5 years for Managing Consultant; 7 years for Sr. Managing Consultant.
• Hands-on delivery of production data pipelines and data products.
• Strong Python and SQL skills across ingestion and transformation.
• Experience with warehouse or lakehouse models and schema design.
• Experience with data quality, lineage, access controls, and PII handling.

Responsibilities

• Lead data engineering workstreams from discovery through production handover.
• Design and build batch and streaming pipelines in Python and SQL.
• Develop ingestion for structured and unstructured data, including metadata capture and indexing.
• Design warehouse and lakehouse models for reporting, analytics, and AI use cases.
• Implement data quality checks, lineage, access controls, and PII handling.
• Prepare retrieval-ready data with embedding pipelines and metadata enrichment.
• Collaborate with AI engineers and client teams to validate approaches.
• Build automated tests, deployment pipelines, monitoring, and recovery procedures.
• Mentor engineers and review code and designs; coordinate with stakeholders.
• Contribute technical estimates, assumptions, and data dependencies to proposals.

Skills

Python

SQL

Data pipelines

Mentoring

Leadership

Tools

Spark

dbt

CI/CD

Snowflake

Job description

Berkeley Research Group is hiring a Senior Managing Consultant Data Engineer (onsite, New York, NY) to lead hands‑on data engineering workstreams supporting AI‑enabled financial services solutions.

Responsibilities

• Lead data engineering workstreams from discovery through production handover, coordinating technical dependencies, delivery risks, and implementation decisions with client business and technology stakeholders.
• Design and build batch and streaming pipelines in Python and SQL to ingest, transform, and reconcile data from operational systems, APIs, files, and third‑party sources.
• Develop ingestion processes for both structured records and unstructured documents, including extraction, metadata capture, exception handling, and traceability back to source material.
• Design warehouse and lakehouse data models for operational reporting, analytics, and AI use cases while preserving business definitions and reconciliation controls.
• Implement data quality checks, lineage, access controls, and PII handling with client security, privacy, and risk teams to support appropriate use of sensitive information.
• Prepare retrieval‑ready data by supporting document parsing, chunking, metadata enrichment, embedding pipelines, and access‑aware indexing in collaboration with AI engineers.
• Work in the AI Co‑lab with domain experts and vendor partners to frame data requirements, compare implementation options, and validate reusable approaches against realistic financial services scenarios.
• Build automated tests, deployment pipelines, monitoring, and recovery procedures to make data products dependable and provide client teams operating guidance.
• Mentor engineers and review code and designs; coordinate with AI, application, change, and risk colleagues to deliver integrated solutions and measure operational impact.
• Capture domain knowledge in reusable schemas, pipeline components, and implementation guides; contribute technical estimates, assumptions, and data dependencies to proposals and next client pursuits.

Requirements

• Minimum experience: 5 years for Managing Consultant level; 7 years for Sr. Managing Consultant level.
• Demonstrated senior, hands‑on delivery of production data pipelines and data products; equivalent practical experience may substitute for formal education or certification.
• Strong Python and SQL skills, including ability to design, debug, optimize, and test ingestion and transformation across APIs, databases, files, and documents.
• Experience implementing warehouse or lakehouse models; ability to explain choices around schema design, storage, orchestration, incremental processing, and reconciliation using tools such as Spark and dbt or comparable options.
• Applied data quality, lineage, least‑privilege access, and sensitive‑data controls in production; ability to translate business and risk requirements into testable engineering controls.
• Experience preparing structured and unstructured data for AI retrieval, including document parsing, chunking, metadata design, indexing and preservation of source provenance and access permissions.
• Delivered maintainable data services using version control, automated testing, CI/CD, monitoring, and cloud infrastructure; able to diagnose reliability, performance, and cost issues.
• Ability to lead complex technical workstreams, mentor engineers, collaborate with client stakeholders, clarify requirements, communicate trade‑offs, and provide credible scope and effort estimates.
• Ability to travel to client sites as needed.

Technologies

• Python
• SQL
• Spark
• dbt
• CI/CD
• Snowflake

Preferred

• Industry experience building operational applications at a leading bank, lender, or financial services institution.
• Financial services experience in consumer lending, commercial lending, or payments (mortgage, credit card, personal loan, or auto lending, including servicing, collections, and default; adjacent banking or wealth and asset management is a plus).
• Experience building retrieval data foundations for production AI applications, including vector or hybrid search, document versioning, and retrieval quality evaluation.
• Experience with change data capture, event‑driven processing, or distributed data platforms across cloud and hybrid environments.
• Experience evaluating vendor data capabilities or converting domain expertise into reusable data assets, accelerators, and client implementation patterns.
• Experience implementing the Snowflake platform.

Required (Legal / Work Authorization)

• Candidates must be able to verify their legal right to work in the United States without employer sponsorship now or in the future.

Location and Compensation

• Location: New York, NY (onsite)
• Salary: USD 158,000 - 210,000 per year