Senior Data Engineer — Lakehouse & AI-Driven Pipelines

Veritas Search Group; Glendale (CA); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Veritas Search Group seeks a Sr. Data Engineer to design, build, and optimize scalable data pipelines within a modern enterprise data platform based in Glendale, CA.

This role emphasizes data quality, governance, and collaboration with internal and external engineering teams to drive lakehouse and AI-enabled data services. The ideal candidate has strong Python and SQL skills, cloud platform experience (AWS/Azure/GCP), and hands-on ETL/ELT expertise, with a track record of delivering

Qualifications

• Strong Python and SQL development experience in production data pipelines.
• Experience with cloud-based data platforms (lakehouse, data lake, data warehouse).
• Understanding of data modeling, data integration, and data governance principles.
• Ability to translate business needs into scalable technical solutions.

Responsibilities

• Design, build, test, deploy, and optimize ETL/ELT pipelines.
• Develop scalable data solutions in cloud lakehouse data environments.
• Create automated, reliable data workflows with monitoring and recovery processes.
• Implement data quality, validation, cleansing, and testing standards.
• Support data governance, security, access controls, masking, lineage, and compliance.
• Collaborate with data architects and stakeholders to define platform designs.
• Provide technical direction to offshore or managed-services data engineers.

Skills

Python

SQL

Data Modeling

ETL/ELT

Cloud Platforms

Data Governance

Independent/Autonomous Work

Communication

Education

Bachelor's degree in CS/IS/Engineering or related

Master's degree in Data Science/CS or related (preferred)

Tools

Databricks

Unity Catalog

Snowflake

AWS

Azure

GCP

Apache Spark

Apache Airflow

Docker

Kubernetes

Job description

This role requires candidates who are currently authorized to work in the U.S. without sponsorship, and C2C arrangements are not accepted. This role is hybrid (x4 days onsite) in the Glendale, CA area.

Position Overview

We are seeking a Sr. Data Engineer to support the continued development and expansion of a modern enterprise data platform. This individual will design, build, and optimize scalable data pipelines, strengthen data quality and governance, and help guide the work of internal and external engineering resources.

The ideal candidate has strong hands-on experience with Python, SQL, cloud-based data platforms, and ETL/ELT development. Experience with Databricks and Unity Catalog is strongly preferred, although candidates with strong backgrounds in Snowflake, AWS, Azure, GCP, or other modern data environments will also be considered.

This role requires someone who is technically strong, self-directed, collaborative, and comfortable balancing traditional data-engineering rigor with emerging artificial-intelligence technologies. The organization is expanding a recently established lakehouse platform and needs an engineer who can contribute directly while also providing technical direction to managed-services resources.

Key Responsibilities

• Design, build, test, deploy, and optimize ETL and ELT pipelines that ingest data from multiple internal and external sources.
• Develop scalable data solutions within a cloud-based lakehouse, data lake, or data warehouse environment.
• Create reliable, automated, and resilient data workflows with appropriate monitoring, alerting, and failure-recovery processes.
• Implement data-quality, validation, cleansing, reconciliation, and testing standards.
• Support data governance, security, access controls, data masking, lineage, and regulatory compliance.
• Partner with data architects to implement scalable and secure platform designs.
• Translate business requirements and ambiguous data requests into detailed technical specifications.
• Collaborate directly with business stakeholders, analytics teams, application teams, and other data consumers.
• Provide technical direction, standards, and oversight to offshore or managed-services data engineers.
• Review technical deliverables and help ensure consistent engineering quality across an extended development team.
• Support the integration of artificial intelligence, machine learning, large language models, and intelligent agents into the data platform.
• Help ensure that AI-enabled tools inherit appropriate data security and governance controls.
• Deploy scalable data services and machine-learning environments using containerization technologies.
• Evaluate emerging technologies and recommend improvements to the organization’s data ecosystem.
• Develop and maintain technical documentation, architecture standards, operational procedures, and support processes.

Required Qualifications

• Bachelor’s degree in Computer Science, Data Science, Information Systems, Engineering, or a related field.
• Strong professional experience in data engineering or a closely related discipline.
• Advanced hands-on experience with Python.
• Strong SQL development skills.
• Experience working with relational and non-relational databases.
• Demonstrated experience designing and building production ETL or ELT pipelines.
• Experience with data lakes, data warehouses, lakehouse architectures, or enterprise analytics platforms.
• Experience with at least one major cloud platform, such as AWS, Microsoft Azure, or Google Cloud Platform.
• Understanding of data modeling, data integration, data quality, and data-governance principles.
• Ability to translate business needs into scalable technical solutions.
• Experience working independently and managing priorities with limited supervision.
• Strong written, verbal, and interpersonal communication skills.
• Ability to collaborate effectively with technical teams, business stakeholders, and third-party service providers.
• Ability to provide technical guidance without requiring formal direct-report authority.
• Authorization to work in the United States without current or future visa sponsorship.

Preferred Qualifications

• Hands-on experience with Databricks and lakehouse architecture.
• Experience with Unity Catalog or a comparable data-governance platform.
• Experience with Apache Spark or PySpark.
• Experience with Apache Airflow or another workflow-orchestration platform.
• Experience with Docker and Kubernetes.
• Familiarity with Snowflake, Microsoft Fabric, or similar modern data platforms.
• Experience with Power BI, Tableau, or other business-intelligence tools.
• Knowledge of data modeling for analytics and reporting.
• Experience supporting machine-learning or artificial-intelligence environments.
• Familiarity with generative AI, large language models, AI agents, or AI-enabled engineering tools.
• Experience working with offshore or managed-services engineering teams.
• Experience in an additional technical discipline, such as C#/.NET development, software quality assurance, systems integration, or application architecture.
• Master’s degree in Data Science, Computer Science, or a related discipline.

Professional Attributes

The successful candidate will be:

• Technically curious and committed to continuous learning.
• Coachable and receptive to guidance from senior technical leaders.
• Confident enough to communicate and defend well-supported technical recommendations.
• Comfortable participating in constructive technical debate.
• Adaptable and open to new tools, platforms, and development approaches.
• Pragmatic about balancing data accuracy, governance, delivery speed, and business value.
• Capable of working with significant autonomy.
• Proactive in identifying risks, solving problems, and recommending improvements.
• Comfortable operating across multiple technical disciplines.
• Collaborative, professional, and focused on achieving the best outcome for the organization.