Senior Data Engineer

LatentView Analytics; San Jose (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

LatentView Analytics seeks a senior Data Engineer in the San Jose area to design and maintain scalable data pipelines on Databricks and AWS.

You will build Airflow workflows, optimize Spark jobs, and collaborate with data scientists and product teams to deliver reliable data assets across Bronze/Silver/Gold layers.

Qualifications

• 7+ years of experience in Data Engineering roles.
• Extensive hands-on experience with Databricks, Delta Lake, Unity Catalog, cluster management.
• Strong experience with AWS cloud services (S3, IAM, Glue, EMR, Lambda, Redshift, CloudWatch).
• Proven expertise in Apache Airflow for workflow orchestration and scheduling.
• Strong programming skills in Python and SQL.
• Solid understanding of Apache Spark (PySpark, Spark SQL, performance tuning).
• Experience with data modeling, warehousing concepts, and Lakehouse architecture.
• Familiarity with version control (Git) and CI/CD practices.
• Strong understanding of data governance, quality, and lineage principles.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks (PySpark/Spark SQL).
• Build, orchestrate, and monitor workflows using Apache Airflow (DAG design, scheduling, dependency management).
• Architect and manage data infrastructure on AWS (S3, Glue, Lambda, EMR, Redshift, IAM, etc.).
• Optimize data pipelines for performance, reliability, and cost-efficiency.
• Implement data quality checks, testing frameworks, and observability/monitoring for pipelines.
• Collaborate with Data Scientists, Analysts, and Product teams to understand data requirements.
• Design and maintain data models (Lakehouse/Medallion architecture - Bronze/Silver/Gold layers).
• Ensure data security, access controls, and compliance across the data platform.
• Document architecture, pipelines, and processes for team knowledge sharing.
• Mentor junior data engineers and contribute to engineering best practices.

Skills

Problem-solving

Communication

Analytical mindset

Tools

Databricks

Airflow

Python

SQL

Spark

AWS

Git

CI/CD

Redshift

Glue

S3

EMR

IAM

Job description

LatentView Analytics is a leading global analytics and decision sciences provider, delivering solutions that help companies drive digital transformation and use data to gain a competitive advantage. With analytics solutions that provide a 360-degree view of the digital consumer, fuel machine learning capabilities and support artificial intelligence initiatives., LatentView Analytics enables leading global brands to predict new revenue streams, anticipate product trends and popularity, improve customer retention rates, optimize investment decisions, and turn unstructured data into valuable business assets

Required Qualifications:

• 7+ years of experience in Data Engineering roles
• Extensive hands-on experience with Databricks (Delta Lake, Unity Catalog, cluster management, notebooks, job orchestration)
• Strong experience with AWS cloud services (S3, IAM, Glue, EMR, Lambda, Redshift, CloudWatch)
• Proven expertise in Apache Airflow for workflow orchestration and scheduling
• Strong programming skills in Python and SQL
• Solid understanding of Apache Spark (PySpark, Spark SQL, performance tuning)
• Experience with data modeling, warehousing concepts, and Lakehouse architecture
• Familiarity with version control (Git) and CI/CD practices
• Strong understanding of data governance, quality, and lineage principles
• Excellent problem-solving and communication skills

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks (PySpark/Spark SQL)
• Build, orchestrate, and monitor workflows using Apache Airflow (DAG design, scheduling, dependency management)
• Architect and manage data infrastructure on AWS (S3, Glue, Lambda, EMR, Redshift, IAM, etc.)
• Optimize data pipelines for performance, reliability, and cost-efficiency
• Implement data quality checks, testing frameworks, and observability/monitoring for pipelines
• Collaborate with Data Scientists, Analysts, and Product teams to understand data requirements
• Design and maintain data models (Lakehouse/Medallion architecture - Bronze/Silver/Gold layers)
• Ensure data security, access controls, and compliance across the data platform
• Document architecture, pipelines, and processes for team knowledge sharing
• Mentor junior data engineers and contribute to engineering best practices

Required skills

• Databricks, AWS, Python, SQL, Pyspark, Genie, GenAI, Analytics, Cybersecurity Knowledge

At LatentView Analytics, we value a diverse, inclusive workforce and provide equal employment opportunities for all applicants and employees. All qualified applicants for employment will be considered without regard to an individual's race, color, sex, gender identity, gender expression, religion, age, national origin or ancestry, citizenship, physical or mental disability, medical condition, family care status, marital status, domestic partner status, sexual orientation, genetic information, military or veteran status, or any other basis protected by federal, state or local laws.