Senior Data Engineer

Tredence Inc.; San Jose (CA); On-site; USD 150,000 - 190,000; Full time

Job summary

Tredence Inc. in San Jose seeks a Data Engineering Architect to lead the design of a high-scale data platform handling retail transactions, inventory, and customer interactions.

You will bridge heavy data engineering with BI delivery through Looker LookML and Power BI, ensuring fast, reliable reporting for executives and analysts.

The role requires hands-on development with Spark, Scala, and GCP Dataproc, plus governance and mentoring of data engineers.

Qualifications

• Experience architecting large-scale data platforms.
• Proven track record in BI and analytics delivery.
• Experience with GCP stack and Spark pipelines.
• Strong collaboration with business stakeholders.

Responsibilities

• Lead end-to-end design of the enterprise data platform.
• Bridge data engineering with BI delivery via Looker/Power BI.
• Optimize batch and real-time pipelines on GCP.
• Provide technical leadership and governance.

Skills

Analytical thinking

Data modeling

Leadership

Mentorship

Tools

Scala

Apache Spark

GCP Dataproc

BigQuery

Looker LookML

Power BI

DAX

CI/CD

Job description

Role Description

Data Engineering Architect with extensive expertise in large-scale data processing, enterprise pipeline design, and business intelligence architecture within the retail domain.

Role Description

Data Engineering Architect with extensive expertise in large-scale data processing, enterprise pipeline design, and business intelligence architecture within the retail domain.

In this role, you will lead the end-to-end design and execution of our enterprise data platform, processing high-volume retail transactions, customer interactions, and inventory feeds. You will bridge the gap between heavy-duty Data Engineering (using Scala, Apache Spark, and GCP Dataproc) and Business Intelligence delivery (via Power BI, Looker semantic models, and executive dashboards). The ideal candidate brings a proven track record of architecting performant pipelines for massive datasets while ensuring seamless, high-speed reporting for analytical end-users.

Key Responsibilities

Data Engineering & Pipeline Architecture

• System Design: Architect, scale, and maintain robust, fault-tolerant batch and streaming data pipelines on Google Cloud Platform (GCP) using GCP Dataproc, GCS, and BigQuery.
• Code Execution & Development: Core pipeline development using Scala and Apache Spark, enforcing high coding standards, modular design patterns, and automated testing strategies.
• Pipeline Optimization: Conduct deep performance tuning on Spark jobs, memory management, shuffle partitioning, and execution plans to ensure ultra-low execution latency and cost-effective cloud resource usage.

Semantic Modeling & BI Layer Architecture

• Semantic Layer Design: Build, standardize, and govern unified semantic models in Looker (LookML) and Power BI (Tabular Models / DAX) for enterprise-wide analytics.
• Dashboard & Report Engineering: Partner with analytical teams to translate retail business requirements into performant, intuitive reporting interfaces and dashboards.
• Query Performance Tuning: Optimize BI connection modes (DirectQuery, Import, Dual Mode, Aggregation Tables) and underlying data store schemas to support sub-second dashboard load times across massive datasets.

Retail Analytics & Business Enablement

• Domain Alignment: Leverage deep retail domain expertise (e.g., POS data, inventory management, customer 360, supply chain, store operations, and promotional effectiveness) to design logical data models that reflect business realities.
• Insight Delivery: Collaborate with business analysts and decision-makers to transform complex datasets into actionable operational and executive insights.

Technical Leadership & Governance

• Standards & Best Practices: Establish technical standards for data modeling, ETL/ELT development, code reviews, CI/CD deployment, and monitoring.
• Data Quality & Governance: Implement data lineage, metadata management, and automated data validation rules to guarantee data integrity across downstream reports.
• Mentorship: Provide architectural guidance and technical mentorship to senior and staff data engineers.