Senior Data Engineer

Inherent Technologies; San Jose (CA); On-site; USD 150,000 - 190,000; Full time

Job summary

Inherent Technologies in San Jose, CA is seeking a Senior Data Engineer to lead a critical migration to a PostgreSQL-based data platform. You will own end-to-end ETL/ELT pipelines, validate data quality, and align migration scope and timelines with business stakeholders.

The role emphasizes hands-on design, performance tuning, and mentoring junior engineers. You will optimize loads for millions of records, implement robust monitoring, and document mappings and runbooks to ensure a reliable,

Qualifications

• 5+ years of data engineering with large-scale migrations.
• Proficient in Python for building data pipelines.
• Expertise in PostgreSQL and Oracle performance tuning.
• Experience designing ETL/ELT at scale with millions of records.
• Knowledge of data validation and reconciliation techniques.

Responsibilities

• Lead migration of high-volume data into PostgreSQL infrastructure.
• Build and maintain robust ETL/ELT pipelines.
• Map legacy data structures to a unified schema.
• Define data validation frameworks for integrity and completeness.
• Optimize pipelines with tuning and incremental loads.
• Troubleshoot data quality issues and migrations failures.
• Document mappings, logic, and runbooks for stakeholders.
• Collaborate with stakeholders to align scope and timelines.
• Establish monitoring and alerting for post-migration quality.
• Mentor junior engineers and share best practices.

Skills

Python

PostgreSQL

Oracle

ETL/ELT pipelines

Data migration

Data modeling

Data validation

CI/CD for data pipelines

Tools

Airflow

Dagster

Prefect

Git

Docker

Terraform

dbt tests

Great Expectations

Job description

Position: Senior Data Engineer

Location: San Jose, CA *Onsite *

Duration: 1 Years

JD

Responsibilities

• Lead the end-to-end migration (transformation and load) of high-volume, high-complexity data into a new PostgreSQL-based infrastructure
• Design, build, and maintain robust ETL/ELT pipelines capable of processing millions of records reliably and efficiently
• Map and reconcile complex legacy data structures across multiple business entities into a unified target schema
• Define and implement data validation frameworks to ensure integrity, completeness, and accuracy throughout the migration
• Optimize pipeline performance, including query tuning, indexing strategy, and batch/incremental load design in PostgreSQL
• Identify, troubleshoot, and resolve data quality issues, schema mismatches, and pipeline failures
• Document data mappings, transformation logic, and migration runbooks for engineering and business stakeholders
• Partner with business and technical stakeholders to align migration scope, timelines, and data requirements
• Establish monitoring, logging, and alerting to track pipeline health and data quality post-migration
• Mentor junior data engineers and contribute to engineering best practices and standards

Required Qualifications

• 5+ years of experience in data engineering, with demonstrated ownership of large-scale data migration projects
• Strong to expert-level proficiency in Python for building and automating data pipelines
• Deep hands-on experience with PostgreSQL/Oracle , including schema design, query optimization, and performance tuning
• Proven experience designing and managing ETL/ELT pipelines at scale (millions of records)
• Experience mapping and transforming complex, legacy data structures across disparate systems or business entities
• Strong understanding of data validation, reconciliation, and quality assurance techniques
• Solid grasp of data modeling principles (normalization, indexing, partitioning)
• Experience with version control (Git) and CI/CD practices for data pipelines

Preferred Qualifications

• Experience with orchestration tools (e.g., Airflow, Dagster, Prefect)
• Familiarity with cloud data platforms (AWS, GCP, or Azure)
• Experience with other relational or NoSQL databases and cross-database migrations
• Background working in regulated or high-stakes data environments (finance, healthcare, etc.)
• Experience with containerization (Docker) and infrastructure-as-code
• Exposure to data quality/testing frameworks (e.g., Great Expectations, dbt tests)

What Success Looks Like

• A fully migrated, validated dataset in PostgreSQL with zero critical data loss or corruption
• ETL/ELT pipelines that are documented, repeatable, and optimized for ongoing operation
• A clear audit trail of data lineage and validation results across all business entities involved