Data engineer-TACO

Tata Consultancy Services; Irvine (CA); On-site; USD 120,000 - 160,000; Full time

Job summary

Tata Consultancy Services in Irvine, CA is seeking a Data Engineer to design and implement scalable data pipelines for Starship Integrations.

You will work with Python, Spark, Airflow, and SQL to ingest data from AWS S3, model schemas, and publish results downstream.

The role emphasizes production support, performance tuning, and collaboration with teams to ensure reliable data flows and lineage documentation.

Qualifications

• Python production code, modular, config-driven, unit tested.
• Spark/PySpark for distributed transformations and performance tuning.
• Airflow DAG authoring, scheduling, and debugging failed runs.
• SQL and relational modeling, including migrations and reconciliation.
• Experience with AWS S3.
• Production support for data pipelines and root cause analysis.

Responsibilities

• Integration applications- Containerized Python/PySpark apps reading from S3 and publishing results downstream.
• Anaplan integration- Anaplan authentication and bulk import/export APIs and data reconciliation.
• Airflow orchestration- Author and maintain DAGs, operators, and deployment configurations.
• Data stores- AWS S3, Delta/Parquet datasets, and PostgreSQL RDS with schema migrations.
• CI/CD & runtime- Docker image builds, Kubernetes workloads, and multi-environment promotion.
• Production support- Respond to pipeline failures, memory issues, schema drift, and reconciliation mismatches.
• Documentation- Runbooks and lineage documentation.
• Salary Range- $120,000-$160,000 a year

Skills

Python

Spark/PySpark

Airflow

SQL

AWS S3

Production support

Tools

Anaplan integration

PostgreSQL / RDS

Alembic migrations

Docker

Kubernetes

CI/CD pipelines

Delta Lake / Databricks

Job description

Job Description

Data engineer - TACO | Starship Integrations

Required Skills

• Python — production application code (modular, config-driven, unit tested).
• Apache Spark / PySpark — building and tuning distributed transformations, including diagnosing memory and performance problems.
• Apache Airflow — DAG authoring, scheduling and dependency design, debugging failed runs.
• SQL and relational modeling — including schema migrations and data reconciliation.
• AWS S3.
• Production support — troubleshooting data pipeline failures and root-cause analysis.

Preferred Skills

• Anaplan integration — REST/bulk API, Anaplan Connect, or CloudWorks.
• PostgreSQL / RDS and Alembic migrations.
• Docker, Kubernetes, and CI/CD pipelines (Harness or similar).
• Delta Lake / Databricks.

Responsibilities

• Integration applications- Containerized Python/PySpark applications that read source data from S3, apply territory and effective-dating business logic, and publish results downstream.
• Anaplan integration- Anaplan authentication and bulk import/export APIs, file formatting and upload, and reconciliation of loaded data against source.
• Airflow orchestration- Author and maintain DAGs, custom operators, and environment-specific deployment configuration across environments.
• Data stores- AWS S3, Delta/Parquet datasets, and PostgreSQL RDS with schema migrations.
• CI/CD & runtime- Docker image builds, Kubernetes workloads, and multi-environment promotion pipelines.
• Production support- Respond to pipeline failures — Spark memory and performance issues, schema drift, and reconciliation mismatches.
• Documentation- Runbooks and lineage documentation.
• Salary Range- $120,000-$160,000 a year