Data Engineer: Spark, Airflow & Anaplan Integrations

Tata Consultancy Services; Irvine (CA); On-site; USD 120,000 - 160,000; Full time

Job summary

Tata Consultancy Services in Irvine, CA is seeking a Data Engineer to design and implement scalable data pipelines for Starship Integrations.

You will work with Python, Spark, Airflow, and SQL to ingest data from AWS S3, model schemas, and publish results downstream.

The role emphasizes production support, performance tuning, and collaboration with teams to ensure reliable data flows and lineage documentation.

Qualifications

• Python production code, modular, config-driven, unit tested.
• Spark/PySpark for distributed transformations and performance tuning.
• Airflow DAG authoring, scheduling, and debugging failed runs.
• SQL and relational modeling, including migrations and reconciliation.
• Experience with AWS S3.
• Production support for data pipelines and root cause analysis.

Responsibilities

• Integration applications- Containerized Python/PySpark apps reading from S3 and publishing results downstream.
• Anaplan integration- Anaplan authentication and bulk import/export APIs and data reconciliation.
• Airflow orchestration- Author and maintain DAGs, operators, and deployment configurations.
• Data stores- AWS S3, Delta/Parquet datasets, and PostgreSQL RDS with schema migrations.
• CI/CD & runtime- Docker image builds, Kubernetes workloads, and multi-environment promotion.
• Production support- Respond to pipeline failures, memory issues, schema drift, and reconciliation mismatches.
• Documentation- Runbooks and lineage documentation.
• Salary Range- $120,000-$160,000 a year

Skills

Python

Spark/PySpark

Airflow

SQL

AWS S3

Production support

Tools

Anaplan integration

PostgreSQL / RDS

Alembic migrations

Docker

Kubernetes

CI/CD pipelines

Delta Lake / Databricks

Job description

Tata Consultancy Services in Irvine, CA is seeking a Data Engineer to design and implement scalable data pipelines for Starship Integrations.

You will work with Python, Spark, Airflow, and SQL to ingest data from AWS S3, model schemas, and publish results downstream.

The role emphasizes production support, performance tuning, and collaboration with teams to ensure reliable data flows and lineage documentation.