AI-Ready Lakehouse Architect for Scalable Data Platforms

Tata Consultancy Services; Marlborough (MA); On-site; USD 150,000 - 160,000; Full time

Benefits offered by this job

Discretionary annual incentive

Comprehensive medical coverage

Parental leave

Insurance options

Training reimbursement

Vacation and holidays

401K plan

Job summary

Tata Consultancy Services in Marlborough, MA is hiring a data platform engineer to define enterprise lakehouse standards, governance, and an implementation roadmap.

You will design and build Iceberg-based Bronze, Silver, and Gold layers, develop ingestion frameworks, and create scalable pipelines with Spark, Flink, Kafka, dbt, SQL, and Python while enabling AI-ready semantic layers and robust metadata management.

Qualifications

• Experience with Iceberg-based architectures and data lakehouse governance.
• Proficiency with Spark, Flink, Kafka, and dbt tooling in cloud environments.
• Strong programming skills in Python/Scala/SQL/Java.
• Hands-on with orchestration tools (Airflow, Dagster, Prefect).
• Familiarity with data catalogs (Polaris, Unity Catalog, Glue Catalog, Nessie).
• Experience with cloud platforms (AWS/Azure/Google Cloud).

Responsibilities

• Define enterprise lakehouse architecture, standards, governance, and implementation roadmap.
• Design and build Iceberg-based Bronze, Silver, and Gold data layers.
• Develop ingestion frameworks for batch, streaming, CDC, APIs, and event-driven integrations.
• Build scalable Spark, Flink, Kafka, dbt, SQL, and Python pipelines.
• Implement metadata management, cataloging, lineage, RBAC, encryption, and data quality.
• Optimize partitioning, compaction, snapshots, metadata, and query performance.
• Enable AI-ready semantic layers, feature engineering, RAG, and GenAI workloads.
• Implement Infrastructure as Code, CI/CD, monitoring, and observability.

Skills

Apache Iceberg

Spark

Flink

Kafka

dbt

Trino

Snowflake

Databricks

AWS/Azure/Google Cloud

Python/Scala/SQL/Java

Kubernetes

Docker

Terraform

Airflow/Dagster/Prefect

Polaris/Unity Catalog/Glue Catalog/Ap.

Tools

Kubernetes

Docker

Terraform

Job description

Tata Consultancy Services in Marlborough, MA is hiring a data platform engineer to define enterprise lakehouse standards, governance, and an implementation roadmap.

You will design and build Iceberg-based Bronze, Silver, and Gold layers, develop ingestion frameworks, and create scalable pipelines with Spark, Flink, Kafka, dbt, SQL, and Python while enabling AI-ready semantic layers and robust metadata management.