Consulting Partner - Data Engineering & Modern Data Platforms

Tata Consultancy Services; Irvine (CA); On-site; USD 190,000 - 240,000; Full time

Job summary

Tata Consultancy Services seeks a Consulting Partner with deep hands-on Data Engineering and Modern Data Platform expertise to lead enterprise data transformation programs in the US market.

The role combines executive-level advisory with architecture and engineering leadership, guiding large-scale data migrations, lakehouse implementations, and modernisation efforts across cloud platforms.

Qualifications

• 15+ years in data engineering, data platforms, analytics, or technology architecture.
• 10+ years hands-on in data engineering and data platform technologies.
• Experience architecting enterprise-scale data platforms.
• Strong hands-on Spark/PySpark and SQL.
• Experience with modern cloud data platforms such as Databricks or Snowflake.
• Experience with dbt.
• Experience with Airflow.
• Understanding of distributed computing, data processing performance, and scalability.
• Experience leading large-scale data modernization or migration programs.
• Ability to engage with engineers, architects, platform teams, and executives.

Responsibilities

• Lead complex enterprise data transformation programs with deep hands-on expertise.
• Move from executive discussions to deep technical reviews of Databricks, Spark/PySpark, dbt, SQL, and data pipelines.
• Oversee architecture reviews, governance, and performance optimization across large data ecosystems.
• Provide hands-on technical guidance for architecture and engineering challenges.
• Define migration patterns and automation strategies for scalable data platforms.

Skills

Spark/PySpark

SQL

Data architecture

Leadership

Tools

Databricks

Snowflake

dbt

Apache Airflow

Apache Spark

PySpark

Job description

Consulting Partner - Data Engineering & Modern Data Platforms

Required Experience

• 15+ years of experience in data engineering, data platforms, analytics, or technology architecture.
• 10+ years of deep hands‑on technical experience in data engineering and data platform technologies.
• Demonstrated experience architecting and delivering complex enterprise‑scale data platforms.
• Strong hands‑on experience with Spark/PySpark and SQL.
• Strong practical experience with at least one modern cloud data platform such as Databricks or Snowflake.
• Experience with modern data transformation frameworks such as dbt.
• Experience with workflow orchestration technologies such as Airflow.
• Strong understanding of distributed computing, data processing performance, and scalability.
• Experience leading large‑scale data modernization or migration programs.
• Ability to engage directly with engineers, architects, platform teams, and executive stakeholders.

Preferred Experience

• Cloud‑to‑cloud data platform migration.
• Migration of large PySpark estates.
• Legacy ETL modernization.
• Stored procedure modernization to dbt or modern transformation frameworks.
• Databricks platform engineering.
• Data platform governance and security.
• Large‑scale data migration involving hundreds of terabytes or petabyte‑scale environments.
• AI‑assisted engineering and migration accelerators.
• Building reusable engineering frameworks and accelerators.
• Experience with platform reliability, FinOps, and performance optimization.

Roles & Responsibilities

We are seeking a highly accomplished Consulting Partner with deep hands‑on Data Engineering and Modern Data Platform expertise to lead complex enterprise data transformation programs.

This is a technical leadership role, not a purely advisory or relationship‑management position. The successful candidate will combine executive consulting capabilities with strong architecture and engineering expertise and must have demonstrated hands‑on experience designing, building, modernising, and troubleshooting large‑scale data platforms and pipelines.

The Consulting Partner will work with client executives to shape transformation strategies while remaining sufficiently close to technology and engineering teams to influence architecture, solve complex technical problems, review implementations, and guide critical delivery decisions.

The role requires an individual who can move seamlessly from an executive discussion with a CIO or CDO to a deep technical discussion involving Databricks, Spark/PySpark, dbt, SQL, cloud architecture, data pipelines , orchestration, performance optimisation, and large‑scale data migration.

Key Responsibilities

• Technical Consulting & Architecture Leadership
• Serve as the senior technical advisor for complex data engineering and data platform transformation programmes.
• Define enterprise data platform, lakehouse, and data engineering architectures.
• Lead architecture assessments and develop modernisation strategies for legacy and cloud‑based data ecosystems.
• Design scalable architectures covering data ingestion, transformation, storage, orchestration, governance, security, and consumption.
• Review and challenge architecture decisions made by engineering and solution teams.
• Establish technology standards, engineering principles, and reference architectures.
• Identify technical risks and scalability challenges early in the programme lifecycle.
• Provide hands‑on technical guidance for complex architecture and engineering issues.
• Hands‑On Data Engineering Leadership

Responsibilities Include

• Reviewing and optimising PySpark and Spark applications.
• Analysing query and pipeline performance issues.
• Reviewing complex SQL transformations.
• Designing and reviewing dbt models, transformation frameworks, and testing strategies.
• Troubleshooting complex data processing and distributed computing issues.
• Reviewing data partitioning, clustering, file sizing, and storage optimisation strategies.
• Providing technical direction on Delta Lake, table design, and data lifecycle management.
• Reviewing Airflow DAGs and orchestration architecture.
• Helping teams resolve critical production issues and performance bottlenecks.
• Conducting architecture and code reviews for high‑risk or complex components.

Complex Program Technical Leadership

• Lead the technical direction of large‑scale data engineering and modernisation programmes.
• Oversee programmes involving hundreds or thousands of pipelines, jobs, tables, and datasets.
• Define technical migration strategies for:
• Legacy ETL to modern ELT
• Stored procedures to dbt
• Legacy Spark platforms to Databricks
• Cloud‑to‑cloud data platform migration
• Autosys and legacy scheduling to Airflow
• Data warehouse modernisation
• Data lake and lakehouse implementation
• Define migration patterns, factory approaches, automation strategies, and validation frameworks.
• Establish technical governance across multiple engineering teams and workstreams.
• Resolve cross‑platform and cross‑domain technical dependencies.
• Define technical KPIs for scalability, reliability, performance, data quality, and cost efficiency.

Modern Data Platform Expertise

The candidate should have strong practical experience across several of the following technologies:

• Data Engineering - Apache Spark, PySpark, SQL, Python, Distributed data processing, Data pipeline design and optimisation
• Modern Data Platforms - Databricks, Delta Lake, Snowflake, Data lakehouse architectures, Data warehouse modernisation
• Transformation & Orchestration - dbt, Apache Airflow, Astronomer, Enterprise workload orchestration, CI/CD for data engineering
• Cloud - AWS, Azure and/or GCP, Cloud storage and data services, IAM and security architecture, Networking for data platforms, Infrastructure as Code, Terraform
• Governance & Reliability - Unity Catalog, Data catalog and metadata management, Data lineage, Data quality, Data observability, DataOps
• Executive Consulting & Client Leadership
• Act as a trusted technical advisor to CIOs, CDOs, CTOs, and senior Data and Analytics executives.
• Translate complex technical issues into business implications and executive decisions.
• Lead technology strategy and architecture workshops.
• Develop modernisation roadmaps and investment strategies.
• Communicate architectural trade‑offs clearly to technical and executive stakeholders.
• Build strong relationships with senior client technology and business leaders.
• Solution Shaping & Programme Growth
• Lead technical solutioning for complex transformation opportunities.
• Develop high‑level architectures, delivery approaches, estimates, and staffing models.
• Shape large‑scale data engineering and modernisation programmes.
• Participate actively in RFP responses and client proposals.
• Develop reusable migration frameworks, accelerators, and engineering assets.

Salary Range- $190,000-$240,000 a year