Consulting Partner - Data Engineering & Modern Data Platforms
Tata Consultancy Services
A complete application in a minute
Tailored resume and cover letter, ready to send.
Job summary
Tata Consultancy Services Limited in Irvine, CA seeks a Consulting Partner specializing in Data Engineering & Modern Data Platforms. You will lead enterprise-scale transformations, architect data platforms, and drive modernization from executive discussions to hands-on engineering reviews.
The role requires deep expertise in Spark/PySpark, SQL, cloud data platforms, dbt, and Airflow, with proven ability to collaborate across engineers, architects, platform teams, and senior stakeholders.
Qualifications
- Minimum 15+ years in data engineering, data platforms, analytics, or technology architecture.
- At least 10+ years of hands-on experience in data engineering and data platforms.
- Demonstrated ability to architect and deliver enterprise-scale data platforms.
- Hands-on Spark/PySpark and SQL expertise.
- Proven experience with cloud data platforms such as Databricks or Snowflake.
- Experience with dbt and modern data transformation frameworks.
- Experience with Airflow and orchestration technologies.
- Strong understanding of distributed computing, performance, and scalability.
- Experience leading large-scale data modernization or migration programs.
- Ability to engage with engineers, architects, platform teams, and executives
Responsibilities
- Lead technical consulting and architecture for large data modernization programs.
- Provide hands-on data engineering leadership when required.
- Oversee programs with hundreds or thousands of pipelines and datasets.
- Define migration strategies for legacy ETL to modern ELT and cloud migrations.
- Review SQL transformations, PySpark/Spark apps, and dbt models.
- Guide data governance, security, and lifecycle management.
- Review Airflow DAGs and orchestration architecture; ensure performance optimization.
- Communicate architecture trade-offs to executive stakeholders.
Skills
Tools
Job description
- 15+ years of experience in data engineering, data platforms, analytics, or technology architecture.
- 10+ years of deep hands-on technical experience in data engineering and data platform technologies.
- Demonstrated experience architecting and delivering complex enterprise-scale data platforms.
- Strong hands-on experience with Spark/PySpark and SQL.
- Strong practical experience with at least one modern cloud data platform such as Databricks or Snowflake.
- Experience with modern data transformation frameworks such as dbt.
- Experience with workflow orchestration technologies such as Airflow.
- Strong understanding of distributed computing, data processing performance, and scalability.
- Experience leading large-scale data modernization or migration programs.
- Ability to engage directly with engineers, architects, platform teams, and executive stakeholders.
- Cloud-to-cloud data platform migration.
- Migration of large PySpark estates.
- Legacy ETL modernization.
- Stored procedure modernization to dbt or modern transformation frameworks.
- Databricks platform engineering.
- Data platform governance and security.
- Large-scale data migration involving hundreds of terabytes or petabyte-scale environments.
- AI-assisted engineering and migration accelerators.
- Building reusable engineering frameworks and accelerators.
- Experience with platform reliability, FinOps, and performance optimization.
We are seeking a highly accomplished Consulting Partner with deep hands-on Data Engineering and Modern Data Platform expertise to lead complex enterprise data transformation programs.
This is a technical leadership role, not a purely advisory or relationship-management position. The successful candidate will combine executive consulting capabilities with strong architecture and engineering expertise and must have demonstrated hands‑on experience designing, building, modernizing, and troubleshooting large‑scale data platforms and pipelines.
The Consulting Partner will work with client executives to shape transformation strategies while remaining sufficiently close to technology and engineering teams to influence architecture, solve complex technical problems, review implementations, and guide critical delivery decisions.
The role requires an individual who can move seamlessly from an executive discussion with a CIO or CDO to a deep technical discussion involving Databricks, Spark/PySpark, dbt, SQL, cloud architecture, data pipelines , orchestration, performance optimization, and large-scale data migration.
- Serve as the senior technical advisor for complex data engineering and data platform transformation programs.
- Define enterprise data platform, lakehouse, and data engineering architectures.
- Lead architecture assessments and develop modernization strategies for legacy and cloud-based data ecosystems.
- Design scalable architectures covering data ingestion, transformation, storage, orchestration, governance, security, and consumption.
- Review and challenge architecture decisions made by engineering and solution teams.
- Establish technology standards, engineering principles, and reference architectures.
- Identify technical risks and scalability challenges early in the program lifecycle.
- Provide hands‑on technical guidance for complex architecture and engineering issues.
The Consulting Partner must maintain strong hands‑on technical depth and be capable of working directly with engineering teams when required.
Responsibilities include:
- Reviewing and optimizing PySpark and Spark applications.
- Analyzing query and pipeline performance issues.
- Reviewing complex SQL transformations.
- Designing and reviewing dbt models, transformation frameworks, and testing strategies.
- Troubleshooting complex data processing and distributed computing issues.
- Reviewing data partitioning, clustering, file sizing, and storage optimization strategies.
- Providing technical direction on Delta Lake, table design, and data lifecycle management.
- Reviewing Airflow DAGs and orchestration architecture.
- Helping teams resolve critical production issues and performance bottlenecks.
- Conducting architecture and code reviews for high‑risk or complex components.
The role does not require day‑to‑day individual contributor coding but requires the ability to get hands‑on with technology when necessary and independently validate engineering decisions.
- Lead the technical direction of large‑scale data engineering and modernization programs.
- Oversee programs involving hundreds or thousands of pipelines, jobs, tables, and datasets.
- Define technical migration strategies for:
oLegacy ETL to modern ELT
oStored procedures to dbt
oLegacy Spark platforms to Databricks
oCloud‑to‑cloud data platform migration
oAutosys and legacy scheduling to Airflow
oData warehouse modernization
oData lake and lakehouse implementation
- Define migration patterns, factory approaches, automation strategies, and validation frameworks.
- Establish technical governance across multiple engineering teams and workstreams.
- Resolve cross‑platform and cross‑domain technical dependencies.
- Define technical KPIs for scalability, reliability, performance, data quality, and cost efficiency.
The candidate should have strong practical experience across several of the following technologies:
- Data Engineering- Apache Spark, PySpark, SQL, Python, Distributed data processing, Data pipeline design and optimization
- Modern Data Platforms- Databricks, Delta Lake, Snowflake, Data lakehouse architectures, Data warehouse modernization
- Transformation & Orchestration- dbt, Apache Airflow, Astronomer, Enterprise workload orchestration, CI/CD for data engineering
- Cloud - AWS, Azure and/or GCP, Cloud storage and data services, IAM and security architecture, Networking for data platforms, Infrastructure as Code, Terraform
- Governance & Reliability- Unity Catalog, Data catalog and metadata management, Data lineage, Data quality, Data observability, DataOps
- Act as a trusted technical advisor to CIOs, CDOs, CTOs, and senior Data and Analytics executives.
- Translate complex technical issues into business implications and executive decisions.
- Lead technology strategy and architecture workshops.
- Develop modernization roadmaps and investment strategies.
- Communicate architectural trade‑offs clearly to technical and executive stakeholders.
- Build strong relationships with senior client technology and business leaders.
- Lead technical solutioning for complex transformation opportunities.
- Develop high‑level architectures, delivery approaches, estimates, and staffing models.
- Shape large‑scale data engineering and modernization programs.
- Participate actively in RFP responses and client proposals.
- Develop reusable migration frameworks, accelerators, and engineering assets.
Salary Range- $190,000-$240,000 a year
Irvine, CA
TECHNOLOGY
Functional Consultant
437799
Data architect
$190,000 - $240,000
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
