Big Data Engineer

Tata Consultancy Services; Mississauga; On-site; CAD 100,000 - 130,000; Full time

Job summary

Tata Consultancy Services Canada is seeking a Data Engineer to design and develop big data pipelines using Spark and Scala, working with Hive/Hadoop in a Canadian enterprise environment.

You will analyze source data, implement transformations, standardize data models, and ensure data quality for high-volume datasets in a collaborative, diverse team in Mississauga.

Qualifications

• Design and develop big data pipelines using Spark and Scala.
• Analyze Hive tables, source schemas and data dictionaries.
• Transform raw data into standardized conformed views/tables.
• Implement complex transformations with joins, filters and aggregations.
• Develop reusable Spark/Scala components for data enrichment and validation.
• Implement data quality controls for completeness and accuracy.
• Standardize data across multiple source systems.
• Handle incremental, partition-based and historical data processing.

Responsibilities

• Design and develop big data pipelines using Spark/Scala.
• Analyze Hive tables and source schemas to understand data.
• Translate raw data into standardized conformed views.
• Implement data quality controls and data validation.

Skills

Spark

Scala

Data pipelines

Big data

Tools

Hive

Hadoop

Job description

Tata Consultancy Services (TCS) is an equal opportunity employer, and embraces diversity in race, nationality, ethnicity, gender, age, physical ability, neurodiversity, and sexual orientation, to create a workforce that reflects the societies we operate in. Our continued commitment to Culture and Diversity is reflected in our people stories across our workforce and implemented through equitable workplace policies and processes.

TCS has been serving the Canadian marketplace for more than 35 years and today is the technology partner of choice for many of the country's leading organizations. As one of the largest IT services providers in Canada and globally, TCS is aiming to become the world's largest AI-led technology services company and is enabling its clients to transform themselves across the full AI stack, from infrastructure to intelligence.

Rooted in the heritage and values of the Tata Group, TCS is focused on creating long-term value for its clients, investors, employees, and the communities it serves. With a highly skilled workforce supporting customers from coast to coast, TCS has consistently been recognized as a Top Employer by the Top Employers Institute, including a recent top ranking at the country level.

TCS also sponsors 14 of the world's most prestigious marathons and endurance events, including the TCS Toronto Waterfront Marathon, reflecting its commitment to health, sustainability, and community

Role Description

• Design and develop Big Data pipelines using Spark and Scala to process large volumes of raw data stored in Hive/Hadoop.
• Analyze raw Hive tables, source schemas, data dictionaries and source-to-target mappings to understand business data.
• Develop Spark/Scala transformation frameworks to convert raw source data into standardized conformed data views/tables.
• Implement complex business transformation rules including joins, filters, aggregations, derivations and reference-data lookups.
• Standardize data across multiple source systems to create a consistent enterprise/conformed data model.
• Implement incremental processing, partition-based processing and historical data handling for high-volume datasets.
• Develop reusable Spark/Scala components for data enrichment, transformation and validation.
• Implement data-quality controls for completeness, uniqueness, validity, referential integrity and business-rule validation.

Salary Range

CA$ 100,000 - CA$ 130,000 Per Year

Note:

TCS does not use artificial intelligence tools for candidate screening or evaluation.

This posting is for a current vacancy

The hiring process includes an initial screening by the TCS Hiring Team, followed by a technical evaluation and managerial discussion conducted by the Business Team, and concluding with the final HR evaluation.

Tata Consultancy Services Canada Inc. is committed to meeting the accessibility needs of all individuals in accordance with the Accessibility for Ontarians with Disabilities Act (AODA) and the Ontario Human Rights Code (OHRC). Should you require accommodation during the recruitment and selection process, please inform Human Resources.

Thank you for your interest in TCS. Candidates that meet the qualifications for this position will be contacted within a 2-week period.