Data Engineer

Tata Consultancy Services; Quebec; Presencial; CAD 90.000 - 130.000; Jornada completa

Descripción de la vacante

Tata Consultancy Services in Canada is seeking a Software Developer to design and implement scalable Python-based data processing solutions using Spark Structured Streaming on Databricks.

You will collaborate with cross-functional teams, develop IaC with Terraform, and contribute to CI/CD pipelines using GitHub Actions in an Agile environment.

Formación

• Proven experience as a Software Developer with Python focus.
• Experience in Spark Structured Streaming and Databricks.
• Strong debugging/optimizing Spark jobs on Databricks.
• Experience with CI/CD pipelines and GitHub Actions.
• Agile/Scrum experience.

Responsabilidades

• Build and maintain Spark Structured Streaming applications using PySpark.
• Ingest and transform data in Avro, JSON, and CSV formats.
• Develop Terraform scripts and DevOps processes to support IaC.
• Lead development initiatives with a focus on quality and on-time delivery.
• Mentor teammates and collaborate with product, security, and DevOps teams.
• Ensure code quality, reviews, tests, and CI/CD adherence.

Conocimientos

Python

PySpark

Spark Structured Streaming

Databricks

Azure

Terraform

CI/CD

GitHub Actions

Agile/Scrum

Problem solving

Herramientas

Databricks

Azure

Terraform

CI/CD tooling

Descripción del empleo

Tata Consultancy Services (TCS) is an equal opportunity employer, and embraces diversity in race, nationality, ethnicity, gender, age, physical ability, neurodiversity, and sexual orientation, to create a workforce that reflects the societies we operate in. Our continued commitment to Culture and Diversity is reflected in our people stories across our workforce and implemented through equitable workplace policies and processes.

TCS has been serving the Canadian marketplace for more than 35 years and today is the technology partner of choice for many of the country's leading organizations. As one of the largest IT services providers in Canada and globally, TCS is aiming to become the world's largest AI-led technology services company and is enabling its clients to transform themselves across the full AI stack, from infrastructure to intelligence.

Rooted in the heritage and values of the Tata Group, TCS is focused on creating long-term value for its clients, investors, employees, and the communities it serves. With a highly skilled workforce supporting customers from coast to coast, TCS has consistently been recognized as a Top Employer by the Top Employers Institute, including a recent top ranking at the country level.

TCS also sponsors 14 of the world's most prestigious marathons and endurance events, including the TCS Toronto Waterfront Marathon, reflecting its commitment to health, sustainability, and community empowerment.

Job Description :-

Must Have Technical/Functional Skills

• Proven experience as a Software Developer, with a strong focus on building scalable and efficient Python applications.
• Experience in developing Spark Structured Streaming applications is highly desirable.
• Minimum of good years of professional software development experience.
• Strong analytical and problem-solving skills, with the ability to debug and optimize Spark jobs running on Databricks.
• Ability to work closely with cross-functional teams to deliver high-quality streaming solutions.

Technical Skills:

• Strong expertise in Python, PySpark, and Spark Structured Streaming.
• Experience with Databricks and Azure.
• Familiarity with Delta Lake and Terraform scripting.
• Proficiency in working with varied data file formats (Avro, JSON, CSV) for ingestion and transformation.

Software Development:

• Proficiency in Object-Oriented Programming (OOP) concepts and software design principles.
• Ability to write clean, maintainable, and scalable Python code.
• GitHub Actions:
• Experience in setting up and managing CI/CD pipelines using GitHub Actions to ensure smooth and automated deployment processes.

Agile Methodology:

• Experience working in an Agile/Scrum environment, with a focus on iterative development, continuous feedback, and delivery.

Nice to Haves:

1. Python Unit Testing.

2. Unity Catalog.

3. Databricks Asset Bundles.

4. Unit Testing/Mocking

• Development:
• Build and maintain Spark Structured Streaming applications using PySpark.
• Work with varied data file formats (Avro, JSON, CSV) for ingestion and transformation.
• Implement Terraform scripts and DevOps processes to support infrastructure as code.
• Lead Development Initiatives:
• Take ownership of development initiatives, ensuring timely delivery while meeting or exceeding quality standards.
• Develop code with a focus on "First Time Right" (FTR) principles and high quality.
• Code Quality and Maintenance:
• Oversee code quality and enforce best practices, including code reviews, unit testing, CI/CD, and adherence to coding standards.
• Identify, prioritize, and implement strategies to reduce technical debt and address vulnerabilities in the codebase.
• Provide mentorship and guidance to team members, fostering a collaborative, innovative, and high-performance environment.
• Work closely with cross-functional teams, including product management, Cloud Security, LOB, Network Security, and DevOps, to integrate features into broader solutions.
• Problem Solving:
• Address and troubleshoot complex technical issues, providing solutions that enhance system performance and user experience.
• Documentation:
• Ensure comprehensive documentation of systems, processes, and code to facilitate knowledge sharing and maintenance.
• Stakeholder Communication:
• Communicate progress, challenges, and solutions to stakeholders, ensuring transparency and alignment with business objectives.

Thank you for your interest in TCS. Candidates that meet the qualifications for this position will be contacted within a 2-week period. We invite you to continue to apply for other opportunities that match your profile.