Lead I - Data Engineering

UST; Coimbatore District; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

UST is seeking a highly skilled Data Engineer to design, develop, and deploy scalable data solutions using AWS Cloud, Apache Spark, PySpark, and ETL development. The ideal candidate will have hands-on experience in building data pipelines, analyzing large datasets, and driving production-ready solutions.

The role emphasizes deploying scalable data architectures, workflow automation, and the use of CI/CD practices to ensure reliability and performance across data processing workflows.

Qualifications

• Bachelor's or Master's degree in Computer Science, IT, Engineering, or a related field.
• 4+ years of experience in Data Engineering, Big Data, or related roles.
• Strong hands-on experience with Apache Spark and PySpark.
• Experience with AWS Cloud services for data processing and analytics.
• Solid understanding of Big Data architectures and distributed computing concepts.
• Expertise in designing and implementing ETL pipelines.
• Experience with workflow automation and pipeline orchestration tools.
• Knowledge of CI/CD practices and deployment automation.
• Strong analytical and problem-solving skills with large datasets.
• Experience in functional testing and validation of data applications.
• Strong programming and scripting skills, preferably in Python.

Responsibilities

• Design, develop, and deploy data tools that support business operations and product initiatives.
• Build and maintain scalable data pipelines and ETL processes using Apache Spark and PySpark.
• Perform analysis and processing of large-scale datasets leveraging AWS cloud services.
• Own product features throughout the software development lifecycle, including design, development, testing, deployment, and production support.
• Evaluate emerging big data technologies and prototype innovative solutions to enhance data processing capabilities and architecture.
• Develop and automate data workflows and pipelines to improve efficiency, reliability, and scalability.
• Implement best practices for CI/CD, code quality, testing, and deployment automation.
• Conduct functional testing and validation of data solutions to ensure accuracy and performance.
• Monitor, troubleshoot, and optimize data processing jobs and workflows.
• Work closely with stakeholders to understand data requirements and translate them into scalable technical solutions.

Skills

Apache Spark

PySpark

AWS Cloud

ETL pipelines

CI/CD

Python

Big Data

Education

Bachelor's or Master's in Computer Science/IT/Engineering

Tools

Airflow

Job description

Who we are:

At UST, we help the world s best organizations grow and succeed through transformation. Bringing together the right talent, tools, and ideas, we work with our client to co-create lasting change. Together, with over 30,000 employees in 30+ countries, we build for boundless impact-touching billions of lives in the process. Visit us at UST.com.

Summary:

We are seeking a highly skilled Data Engineer to design, develop, and deploy scalable data solutions that support both operational and product-driven business needs. The ideal candidate will have strong expertise in AWS Cloud, Apache Spark/PySpark, ETL development, workflow automation, and big data technologies. This role requires hands‑on experience in building data pipelines, analysing large datasets, optimizing data processing architectures, and driving solutions from development through production deployment.

Key Responsibilities

• Collaborate with cross-functional teams to design, develop, and deploy data tools that support business operations and product initiatives.
• Build and maintain scalable data pipelines and ETL processes using Apache Spark and PySpark.
• Perform analysis and processing of large-scale datasets leveraging AWS cloud services.
• Own product features throughout the software development lifecycle, including design, development, testing, deployment, and production support.
• Evaluate emerging big data technologies and prototype innovative solutions to enhance data processing capabilities and architecture.
• Develop and automate data workflows and pipelines to improve efficiency, reliability, and scalability.
• Implement best practices for CI/CD, code quality, testing, and deployment automation.
• Conduct functional testing and validation of data solutions to ensure accuracy and performance.
• Monitor, troubleshoot, and optimize data processing jobs and workflows.
• Work closely with stakeholders to understand data requirements and translate them into scalable technical solutions.

Required Skills

• Bachelors or Masters degree in Computer Science, Information Technology, Engineering, or a related field.
• 4+ years of experience in Data Engineering, Big Data, or related roles.
• Strong hands‑on experience with Apache Spark and PySpark.
• Experience working with AWS Cloud services for data processing and analytics.
• Solid understanding of Big Data architectures and distributed computing concepts.
• Expertise in designing and implementing ETL pipelines.
• Experience with workflow automation and pipeline orchestration tools.
• Knowledge of CI/CD practices and deployment automation.
• Strong analytical and problem‑solving skills with the ability to work with large and complex datasets.
• Experience in functional testing and validation of data applications.
• Strong programming and scripting skills, preferably in Python.

Preferred Skills

• Experience with data lake and cloud‑native data architectures.
• Knowledge of containerization technologies and DevOps practices.
• Familiarity with real‑time and batch data processing frameworks.
• Experience in performance tuning and optimization of Spark applications.
• Understanding of data governance, security, and data quality best practices

What we believe:

We re proud to embrace the same values that have shaped UST since the beginning. Since day one, we ve been building enduring relationships and a culture of integrity. And today, its those same values that are inspiring us to encourage innovation from everyone, to champion diversity and inclusion and to place people at the centre of everything we do.

Humility:

We will listen, learn, be empathetic and help selflessly in our interactions with everyone.

Humanity:

Through business, we will better the lives of those less fortunate than ourselves.

Integrity:

We honour our commitments and act with responsibility in all our relationships.

Equal Employment Opportunity Statement

UST is an Equal Opportunity Employer. We believe that no one should be discriminated against because of their differences, such as age, disability, ethnicity, gender, gender identity and expression, religion, or sexual orientation.

All employment decisions shall be made without regard to age, race, creed, colour, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

UST reserves the right to periodically redefine your roles and responsibilities based on the requirements of the organization and/or your performance.

- To support and promote the values of UST.

- Comply with all Company policies and procedures