Data Engineer - AWS & Azure

HCL Technologies Limited; Dadri, Chennai, Bengaluru; Hybrid; INR 2,400,000 - 4,000,000; Full time

Job summary

HCL Technologies Limited in PAN India seeks a seasoned Data Engineer to design, build, and maintain scalable data pipelines on Azure and AWS. The role requires 9+ years of experience and a strong background in data architecture, integration, and analytics.

You will work with product owners and core teams in a hybrid setup, improve data quality, automate ETL jobs, and ensure secure data storage while following SDLC and Agile practices.

Qualifications

• Over 9 years of experience as a Data Engineer or in a similar role.
• Experience with Azure & AWS.
• Experience with data pipeline, ETL and workflow management tools (e.g., Databricks, Data Factory).

Responsibilities

• Data Pipeline Development: Design, build, and maintain robust, scalable data pipelines to collect, process, and store large volumes of data from various sources.
• Data Integration: Integrate data from multiple sources, including APIs, databases, and external datasets, ensuring data consistency and reliability.
• Data Modelling: Develop and maintain data models and schemas that support efficient data storage, retrieval, and analytics.
• Database Management: Manage and optimize databases, ensuring their performance, availability, and security.
• Data Quality: Implement and monitor data quality checks to ensure the accuracy, completeness, and consistency of data.
• Automation: configure data extraction and load jobs and automate repetitive tasks and processes to improve efficiency.
• Documentation: Maintain clear and comprehensive documentation of data pipelines, data models, and data integration processes.
• Develop understanding of business/processes and high-level understanding of high-quality digital product delivery.
• Ways of working: Follow Agile and SDLC processes including the creation of data related user stories.
• Perform code peer reviews and testing as per SDLC or industry standards.
• Collaboration: Work closely with internal customers, product owners, core team members, and other stakeholders to understand their data needs and provide appropriate solutions.
• Technical collaboration and oversight with Vendors as required. Create SOWs and work packets with vendors as required.

Skills

Azure

AWS

SQL

Python

Data modeling

ETL

Databricks

Power BI

JIRA

Education

Bachelor’s or Master’s degree in CS/Data/SE or related field

Tools

Data Factory

Databricks

Jenkins

Azure DevOps

Bitbucket

Confluence

Job description

Job Title: Data Engineer - Azure & AWS

Location:PAN India

Experience: 9+ Years

Work Mode: Hybrid

We are seeking a skilled and motivated Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, data architecture, and data integration. This role involves building and maintaining scalable data pipelines, ensuring data quality and integrity, and supporting data analytics initiatives. You will work closely with internal customers, product owners, core team members, and other stakeholders to ensure that quality data is available, reliable, and easily accessible in a cost effective efficient, scalable way.

Key Responsibilities:

• Data Pipeline Development: Design, build, and maintain robust, scalable, and efficient data pipelines to collect, process, and store large volumes of data from various sources.
• Data Integration: Integrate data from multiple sources, including APIs, databases, and external datasets, ensuring data consistency and reliability.
• Data Modelling: Develop and maintain data models and schemas that support efficient data
• storage, retrieval, and analytics.
• Database Management: Manage and optimize databases, ensuring their performance,
• availability, and security
• Data Quality: Implement and monitor data quality checks to ensure the accuracy, completeness, and consistency of data. Perform analysis required to troubleshoot data related
• issues and assist in the resolution of data issues.
• Automation: configure data extraction and load jobs and automate repetitive tasks and processes to improve efficiency and reduce errors. Improve performances of jobs and
• pipelines by applying optimisation and automation techniques.
• Documentation: Maintain clear and comprehensive documentation of data pipelines, data models, and data integration processes. Maintenance of time allocation reports.
• Develop understanding of business/ processes and high-level understanding of high-quality
• digital product delivery
• Ways of working: Follow Agile and SDLC processes including the creation of data related user
• stories.
• Perform code peer reviews and testing as per SDLC or industry standards.
• Collaboration: Work closely with internal customers, product owners, various core team
• members, and other stakeholders to understand their data needs and provide appropriate
• solutions.
• Technical collaboration and oversight with Vendors as required. Create SOWs and work packets with vendors as required.

Education:

• Bachelors or Masters degree in computer science, data science, software engineering, information systems or a similar field or equivalent experience.

Experience and skills:

• Over 9years experience as a Data Engineer or in a similar role.
• Experience with both Azure & AWS.
• Experience with data pipeline, ETL and workflow management tools (e.g., Databricks, Data Factory).
• Proficiency in programming languages such as SQL, Python, R, or Scala.
• Strong experience with SQL and database management (e.g., MySQL, PostgreSQL, SQL Server).
• Excellent problem-solving and analytical skills.
• Strong understanding of data architecture, data modelling, and ETL processes.
• Ability to work in a fast-paced, dynamic environment and manage multiple tasks simultaneously.
• Strong communication and collaboration skills.
• Experience with best practice UX / UI design & development of Business Intelligence data visualisation / exploration tools (Power Bi, Qlik etc)
• Familiarity with SDLC and Agile ways of working.
• Familiarity with DevOps tools and practices (e.g. Jenkins, Azure DevOps, etc).
• Familiarity with Atlassian tools like Bitbucket, JIRA, Confluence.

Preferred:

• GxP and Non GxP SDLC experience
• Knowledge of data governance and data security best practices.
• MS Power Platform / power apps experience
• Cloud DevOps knowledge (e.g. Terraform)
• Certifications in cloud data technologies (e.g. Azure Data Fundamentals)
• Basic data science experience and knowledge.

Other:

• English language required.
• May require travel 10-20% of time based on project locations.