Data Engineer

Techversantinfotech; Ernakulam; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A technology firm based in Kerala is looking for a skilled Data Engineer with over 4 years of experience. This role involves designing and optimizing ETL/ELT pipelines and working with cloud data services like AWS, Azure, or GCP. The ideal candidate should be proficient in Python and SQL, have experience with data modeling, and possess strong problem-solving skills. A collaborative mindset and the ability to work in a fast-paced environment are essential. Competitive compensation and benefits are offered.

Qualifications

• Minimum 4 years of hands-on experience as a Data Engineer.
• Advanced proficiency in SQL for complex queries and performance tuning.
• Strong understanding of cloud ecosystems and data architecture.

Responsibilities

• Design and build end-to-end ETL/ELT pipelines using AWS, Azure, or GCP.
• Collaborate with teams to deliver clean and reliable data.
• Implement data security policies and ensure compliance.

Skills

Data ingestion and transformation workflows

Cloud data services (AWS, Azure, GCP)

Data modeling and schema design

Python and PySpark

ETL/ELT processes

Education

Bachelor's or Master's degree in Computer Science or related field

Tools

AWS Glue

Airflow

Jenkins

Terraform

Job description

Overview

What’s important to us: We are looking for a skilled and experienced Data Engineer with over 4 years of professional experience in building, automating, and optimizing data pipelines and cloud-based architectures. The ideal candidate will have hands-on experience with cloud data services (AWS, Azure, or GCP) and CI/CD pipelines for deploying scalable, reliable, and secure data solutions.

The candidate will collaborate with cross-functional teams including data analysts, data scientists, and software engineers to design and maintain robust data infrastructure that supports analytics, AI/ML workflows, and enterprise reporting systems.

Responsibilities

• Design, build, and maintain end-to-end ETL/ELT pipelines using both on-premise and cloud-based technologies.
• Architect and operate data storage and streaming solutions leveraging cloud-based services on AWS, Azure, or GCP.
• Design and implement data ingestion and transformation workflows using Airflow, AWS Glue, or Azure Data Factory.
• Develop and optimize data pipelines using Python and PySpark for large-scale distributed data processing.
• Build data models — normalized, denormalized, and dimensional (Star/Snowflake) — for analytics and warehousing solutions.
• Implement data quality, lineage, and governance using metadata management and monitoring tools.
• Collaborate with cross-functional teams to deliver clean, reliable, and timely data for analytics and machine learning use cases.
• Integrate CI/CD pipelines for data infrastructure deployment using GitHub Actions, Jenkins, or Azure DevOps.
• Automate infrastructure provisioning using Infrastructure as Code (IaC) tools such as AWS CloudFormation or Terraform.
• Monitor and optimize data processing performance for scalability, reliability, and cost-efficiency.
• Enforce data security policies and ensure compliance with standards such as GDPR and HIPAA.

Must-Have Skills & Qualifications

• Education: Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Engineering, or a related field.
• Experience: Minimum 4 years of hands-on experience as a Data Engineer or in data-intensive environments.
• SQL Expertise: Advanced proficiency in SQL for complex queries, joins, window functions, and performance tuning.
• Analytical Databases: Experience working with Snowflake, Amazon Redshift, Google BigQuery, Azure Synapse, and PostgreSQL.
• Query Optimization: Skilled in query optimization, indexing, and execution plan analysis for high-performance analytics workloads.
• Programming: Proficient in Python and PySpark for data manipulation, automation, and pipeline orchestration.
• Data Processing Frameworks: Strong understanding of Apache Spark (RDD, DataFrame, Spark SQL, optimization), Hive, Hadoop, and Flink for large-scale distributed data processing.
• ETL/ELT Frameworks: Hands-on experience designing and maintaining pipelines using Airflow, AWS Glue, or Azure Data Factory.
• Data Integration Patterns: Familiar with incremental loading, Slowly Changing Dimensions (SCD), Change Data Capture (CDC), and error handling in data pipelines.
• Data Modeling: Expertise in data modeling, schema design, and building normalized, denormalized, and dimensional (Star/Snowflake) schemas.
• Data Architecture: Strong understanding of Data Warehousing, Data Lakes, and Lakehouse architectures, including Delta Lake, ACID transactions, and partitioning strategies.
• Cloud Platforms: Practical experience with major cloud ecosystems —
  
  • AWS: S3, Glue, Redshift, Athena, Lambda, Step Functions, EMR
  • Azure: Data Factory, Data Lake Gen2, Synapse, Databricks
• Cloud Security: Experience managing IAM roles, access control, and encryption in cloud environments.
• Pipeline Optimization: Skilled in optimizing data pipelines for performance, scalability, and cost-efficiency.
• CI/CD and DevOps: Hands-on experience with CI/CD tools such as GitHub Actions, GitLab CI, or Azure DevOps.
• Version Control: Proficient with Git and familiar with agile development practices.

Good-to-Have Skills

• Experience with containerization and orchestration.
• Exposure to data cataloging and governance tools.
• Experience with monitoring tools.
• Familiarity with data APIs and microservices architecture.
• Certification in cloud data engineering (e.g., AWS Certified Data Engineer, Azure Data Engineer Associate, or GCP Professional Data Engineer).
• Experience supporting machine learning and analytics pipelines.

Soft Skills

• Strong analytical and problem-solving mindset.
• Excellent communication and documentation skills.
• Ability to work collaboratively in a cross-functional, fast-paced environment.
• Strong attention to detail with a focus on data accuracy and reliability.
• Eagerness to learn and adopt emerging data technologies.