Data Engineer

Techversant Infotech; Ernakulam; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Techversant is seeking a skilled Data Engineer with 4+ years of experience to design, build, and optimize data pipelines across on-prem and cloud environments. You will implement ingestion, transformation, and governance workflows to support analytics, AI/ML, and enterprise reporting.

Ideal candidates will have hands-on experience with Python, PySpark, Spark, and modern ETL/ELT tools. Collaboration with analysts, data scientists, and engineers is essential for reliable data delivery.

Qualifications

• Bachelor’s or Master’s degree in CS/IT or related field.
• 4+ years of hands-on Data Engineer experience.
• Advanced SQL skills for complex queries, joins, and tuning.
• Experience with Snowflake, Redshift, BigQuery, Synapse, PostgreSQL.
• Proficient in Python and PySpark for data manipulation.
• Strong knowledge of Spark, Hive, Hadoop, Flink for large-scale processing.
• Experience with Airflow, AWS Glue, or Azure Data Factory.
• Experience in data modeling and data warehousing.

Responsibilities

• Design, build, and maintain end-to-end ETL/ELT pipelines.
• Architect and operate data storage and streaming solutions on cloud platforms (AWS/Azure/GCP).
• Design and implement ingestion and transformation workflows using Airflow, AWS Glue, or Azure Data Factory.
• Develop data pipelines with Python and PySpark for large-scale processing.
• Build data models (Star/Snowflake) for analytics and warehousing.
• Implement data quality, lineage, and governance using metadata tooling.
• Collaborate with cross-functional teams to deliver reliable data for analytics and ML use cases.
• Integrate CI/CD pipelines for data deployment (GitHub Actions, Jenkins, Azure DevOps).

Skills

Python

PySpark

SQL

Data Modeling

ETL/ELT pipelines

Airflow

Cloud platforms

Spark

Hadoop

Delta Lake

CI/CD

GitHub Actions

DevOps

Data Governance

Data Warehousing

Education

Bachelor’s or Master’s degree in CS/IT/Data Engineering

Tools

Airflow

AWS Glue

Azure Data Factory

Terraform

CloudFormation

GitHub Actions

Jenkins

Azure DevOps

Delta Lake

Job description

What’s important to us:

We are looking for a skilled and experiencedData Engineerwith over 4 years of professional experience in building, automating, and optimizing data pipelines and cloud-based architectures. The ideal candidate will have hands-on experience withcloud data services (AWS, Azure, or GCP)andCI/CD pipelinesfor deploying scalable, reliable, and secure data solutions.

The candidate will collaborate with cross-functional teams including data analysts, data scientists, and software engineers to design and maintain robust data infrastructure that supports analytics, AI/ML workflows, and enterprise reporting systems.

Key Responsibilities:

• Design, build, and maintainend-to-end ETL/ELT pipelinesusing both on-premise and cloud-based technologies.
• Architect and operatedata storage and streaming solutionsleveraging cloud-based services onAWS,Azure, orGCP.
• Design and implementdata ingestion and transformation workflowsusingAirflow,AWS Glue, orAzure Data Factory.
• Develop and optimizedata pipelinesusingPythonandPySparkfor large-scale distributed data processing.
• Builddata models— normalized, denormalized, and dimensional (Star/Snowflake) — for analytics and warehousing solutions.
• Implementdata quality,lineage, andgovernanceusing metadata management and monitoring tools.
• Collaborate withcross-functional teamsto deliver clean, reliable, and timely data for analytics and machine learning use cases.
• IntegrateCI/CD pipelinesfor data infrastructure deployment usingGitHub Actions,Jenkins, orAzure DevOps.
• Automate infrastructure provisioning usingInfrastructure as Code (IaC)tools such asAWS CloudFormationorTerraform.
• Monitor and optimizedata processing performancefor scalability, reliability, and cost-efficiency.
• Enforcedata security policiesand ensure compliance with standards such asGDPRandHIPAA.

Must-Have Skills & Qualifications:

• Education:Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Engineering, or a related field.
• Experience:Minimum 4 years of hands-on experience as a Data Engineer or in data-intensive environments.
• SQL Expertise:Advanced proficiency in SQLfor complex queries, joins, window functions, and performance tuning.
• Analytical Databases:Experience working withSnowflake, Amazon Redshift, Google BigQuery, Azure Synapse, and PostgreSQL.
• Query Optimization:Skilled in query optimization, indexing, and execution plan analysis for high-performance analytics workloads.
• Programming:Proficient inPython and PySparkfor data manipulation, automation, and pipeline orchestration.
• Data Processing Frameworks:Strong understanding ofApache Spark (RDD, DataFrame, Spark SQL, optimization), Hive, Hadoop, and Flinkfor large-scale distributed data processing.
• ETL/ELT Frameworks:Hands-on experience designing and maintaining pipelines usingAirflow, AWS Glue, or Azure Data Factory.
• Data Integration Patterns:Familiar withincremental loading, Slowly Changing Dimensions (SCD), Change Data Capture (CDC), and error handlingin data pipelines.
• Data Modeling:Expertise in data modeling, schema design, and building normalized, denormalized, and dimensional (Star/Snowflake) schemas.
• Data Architecture:Strong understanding ofData Warehousing, Data Lakes, and Lakehouse architectures, including Delta Lake, ACID transactions, and partitioning strategies.
• Cloud Platforms:Practical experience with major cloud ecosystems —
  • AWS:S3, Glue, Redshift, Athena, Lambda, Step Functions, EMR
  • Azure:Data Factory, Data Lake Gen2, Synapse, Databricks
• Cloud Security:Experience managing IAM roles, access control, and encryption in cloud environments.
• Pipeline Optimization:Skilled in optimizing data pipelines for performance, scalability, and cost-efficiency.
• CI/CD and DevOps:Hands-on experience with CI/CD tools such asGitHub Actions, GitLab CI, orAzure DevOps.
• Version Control:Proficient with Git and familiar with agile development practices.

Good-to-Have Skills:

• Experience withcontainerization and orchestration.
• Exposure todata cataloging and governance tools.
• Experience withmonitoring tools.
• Familiarity withdata APIs and microservices architecture.
• Certification in cloud data engineering (e.g., AWS Certified Data Engineer, Azure Data Engineer Associate, or GCP Professional Data Engineer).
• Experience supportingmachine learning and analytics pipelines.

Soft Skills:

• Strong analytical and problem-solving mindset.
• Excellent communication and documentation skills.
• Ability to work collaboratively in a cross-functional, fast-paced environment.
• Strong attention to detail with a focus on data accuracy and reliability.
• Eagerness to learn and adopt emerging data technologies.