Data Engineer

Team Computers; Mumbai, Ahmedabad District; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Team Computers is seeking a skilled Data Engineer to design, develop, and optimize enterprise-grade data pipelines within an on-premises, containerized data platform environment. The candidate should have strong expertise in Python, Apache Spark, SQL/PL-SQL, and container orchestration to support scalable data ingestion, transformation, and analytics workloads.

The role involves building and maintaining robust data solutions that enable Business Intelligence, Analytics, AI/ML initiatives, and

Qualifications

• Bachelor's degree in CS/IT/Engineering or related field.
• Certifications in Data Engineering technologies preferred.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Apache Spark (PySpark/Scala).
• Develop data ingestion frameworks for structured, semi-structured, and unstructured data sources.
• Deploy and manage containerized data workloads within OpenShift or Kubernetes environments.
• Build and maintain Lakehouse architectures using Medallion Architecture (Bronze, Silver, Gold).
• Implement data quality validation, monitoring, and alerting mechanisms.
• Optimize Spark workloads for performance, scalability, and resource efficiency.
• Develop and manage workflow orchestration using Apache Airflow.
• Collaborate with Business, Analytics, DevOps, and Data Governance teams to deliver enterprise data solutions.
• Ensure adherence to data governance, compliance, and security standards.
• Participate in code reviews, troubleshooting, and performance tuning activities.

Skills

Python

Apache Spark / PySpark

SQL / PL SQL

OpenShift / Kubernetes

Airflow

Git / CI/CD

Data warehousing concepts

Delta Lake

Data quality frameworks

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or related field

Tools

Oracle DB

PostgreSQL

OpenShift

Kubernetes

Apache Airflow

Git

CI/CD pipelines

Delta Lake

Job description

Job Summary:

We are looking for a skilled Data Engineer to design, develop, and optimize enterprise‑grade data pipelines within an on‑premises, containerized data platform environment. The ideal candidate should have strong expertise in Apache Spark, Python, SQL/PL‑SQL, and container orchestration technologies to support scalable data ingestion, transformation, and analytics workloads.

The role involves building and maintaining robust data solutions that enable Business Intelligence, Analytics, AI/ML initiatives, and regulatory reporting across the organization.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Apache Spark (PySpark/Scala).
• Develop data ingestion frameworks for structured, semi‑structured, and unstructured data sources.
• Deploy and manage containerized data workloads within OpenShift or Kubernetes environments.
• Build and maintain Lakehouse architectures using Medallion Architecture (Bronze, Silver, Gold layers).
• Implement data quality validation, monitoring, and alerting mechanisms.
• Optimize Spark workloads for performance, scalability, and resource efficiency.
• Develop and manage workflow orchestration using Apache Airflow.
• Collaborate with Business, Analytics, DevOps, and Data Governance teams to deliver enterprise data solutions.
• Ensure adherence to data governance, compliance, and security standards.
• Participate in code reviews, troubleshooting, and performance tuning activities.

Required Skills

Technical Skills:

• Strong experience in Python programming.
• Hands‑on experience with Apache Spark (PySpark preferred; Scala is an advantage).
• Expertise in SQL, PL/SQL, and relational databases such as Oracle and PostgreSQL.
• Experience with Red Hat OpenShift (OCP) or Kubernetes.
• Knowledge of Apache Airflow for workflow orchestration.
• Experience with Git and CI/CD pipelines.
• Understanding of Data Warehousing concepts and Lakehouse architecture.
• Familiarity with Delta Lake or equivalent open‑source storage formats.
• Experience with Data Quality frameworks and monitoring solutions.

Preferred Skills:

• Exposure to Medallion Architecture (Bronze, Silver, Gold).
• Experience working in large‑scale enterprise data environments.
• Certifications in Spark, Kubernetes/OpenShift, Python, or Data Engineering technologies will be an added advantage.

Qualifications:

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• Relevant certifications in Data Engineering technologies are preferred.

Soft Skills:

• Strong analytical and problem‑solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work independently as well as in cross‑functional teams.
• Strong ownership mindset with a focus on quality and timely delivery.