Big Data Engineer

Altimetrik; Hyderabad, Pune District, Bengaluru; Hybrid; INR 1,500,000 - 2,800,000; Full time

Job summary

Altimetrik in Hyderabad is seeking a skilled Data Engineer to design and optimize large-scale data pipelines. You will work with Python, Spark, and SQL to transform, load, and quality-check data across batch and streaming workloads.

You will collaborate with data scientists and product teams, implement Airflow workflows, monitor performance, and uphold best practices for distributed systems in a cloud environment.

Qualifications

• Strong Python programming including OOP, multithreading, memory management and performance optimization.

Skills

Python

SQL

Memory management

Debugging

Tools

PySpark

Apache Spark

Airflow

Kafka

Databricks

Docker

Terraform

CloudFormation

Git

Job description

Mandatory Skill sets

• Strong Python Programming: Core Python, OOP, multithreading/multiprocessing, memory management, exception handling, debugging, and performance optimization.
• PySpark & Apache Spark: DataFrames/RDDs, transformations, joins, partitioning, shuffle, Spark internals (DAG, Catalyst, AQE), and performance tuning
• Advanced SQL: Complex joins, window functions, CTEs, query optimization, indexing, execution plans, stored procedures, and database design.

Good to have

• ETL: ETL/ELT pipeline development, data modelling, batch & streaming processing, data lakes, Parquet/Delta Lake, and data quality frameworks.
• Cloud & Big Data Ecosystem: AWS (S3, EMR, Glue, IAM), Databricks, Airflow, Kafka, CI/CD, Git, Docker, and Infrastructure-as-Code (Terraform/CloudFormation).
• System Design & Production Support: Designing scalable data pipelines, troubles
• hooting production issues, monitoring, performance tuning, root cause analysis, and best practices for distributed systems.