AWS Data Engineer

DATAECONOMY Inc; Hyderabad; Hybrid; INR 1,500,000 - 2,000,000; Full time

Benefits offered by this job

Health Insurance

Group Personal Accident Insurance

Group Term Life Insurance

Hybrid work arrangements

Annual leave 21 days + 10 holidays

Break-out area with refreshments

Job summary

DATAECONOMY Inc is seeking an experienced Data Engineer for our Hyderabad/Pune teams. The role focuses on building and optimizing data pipelines using Python, PySpark, and Spark SQL in a hybrid work environment.

You will design end-to-end ETL/ELT processes, handle large datasets, and collaborate with data architects and stakeholders to deliver robust data solutions.

Qualifications

• Strong Python programming with OOP and data structures.
• Hands-on PySpark DataFrames and Spark SQL experience.
• Proficient in SQL with complex queries (joins, subqueries, CTEs, window functions).
• Experience building ETL/ELT pipelines for large datasets.
• Familiar with Git and SDLC best practices.

Responsibilities

• Develop, maintain, and optimize data processing applications using Python and PySpark.
• Build scalable ETL/ELT pipelines for large volumes of data.
• Perform data extraction, cleansing, transformation, and validation.
• Work with Apache Spark for distributed data processing and large-scale transformations.
• Write complex SQL queries and optimize database operations.
• Develop reusable Python modules and libraries.
• Collaborate with data engineers, data architects, analysts, and stakeholders.
• Follow coding standards, version control, and Agile methodologies.

Skills

Python

PySpark

Spark SQL

SQL

ETL/ELT pipelines

Git

Airflow

CI/CD

Docker

Education

BTech/MTech/MCA

Tools

Databricks

Azure Data Factory

AWS Glue

Delta Lake

Kafka

Job description

Job Title: Data Engineer Experience Level: 4 - 6 years Location: Hyderabad / Pune

Qualification: BTech/MTech/MCA Mode of Work: Hybrid

We are seeking an experienced AWS DataEngineer with deep expertise in Banking, Financial Services, and Insurance systems. The candidatemust be familiar at least in one Data ware housing conceptssuch as ODS/Marts and must have done at least one end-to-enddata processing pipeline designing.He/she should be capable of handling the ETL design andimplementation solely and would work towards creating apositive and innovation friendly environment. The incumbentwould be responsible for creating new capabilities.

Responsibilities

• Develop, maintain, and optimize data processing applications using Python and PySpark.
• Build scalable ETL/ELT pipelines to process large volumes of structured and unstructured data.
• Perform data extraction, cleansing, transformation, aggregation, and validation.
• Work with Apache Spark for distributed data processing and large-scale data transformations.
• Write complex SQL queries and optimize database operations.
• Develop reusable Python modules, functions, and libraries.
• Optimize Spark jobs by analyzing execution plans, partitioning, caching, and performance bottlenecks.
• Work with file formats such as CSV, JSON, Parquet.
• Integrate data from multiple sources, including databases, APIs, and cloud storage.
• Implement error handling, logging, monitoring, and data quality checks.
• Collaborate with data engineers, data architects, analysts, and business stakeholders.
• Follow coding standards, version control practices, and Agile development methodologies.

Requirements

• Strong programming knowledge in Python, including functions, OOP, exception handling, and data structures.
• Hands-on experience with PySpark DataFrames, Spark SQL, transformations, and actions.
• Good understanding of Apache Spark architecture and distributed computing.
• Strong SQL skills, including joins, sub queries, CTEs, and window functions.
• Experience developing ETL/ELT pipelines and handling large datasets.
• Knowledge of data processing formats such as Parquet, JSON, and CSV.
• Experience with Git and software development best practices.
• Understanding of debugging, performance tuning, and troubleshooting.

Preferred Skills

• Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Familiarity with Databricks, Azure Data Factory, AWS Glue, or similar data engineering tools.
• Knowledge of data warehouses and lake house architectures.
• Experience with Airflow or other workflow orchestration tools.
• Familiarity with CI/CD pipelines, Docker, and automated testing.
• Knowledge of Delta Lake, Kafka, or streaming data processing.

Benefits & Perks

• Healthinsurance of INR 5.0 Lakhs for you and your family (up to 6members), ensuring complete peace of mind.
• GroupPersonal Accident Insurance and Group Term Life Insurance tosafeguard you and your loved ones.
• PF andGratuity provided as per standard government regulations.
• Enjoy hybridwork arrangements & flexible working hours.
• 21 days ofannual leave, in addition to 10 company-declared holidays.
• Access to adedicated break-out area with round-the-clock refreshmentsfor relaxation and rejuvenation.