Senior Associate-Date Engineer

Acuity Analytics; Bengaluru; On-site; INR 1,500,000 - 3,000,000; Full time

Job summary

Acuity Analytics in Bengaluru seeks a Data Engineer with a strong software engineering mindset focused on Python-based data engineering. This role emphasizes building scalable data platforms using cloud and lakehouse technologies rather than traditional SQL-centric approaches.

You will develop Python and Spark pipelines, work with modern data lake platforms such as Databricks and Snowflake, and deploy solutions on AWS. Experience with Dagster for data orchestration is highly desirable.

Qualifications

• 5+ years of professional experience in Data Engineering.
• Hands-on experience building production-grade data platforms and pipelines.
• Strong software engineering and coding background.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and distributed processing frameworks.
• Build reusable, production-grade data engineering components and frameworks.
• Develop and optimize high-volume batch and streaming data processing solutions.
• Implement robust testing, monitoring, and observability for data pipelines.
• Troubleshoot complex data processing and performance issues.
• Build and maintain modern lakehouse architectures using Databricks, Snowflake, and AWS data services.
• Design scalable data models and storage strategies for structured and semi-structured data.
• Develop ingestion, transformation, validation, and enrichment pipelines.
• Support data quality, lineage, and governance initiatives.
• Develop and manage workflow orchestration solutions using Dagster or similar orchestration frameworks.
• Automate data operations and deployment processes.
• Monitor pipeline performance and optimize reliability and efficiency.
• Build cloud-native data solutions on AWS.

Skills

Python (Expert Level)

PySpark

Data Pipeline Development

Object-Oriented Programming

Debugging and Performance

REST APIs

Git

Tools

Databricks

Snowflake

Delta Lake

Apache Iceberg

Modern Lakehouse Architectures

Job description

Job Purpose

We are seeking a highly skilled Data Engineer with a strong software engineering mindset and deep expertise in Python-based data engineering. This is not a traditional SQL-centric data engineering role. We are specifically looking for engineers with strong programming, debugging, and distributed data processing experience, who can build scalable and maintainable data platforms using modern cloud and lakehouse technologies.

We are seeking a highly skilled Data Engineer with a strong software engineering mindset and deep expertise in Python-based data engineering. This is not a traditional SQL-centric data engineering role. We are specifically looking for engineers with strong programming, debugging, and distributed data processing experience, who can build scalable and maintainable data platforms using modern cloud and lakehouse technologies. The ideal candidate will have experience developing data pipelines using Python and Spark, working with modern data lake platforms such as Databricks and Snowflake, and deploying solutions on AWS. Experience with Dagster for data orchestration is highly desirable.

Key Responsibilities

Data Platform Development

• Design, develop, and maintain scalable data pipelines using Python and distributed processing frameworks.


• Build reusable, production-grade data engineering components and frameworks.


• Develop and optimize high-volume batch and streaming data processing solutions.


• Implement robust testing, monitoring, and observability for data pipelines.


• Troubleshoot complex data processing and performance issues.

Lakehouse & Data Engineering

• Build and maintain modern lakehouse architectures using Databricks, Snowflake, and AWS data services.


• Design scalable data models and storage strategies for structured and semi-structured data.


• Develop ingestion, transformation, validation, and enrichment pipelines.


• Support data quality, lineage, and governance initiatives.

Orchestration & Automation

• Develop and manage workflow orchestration solutions using Dagster or similar orchestration frameworks.


• Automate data operations and deployment processes.


• Monitor pipeline performance and optimize reliability and efficiency.

Cloud Engineering

• Build cloud-native data solutions on AWS.


• Integrate data services across storage, processing, orchestration, and analytics platforms.


• Implement security, scalability, and resiliency best practices.

Engineering Excellence

• Write clean, maintainable, and well-tested code.


• Participate in peer code reviews and technical design discussions.


• Collaborate with product managers, architects, and engineering teams.


• Follow software engineering best practices, including CI/CD and version control.

Key competencies

Experience

• 5+ years of professional experience in Data Engineering.


• Hands-on experience building production-grade data platforms and pipelines.


• Strong software engineering and coding background.

Technical Skills

Primary Skills (Must Have)

• Python (Expert Level)


• PySpark / Apache Spark


• Data Pipeline Development


• Object-Oriented Programming


• Debugging and Performance Optimization


• REST APIs


• Git

Modern Data Platforms

• Experience with one or more:


• Databricks


• Snowflake


• Delta Lake


• Apache Iceberg


• Modern Lakehouse Architectures

AWS Experience

Experience with multiple services such as:

• S3


• Lambda


• Glue


• EMR


• ECS/EKS


• Step Functions


• RDS


• CloudWatch

Data Engineering Practices

• ETL/ELT Development


• Data Modeling


• Data Quality Frameworks


• Data Validation and Monitoring


• CI/CD for Data Pipelines