Senior Associate

Axtria - Ingenious Insights; Bengaluru; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

Axtria - Ingenious Insights in Bengaluru seeks a hands-on Project Lead with 4-6 years of experience to design, develop, and deploy data integration pipelines using Python/PySpark, AWS Glue, and Databricks. The role requires deep expertise in Spark-based distributed computing and proficiency with AWS services such as EC2, S3, IAM, Athena, and VPC.

You will lead by example, mentor 1-2 team members, and contribute to architecture decisions, data modeling (CRISP-DM), and pharma-domain insights.

Qualifications

• 4-6 years of experience designing, developing, and deploying data integration pipelines using Python/PySpark on AWS Glue/Databricks.
• Deep understanding of AWS Databricks, Spark-based distributed computing, and hands-on programming with DataFrames in Python/Scala.
• Experience with notebooks/workspace/MLFlow, security management, dashboards, and jobs in Databricks.
• Knowledge of CRISP-DM data modeling and pharma-domain awareness is a strong plus.

Responsibilities

• Be an Individual Contributor in analytics and development teams and solve real-world problems with cutting-edge capabilities.
• Join large delivery teams on advanced projects and provide expert assistance as needed.
• Deliver advanced Data Science capabilities through PoC solutions and transition them to production.
• Create LLD/STM documents and develop data integration processes on AWS/Azure platforms using Python/PySpark with Glue/Databricks.
• Manage ingestion, processing, labeling, and curation of data; handle ETL and data lake transformations.
• Collaborate with onshore/offshore lead designers, review LLD/STM, translate business rules to data transformations.
• Handle data connectors for APIs, Kafka, and batch processing; manage scoping of Databricks and AWS components.

Skills

Python

PySpark

Scala

SQL

Spark

Databricks

AWS

ETL

Data Modeling

ML Concepts

Leadership

R

Java

Education

BE/B.Tech

MCA

Tools

AWS Glue

Databricks

EC2

S3

IAM

Athena

VPC

DBFS

Job description

Position Summary

We are seeking a hands‑on Project Lead with 4-6 years of experience in designing, developing, and deploying data integration pipelines using Python/PySpark, AWS Glue, and Databricks, with strong proficiency across AWS services including EC2, S3, IAM, Athena, and VPC. The ideal candidate brings deep expertise in Spark‑based distributed computing, Databricks components (Databricks certified woudl be preferred) and the ability to code across R/Python/Scala/SQL while performing performance tuning and executor management. This role requires someone who can lead by example — independently driving technical delivery while mentoring and supervising 1-2 team members, with awareness of ML algorithms, data modelling frameworks (CRISP‑DM), and a pharma domain background being a strong added advantage.

Job Responsibilities

• Be an Individual Contributor in the Analytics and Development team and solve real-world problems using cutting‑edge capabilities and emerging technologies
• Be a part of large delivery teams working on advanced projects when expert assistance is required.
• Deliver advanced Data Science capabilities to businesses in a meaningful manner through successful proof‑of‑concept solutions, and later smoothly transition the proof‑of‑concept into production.
• Create LLD/STM documents, develop, test, and deploy data integration processes using Python/PySpark on AWS/Azure platform preferably using AWS Glue /Databricks - Collaborate with onshore/offshore lead designers, analyze LLD/STM, translate & apply business rules to data transformations
• Expertise Data Management principles. Experience in setting up ingestion, data processing- versioning, labelling, curating.
• Building ETL / data warehouse transformation processes and implementing Data lake project. Solutioning with large data sets, data processing at scale involving structured, semi‑structured, and unstructured data formats.
• Data connectors for APIs, Kafka, batch processing.
• Managing Descopes involving Databricks, and various AWS components

Education

• BE/B.Tech
• Master of Computer Application

Work Experience

• Minimum of 4-6 years develop, test, and deploy data integration processes using Python/PySpark on AWS platform preferably using AWS Glue Databricks.
• Deep understanding of architecture and work experience on AWS Databricks, hands‑on programming with Data Frames, Python, Scala.
• Good experience of Databrick components like Notebook, workspace, MLFlow and security management, creating dashboards, importing packages, DBFS, ML Flow, widgets and jobs in Databricks
• Familiar with cluster management, capacity & cost control in Databricks. Familiar with user & group access management and control
• Is aware of the overall architecture and UI
• Good experience of the notebook access management,
• Can code in R/Python/Scala/SQL
• 2-3 years of experience on Spark with Scala/Python/Java. Good experience of the distributed computing paradigm and various concepts in Spark
• Can perform performance tuning, executor management etc.
• Thorough knowledge of EC2, IAM, S3, VPC, Athena and Glue
• Is aware of the security and process design in AWS environment
• Is aware of the different phases involved in data modelling (CRISP-DM Model)
• Is aware of relevant modelling exercise to be performed (Forecasting, Clustering, Association, Classification etc.)

Behavioural Competencies

• Ownership
• Teamwork & Leadership
• Cultural Fit
• Motivation to Learn and Grow
• Databricks
• Decision Making
• Delivery Management- BIM/ Cloud Info Management

Technical Competencies

• Problem Solving
• Lifescience Knowledge
• Communication
• Capability Building / Thought Leadership
• Amazon Redshift
• SQL
• Python