Senior Data Engineer with Python (IR-491)

Intellectsoft; Bengaluru; On-site; INR 800,000 - 1,200,000; Full time

Benefits offered by this job

Awesome projects with an impact

Udemy courses of your choice

Team-building events

Work-life balance days

Flexible hours and work set-up

Job summary

A leading software development firm in Bengaluru is seeking an experienced Data Engineer to design and build scalable data pipelines using PySpark and big data technologies. The role involves collaborating with the data science team to enhance accuracy and performance of models, whilst managing ETL pipelines. Candidates should have over 4 years of experience, proficiency in Apache Spark and SQL, and a bachelor's degree in Computer Science. This position offers flexibility to work from anywhere and engage in career development activities.

Qualifications

• 4+ years of professional experience in data engineering.
• 2+ years of experience with Apache Spark and SQL.
• Strong knowledge of database systems and data modelling.

Responsibilities

• Design and build reliable and scalable data pipelines using PySpark.
• Collaborate with data science team on feature development.
• Troubleshoot and optimize existing ETL pipelines.

Skills

Data engineering with Apache Spark

Python for data processing

ETL pipeline management

Proficiency in SQL

Knowledge of PySpark

Experience with Apache Airflow

Familiarity with cloud platforms

Education

Bachelor’s or master’s degree in Computer Science

Job description

Intellectsoft is a software development company delivering innovative solutions since 2007. We operate across North America, Latin America, the Nordic region, the UK, and Europe. We specialise in industries like Fintech, Healthcare, EdTech, Construction, Hospitality, and more, partnering with startups, mid‑sized businesses, and Fortune 500 companies to drive innovation and scalability. Our clients include Jaguar Motors, Universal Pictures, Harley-Davidson, and many more where our teams are making daily impactTogether, our team delivers solutions that make a difference. Learn more at www.intellectsoft.net

Our customer’s product is an AI-powered platform that helps businesses make better decisions and work more efficiently. It uses advanced analytics and machine learning to analyse large amounts of data and provide useful insights and predictions. The platform is widely used in various industries, including healthcare, to optimise processes, improve customer experiences, and support innovation. It integrates easily with existing systems, making it easier for teams to make quick, data-driven decisions to deliver cutting‑edge solutions.

• 4+ years of professional experience, including 2+ years of data engineering with Apache Spark and SQL.
• Proficiency in Python for data processing and automation.
• Knowledge of PySpark, distributed computing, analytical databases and other big data
• technologies.
• Expertise in designing and managing ETL pipelines and distributed data processing frameworks.
• Strong knowledge of database systems, data modelling, and analytical databases.
• Hands‑on experience with workflow orchestration tools such as Apache Airflow.
• Familiarity with cloud platforms like AWS, GCP, or Azure.
• Solid understanding of software development lifecycles, including coding standards, version control, and testing.

Nice to have skills

• Bachelor’s or master’s degree in Computer Science or a related field.
• Familiarity with the data science and machine learning development process.
• Understanding of Machine Learning pipelines or frameworks.

Responsibilities

• Design and build highly reliable and scalable data pipelines using PySpark and big data technologies.
• Collaborate with the data science team to develop new features that enhance model accuracy and performance.
• Create standardised data models to improve consistency across various deployments.
• Troubleshoot and resolve issues in existing ETL pipelines and optimise workflows.
• Conduct POCs to evaluate new technologies and integrate additional data sources.
• Follow and promote best practices for software development, ensuring high-quality solutions that meet requirements and deadlines.
• Document development updates and maintain clear technical documentation.

• Awesome projects with an impact
• Udemy courses of your choice
• Team‑buildings, events, marathons & charity activities to connect and recharge
• Workshops, training, expert knowledge-sharing that keep you growing
• Clear career path
• Absence days for work‑life balance
• Flexible hours & work set‑up - work from anywhere and organise your day your way