Senior Data Engineer

SourcingXPress; Hyderabad; On-site; INR 1,500,000 - 1,700,000; Full time

Job summary

A leading data solutions firm in Hyderabad is looking for an experienced Data Engineer to build and enhance their enterprise data platform. The role involves developing data integration and analytics tools while collaborating with client teams. Ideal candidates will have at least 5 years’ experience using Python and Databricks, along with a recognized data engineering certification. The position requires strong technical skills and effective communication abilities to meet diverse client needs.

Qualifications

• At least 5 years’ experience in data engineering using Python.
• Extensive experience with Databricks and relational database technologies.
• At least 6 months experience in web application development is preferred.

Responsibilities

• Builds front-end and back-end infrastructure for the enterprise data platform.
• Enhances data integration, management, and analytics tools.
• Collaborates with internal teams to complete client projects.

Skills

Data Engineering using Python

Pandas or PySpark

Databricks

Relational Database Technologies

Agile Development

AI Tools

Docker

Education

Recognized data engineering certification

Tools

Databricks APIs

Azure Data Factory

Git

Job description

Overview

Company: Beghou

Website: Visit Website

Business Type: Enterprise

Company Type: Product & Service

Business Model: B2B

Funding Stage: Bootstrapped

Industry: Information Technology

Salary Range: ₹ 15-17 Lacs PA

Beghou brings over three decades of experience helping life sciences companies optimize their commercialization through strategic insight, advanced analytics, and technology. From developing go-to-market strategies and building foundational data analytics infrastructures to leveraging artificial intelligence to improve customer insights and engagement, Beghou helps life sciences companies maximize performance across their portfolios. Beghou also deploys proprietary and third-party technology solutions to help companies forecast performance, design territories, manage customer data, organize, and report on medical and commercial data, and more. Headquartered in Evanston, Illinois, we have 10 global offices.

Position Summary

This role builds front-end and back-end infrastructure for the firm’s in-house enterprise data platform as well as explores, enables, and documents new technologies. This position also works independently to support client teams with different data needs, including deployment for new clients, support for existing clients, development of backend datasets for dashboards, and support for the front-end.

Duties & Responsibilities

• Ensures reliability and stability of the enterprise data platform and tools used by internal teams and clients.
• Demonstrates knowledge of the data platform components (warehouse, monitoring, language, etc.) and data engineering (ETL design, schema design, etc.) and performs unit testing.
• Builds and enhances data integration, management, and analytics tools and pipelines.
• Documents code, deployments, best practices, and improvements.
• Designs and builds ETL/ELT data systems including cloud-based APIs, automated data transfer systems, and code management solutions.
• Acts as regular point of contact for data engineering projects and collaborates with the internal consulting team and the data platform and technology teams to successfully complete projects for clients.
• Participates in and contributes ideas to brainstorm, design, and implement more complex processes and analyses to solve challenging and abstract problems.
• Develops and implements best practice guidance and supporting materials for data management and advanced analytics pipelines.
• Creates high quality presentations, deliverables, and code that meets objectives, with project manager guidance.
• Performs other job-related duties as assigned.

Desired Technical Skills & Experience

• At least 5 years’ experience in data engineering using Python including use of pandas or PySpark.
• Experience working with Databricks extensively (including installing packages, understand and setting cluster configurations, managing jobs, user management, handling permissions, managing Unity Catalogs, Databricks APIs and AI tools, and handling configuration issues) and relational database technologies, such as PostgreSQL, Oracle, MySQL, Redshift, Snowflake.
• Software development fundamentals, including Agile development, version control systems such as Git or DevOps, code reviews, testing, and documentation.
• Experience working with AI tools.
• Experience configuring AzureAD/SAML/Okta/Oauth and administering AWS or Azure security best practices, preferred.
• Experience with WYSIWYG ETL tools (Azure Data Factory, Informatica, SnapLogic, Boomi), preferred.
• Container orchestration systems experience using Docker, Kubernetes, AWS ECS, preferred.
• At least 6 months experience in web application development using Flask, Django, JavaScript, Ajax, or CSS/HTML is preferred.
• Candidates holding a recognized data engineering certification are preferred (e.g., Databricks Data Engineer, Google Cloud Professional Data Engineer, Azure Data/Fabric Data Engineer, AWS Certified Data Engineer).
• Proficiency using Microsoft Office products, including Excel, PowerPoint, and Word.
• Life Sciences industry experience, preferred.

Desired Soft Skills

• Excellent communication skills. Ability to communicate with business stakeholders as well as technical development teams
• Strong problem-solving skills, excellent critical and creative thinking skills and ability to brainstorm ideas
• Ability to work in a team and collaborate across teams effectively
• Ability to adapt quickly to changing technologies
• Ability to assess and identify bottlenecks and potential failures