Data Engineer

Toptal; Bengaluru; Hybrid; INR 1,500,000 - 2,700,000; Full time

Job summary

Toptal is seeking a Data Engineer to design and deliver scalable data pipelines and data warehouses in a hybrid setup based in Bangalore. You will collaborate across actuaries, developers and business experts to build robust data solutions that power web applications and analytics.

Ideal candidates have 5+ years in data integration, strong Spark/Databricks experience, SQL tuning and data modelling, plus hands-on BI tool knowledge with Power BI or Tableau.

Qualifications

• 5+ years of total experience working with data integration teams.
• 1+ years of in-depth experience developing data pipelines in an Apache Spark environment (Databricks preferred).
• 1+ years of active hands-on work with Databricks.
• Experience building data pipelines ingesting diverse data sources into a database or data lake and populating a structured warehouse or data mart.
• Fluent in complex SQL and database performance tuning.
• Knowledge of data modelling and data warehousing.
• Experience building data warehouses/data marts.
• Understanding of performance and maintainability for data-centric projects.
• Quick learner with ability to acquire new technologies on the spot.
• Experience with BI tools like Power BI or Tableau.

Responsibilities

• Collaborate with teams of actuaries, developers and business experts to deliver projects.
• Design and propose end-to-end data pipelines for data projects as well as databases to support web-based applications.
• Implementation of data warehouses and data marts to serve data consumers.
• Execute implementation of the designs from design stages to implementation and operationalization.
• Database design and modelling to organize data at macro and micro levels and provide logical data models.
• Database performance tuning and data lifecycle management.
• Assist in support and enhancements of existing data pipelines and databases.

Skills

Apache Spark

Databricks

SQL performance tuning

Data modelling

Data warehousing

ETL pipelines

BI tools

Tools

Power BI

Tableau

Job description

Join Toptal’s Elite Network of Freelance Talent

Toptal is an exclusive network of the world’s top freelance software developers, designers, finance experts, product managers, and project managers. As one of the fastest-growing fully remote networks globally, we empower professionals to thrive in their freelance careers while working with leading companies around the world.

Role: Data Engineer

Responsibilities:

• Collaborate with teams of experienced actuaries, developers and business experts to deliver projects
• Design and propose end-to-end data pipelines (ingestion to storage to consumption) for data projects as well as databases to support web-based applications
• Implementation of data warehouses and data marts to serve data consumers
• Execute implementation of the designs and bring it from design stages to implementation ,followed by operationalization to maintenance
• Database design and modelling. To be able to organize data at both macro and micro level and provide logical data models for the consumers
• Database performance tuning and data lifecycle management
• Assist in support and enhancements of existing data pipelines and databases

Must-Have:

• 5+ years of total experience working with data integration teams.
• 1+ years of in-depth experience developing data pipelines within an Apache Spark environment (preferably Databricks).
• 1+ years of active hands-on work with Databricks, demonstrating in-depth experience.
• Good experience in building data pipelines that ingests various types of data sources into a database or data lake and populating a structured warehouse or data mart that fuels the data consumer.
• Fluent in both complex SQL query performance tuning and database performance tuning
• Good knowledge of data modelling, database technologies, data warehousing.
• Have experience in building data warehouses/data marts
• Understand the importance of performance and able to implement best practices in ensuring performance and maintainability for data-centric projects
• This person is required to be dynamic and be a quick eager learner as knowledge about new technologies will need to be acquired on the spot
• Experience with BI and analytics tools like Power BI or Tableau.

Nice-to-Have:

• Experience integrating AI into data engineering.
• Certifications in Databricks, cloud computing or database management.
• Experience developing data solutions using native IaaS and PaaS solutions on AWS (RDS, S3).
• Experience in any object-oriented programming language.

Additional Details

• Location: Hybrid in Bangalore, India (2-3 days/week in-office)
• Overlap: Singapore timezone
• Type: 40h/week contract
• Duration: 12+ months
• Authorization: Applicants must be authorized to work in their country of residence without employer sponsorship