Specialist - Data Engineering

Cvali; Bengaluru; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Cvali in Bengaluru is seeking a Data Engineer to strengthen the data platform that supports one of the world’s leading experimentation tools. You will own and operate data pipelines, build scalable data lakes in the cloud, and collaborate with data scientists and engineers across time zones to deliver reliable metrics for experimentation at scale.

You will design end-to-end data models, ensure data quality, and build monitoring, testing, and debugging capabilities to continuously improve

Qualifications

• Minimum 5 years of hands-on experience in data engineering or as a software engineer developing data pipelines and products.
• Bachelor's degree in Computer Science, Computer or Electrical Engineering, Mathematics, or equivalent (5 years experience).
• Solid experience in at least one programming language (Java and Python).
• Experience building production data pipelines in the cloud, setting up data lakes and serverless solutions.
• Hands-on experience with schema design and data modeling.
• Experience designing end-to-end systems; knowledge of basics like LB, DB caching, NoSQL, etc.
• Knowledge of Flink, CDC, Kafka, Airflow, Spark, Snowflake, DBT or equivalent tools.
• Practical experience building data platform capabilities like testing, monitoring, debugging, security.
• Experience working with Big Data and teams across time zones.
• Experience with experimentation statistics and AB testing is a plus.

Responsibilities

• Take ownership of multiple data pipelines and products and provide innovative solutions to reduce operational workload.
• Rapidly develop next-generation scalable, flexible, high-performance data pipelines.
• Contribute to data platform capabilities such as testing, monitoring, debugging, and ing to improve developer experience.
• Solve data and pipeline issues prioritizing by customer impact.
• End-to-end ownership of data quality in complex datasets and pipelines.
• Experiment with new tools and technologies to meet performance, scaling, and data quality requirements.
• Provide self-organizing tools that help analytics discover data, assess quality, explore usage, and find peers with expertise.
• Serve as main technical and business stakeholder contact for data engineering issues.

Skills

Data Modeling

Snowflake

Testing

Spark

Kafka

Debugging

Monitoring

Schema Design

NoSQL

Security

CDC

dbt

Big Data

Java

Airflow

Cloud

Python

Education

Bachelors degree in CS/CE/EE/Math or related field

Tools

Flink

Airflow

Snowflake

DBT

Kafka

Spark

Job description

Role Description Location- Bangalore

We are looking for a Data Engineer to join our team and help us to improve the platform that supports one of the best experimentation tools in the world

The selected candidate will work side by side with other data engineers and site reliability engineers to improve the reliability scalability maintenance and operations of all the data products that are part of the experimentation tool

Day to day work includes but is not limited to maintenance and operations of data pipelines and products that handles data at big scale the development of capabilities for monitoring ing testing and troubleshooting of the data ecosystem of the experiment platform and the delivery of data products that produce metrics for experimentation at scale The selected candidate will collaborate with colleagues in Amsterdam to achieve results the right way This will include engineering managers product managers engineers and data scientists

Key Responsibilities and Duties

• Take ownership of multiple data pipelines and products and provide innovative solutions to reduce the operational workload required to maintain them
• Rapidly developing nextgeneration scalable flexible and highperformance data pipelines
• Contribute to the development of data platform capabilities such as testing monitoring debugging and ing to improve the development environment of data products
• Solve issues with data and data pipelines prioritizing based on customer impact
• Endtoend ownership of data quality in complex datasets and data pipelines
• Experiment with new tools and technologies driving innovative engineering solutions to meet business requirements regarding performance scaling and data quality
• Provide selforganizing tools that help the analytics community discover data assess quality explore usage and find peers with relevant expertise
• Serve as the main point of contact for technical and business stakeholders regarding data engineering issues such as pipeline failures and data quality concerns

The job requirements

• Minimum 5 years of handson experience in data engineering as a Data Engineer or as a Software Engineer developing data pipelines and products
• Bachelors degree in Computer Science Computer or Electrical Engineering Mathematics or a related field or 5 years of progressively responsible experience in the specialty as equivalent
• Solid experience in at least one programming language We use Java and Python
• Experience building production data pipelines in the cloud setting up datalakes and serverless solutions
• Handson experience with schema design and data modeling
• Experience designing systems E2E and knowledge of basic concepts lb db caching NoSQL etc
• Knowledge of Flink CDC Kafka Airflow Spark Snowflake DBT or equivalent tools
• Practical experience building data platform capabilities like testing ing monitoring debugging security
• Experience Working With Big Data
• Experience working with teams located in different timezones is a plus
• Experience with experimentation statistics and AB testing is a plus
• Skills: Data Modeling, snowflake , Testing, Spark, Kafka, Debugging, Monitoring, Schema Design, Nosql, Security, cdc, dbt, Big Data, Java, Airflow, Cloud, Python
• Experience: 5.00-7.00 Years