Data Engineer

Span; Bengaluru; On-site; INR 800,000 - 1,200,000; Full time

Benefits offered by this job

Generous paid leave

Comprehensive Insurance & Health Benefits

Centrally located office with easy access to public transit

Job summary

Span is seeking a Data Engineer in Bengaluru to design, build, and test infrastructure for real-time and batch analytics. Your role includes writing ETL processes, advocating for data quality, and collaborating within the organization.

The ideal candidate will have a Bachelor's degree in a quantitative field, 2-5 years of experience, and proficient programming skills in Python or Kotlin. Benefits include generous paid leave and comprehensive health insurance.

Qualifications

• 2 - 5 years of relevant work experience in data engineering, business intelligence, or related fields.
• Experience managing data engineering infrastructure through Docker and Kubernetes.
• Expert level production-grade programming experience in at least one of Python, Kotlin, or JVM based languages.

Responsibilities

• Design, build, and create infrastructure necessary for real-time and batch analytics.
• Work with multiple teams to provide analysis and insights on data.
• Write ETL processes supporting data ingestion and enforce data management best practices.

Skills

Python

Kotlin

Docker

Kubernetes

SQL

AWS

Education

Bachelor's Degree in a quantitative discipline

Tools

Pulumi

Terraform

Circle-CI

GitHub Actions

Argo-CD

Tableau

FiveTran

DataDog

Job description

The Role

As a Data Engineer you would be working to design, build, test and create infrastructure necessary for real time analytics and batch analytics pipelines. You will work with multiple teams within the org to provide analysis, insights on the data. You will also be involved in writing ETL processes that support data ingestion. You will also guide and enforce best practices for data management, governance and security. You will build infrastructure to monitor these data pipelines / ETL jobs / tasks and create tooling/infrastructure for providing visibility into these.

Responsibilities

• Acquire deep business understanding on how SPAN data flows from IoT device to cloud through the system and build scalable and optimized data solutions that impact many stakeholders.
• Be an advocate for data quality and excellence of our platform.
• Build tools that help streamline the management and operation of our data ecosystem.
• Ensure best practices and standards in our data ecosystem are shared across teams.
• Work with teams within the company to build close relationships with our partners to understand the value our platform can bring and how we can make it better.
• Improve data discovery by creating data exploration processes and promoting adoption of data sources across the company.
• Have a desire to write tools and applications to automate work rather than do everything by hand.
• Assist internal teams in building out data logging, alerting and monitoring for their applications.
• Be passionate about CI/CD processes.
• Design, develop and establish KPIs to monitor analysis and provide strategic insights to drive growth and performance.

Required Qualifications

• Bachelor's Degree in a quantitative discipline: computer science, statistics, operations research, informatics, engineering, applied mathematics, economics, etc.
• 2 - 5 years of relevant work experience in data engineering, business intelligence, research or related fields.
• Expert level production‑grade, programming experience in at least one of these languages (Python, Kotlin, or other JVM based languages).
• Experience in writing clean, concise and well structured code in one of the above languages.
• Experience working with Infrastructure‑as‑code tools: Pulumi, Terraform, etc.
• Experience working with CI/CD systems: Circle‑CI, GitHub Actions, Argo‑CD, etc.
• Experience managing data engineering infrastructure through Docker and Kubernetes.
• Experience working with latency data processing solutions like Flink, Prefect, AWS Kinesis, Kafka, Spark Stream processing etc.
• Experience with SQL/Relational databases, OLAP databases like Snowflake.
• Experience working in AWS: S3, Glue, Athena, MSK, EMR, ECR etc.

Bonus Qualifications

• Experience with the Energy industry.
• Experience with building IoT and/or hardware products.
• Understanding of electrical systems and residential loads.
• Experience with data visualization using Tableau.
• Experience in Data loading tools like FiveTran as well as data debugging tools such as DataDog.

Benefits at SPAN India

• Generous paid leave.
• Comprehensive Insurance & Health Benefits.
• Centrally located office in Bengaluru with easy access to public transit, dining, and city amenities.