Senior Data Engineer

PradeepIT Consulting Services Pvt Ltd; Bengaluru; On-site; INR 800,000 - 1,200,000; Full time

Job summary

A consulting firm in Bengaluru is seeking a skilled automation specialist with proficiency in Python and SQL. The ideal candidate will manage ETL processes, provide operational support, and utilize monitoring tools like Prometheus and Grafana. Familiarity with cloud platforms (AWS, GCP, Azure) and experience in Kubernetes are preferred. The position is based in Bengaluru, offering opportunities for impactful work in a collaborative environment.

Qualifications

• Proficiency in Python for automating processes.
• Strong SQL skills for querying data and monitoring.
• Experience with ETL tools and managing data pipelines.
• Knowledge of cloud platforms like AWS, GCP, or Azure.

Responsibilities

• Automate tasks and support services using Python.
• Manage and optimize ETL processes.
• Provide operational support and troubleshoot incidents.
• Set up monitoring dashboards and perform anomaly detection.

Skills

Python automation

SQL

ETL tools

Operational support

Monitoring tools

DevOps principles

Tools

Informatica Cloud

Prometheus

Grafana

AWS

GCP

Azure

OpenShift

Kubernetes

Apache Kafka

Job description

Responsibilities

• Proficiency in Python for automating processes and supporting services.
• Shell scripting for task automation and system-level troubleshooting.
• Experience with ETL tools like Informatica Cloud; ability to manage and optimize data extraction, transformation, and loading (ETL) processes.
• Strong SQL skills for querying data, creating custom checks, and monitoring services; experience managing and monitoring data pipelines in upstream, batch, and streaming environments.
• Familiarity with pipeline orchestration tools like Luigi or Airflow.
• Experience providing operational support, including troubleshooting and resolving incidents; knowledge of cloud platforms like AWS, GCP, or Azure.
• Experience with OpenShift and Kubernetes environments.
• Proficiency in monitoring and automation tools such as Prometheus, Grafana, or similar tools for monitoring service health; ability to set up monitoring dashboards and perform anomaly detection.
• Knowledge of user access management, including account setup and permissions validation; ability to understand and document data flows and dependencies between different data services.
• Familiarity with DevOps principles and Agile environments.

Nice-to-Have Skills

• Collaboration and Communication: Proficiency in using Google Docs and Jira for documentation and incident tracking.
• Strong communication skills for working with cross-functional teams.
• Incident Management and Optimization: Experience in incident management and root cause analysis; ability to identify and fix common failure points.
• Knowledge of Apache Kafka and experience with streaming services.
• Awareness of data governance principles for maintaining data quality and compliance.