Interesting Job Opportunity: Infometry - Senior Data Engineer - SQL/Python

Infometry Inc; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Infometry Inc in Bengaluru is seeking an experienced Senior Data Engineer with deep expertise in Databricks, Apache Airflow, SQL, and Python to design and optimize data pipelines, ETL workflows, and cloud-based data solutions. The ideal candidate will have deep experience in big data processing, cloud-based analytics, and automation to drive efficient and scalable data engineering solutions.

You will implement PySpark transformations, build robust data pipelines, and collaborate with

Qualifications

• 8-9 years of experience in Data Engineering or related fields.
• Strong expertise in Databricks (PySpark, Delta Lake, DBSQL).
• Proficiency in Apache Airflow for scheduling and orchestrating workflows.
• Advanced SQL skills for data extraction, transformation, and performance tuning.
• Strong programming skills in Python (pandas, NumPy, PySpark, APIs).
• Experience with big data technologies and distributed computing.
• Hands-on experience with cloud platforms (AWS / Azure / GCP).
• Expertise in data warehousing and data modeling concepts.
• Understanding of CI/CD pipelines, version control (Git)
• Experience with data governance, security, and compliance best practices.
• Excellent troubleshooting, debugging, and performance optimization skills.

Responsibilities

• Design, develop, and optimize data pipelines using Databricks and Apache Airflow.
• Implement PySpark-based transformations and processing in Databricks for handling large-scale data.
• Develop and maintain SQL-based data pipelines, ensuring performance tuning and optimization.
• Create Python scripts for automation, data transformation, and API-based data ingestion.
• Work with Airflow DAGs to schedule and orchestrate data workflows efficiently.
• Optimize data lake and data warehouse performance for scalability and reliability.
• Integrate data pipelines with cloud platforms (AWS, Azure, or GCP) and various data storage solutions.
• Ensure adherence to data security, governance, and compliance standards.

Skills

Databricks

PySpark

Delta Lake

DBSQL

Apache Airflow

SQL

Python

Git

CI/CD

Cloud Platforms (AWS/Azure/GCP)

Data Warehousing

Data Modeling

Job description

Job Summary

We are seeking an experienced Senior Data Engineer with strong expertise in Databricks, Apache Airflow, SQL, and Python to design and optimize data pipelines, ETL workflows, and cloud-based data solutions. The ideal candidate will have deep experience in big data processing, cloud-based analytics, and automation to drive efficient and scalable data engineering solutions.

Job Summary

We are seeking an experienced Senior Data Engineer with strong expertise in Databricks, Apache Airflow, SQL, and Python to design and optimize data pipelines, ETL workflows, and cloud-based data solutions. The ideal candidate will have deep experience in big data processing, cloud-based analytics, and automation to drive efficient and scalable data engineering solutions.

Key Responsibilities

• Design, develop, and optimize data pipelines using Databricks and Apache Airflow.
• Implement PySpark-based transformations and processing in Databricks for handling large-scale data.
• Develop and maintain SQL-based data pipelines, ensuring performance tuning and optimization.
• Create Python scripts for automation, data transformation, and API-based data ingestion.
• Work with Airflow DAGs to schedule and orchestrate data workflows efficiently.
• Optimize data lake and data warehouse performance for scalability and reliability.
• Integrate data pipelines with cloud platforms (AWS, Azure, or GCP) and various data storage solutions.
• Ensure adherence to data security, governance, and compliance standards.

Required Skills & Qualifications

• 8-9 years of experience in Data Engineering or related fields.
• Strong expertise in Databricks (PySpark, Delta Lake, DBSQL).
• Proficiency in Apache Airflow for scheduling and orchestrating workflows.
• Advanced SQL skills for data extraction, transformation, and performance tuning.
• Strong programming skills in Python (pandas, NumPy, PySpark, APIs).
• Experience with big data technologies and distributed computing.
• Hands-on experience with cloud platforms (AWS / Azure / GCP).
• Expertise in data warehousing and data modeling concepts.
• Understanding of CI/CD pipelines, version control (Git)
• Experience with data governance, security, and compliance best practices.
• Excellent troubleshooting, debugging, and performance optimization skills.

(ref:hirist.tech)