Python Developer

Bct Consulting; Chennai District; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Bct Consulting in Chennai, India, is seeking a Data Engineer to design scalable ETL/ELT pipelines using Python and optimize Oracle Database performance. The role includes modeling data for OLTP and OLAP, and delivering robust data solutions.

You will implement data integration using Airflow, Kafka, and PySpark, ensure data quality and governance, and monitor pipelines in a CI/CD setup with Git/Jenkins/GitHub Actions.

Qualifications

• Strong expertise in Python programming, data pipelines, and data engineering concepts.

Responsibilities

• Design and develop scalable ETL/ELT data pipelines using Python.
• Work with Oracle Database for SQL/PLSQL development and performance tuning.
• Build and optimize data models for OLTP and OLAP systems.
• Develop data integration solutions using Airflow, Kafka, and PySpark.
• Ensure data quality, validation, and governance standards.
• Monitor and troubleshoot data pipelines and database performance.
• Work in CI/CD environment using Git, Jenkins, and GitHub Actions.

Skills

Python

Shell scripting

Git

CI/CD

Testing

Tools

Oracle Database

SQL/PLSQL

Airflow

Kafka

PySpark

Pandas

PyArrow

SQLAlchemy

cx_Oracle

oracledb

Polars

DuckDB

Docker

Kubernetes

GitHub Actions

Terraform

OCI

Job description

Job Title: Data Engineer (Python / Oracle)

We are looking for a Data Engineer with strong expertise in Python, Oracle Database, and data pipeline development.

Key Responsibilities:

• Design and develop scalable ETL/ELT data pipelines using Python
• Work with Oracle Database for SQL/PLSQL development and performance tuning
• Build and optimize data models for OLTP and OLAP systems
• Develop data integration solutions using tools like Airflow, Kafka, and PySpark
• Ensure data quality, validation, and governance standards
• Monitor and troubleshoot data pipelines and database performance
• Work in CI/CD environment using Git, Jenkins, and GitHub Actions

Required Skills:

• Programming & Scripting (Primary)
  • Advanced Python (3.x) OOP, typing, async, performance profiling
  • Familiarity with Python dataengineer libraries: pandas, pyarrow, sqlalchemy, cx_Oracle, oracledb,polars, duckdb
  • Shell scripting (bash, PowerShell) for automation and orchestration
• Oracle Database Expertise (Primary)
  • Oracle Database (11g/12c/19c/21c) administration basics
  • SQL proficiency: complex queries, analytic functions, hierarchical queries, PL/SQL development
  • Data modeling (ER, dimensional) and schema design for OLTP & OLAP
  • Performance tuning: indexing, partitioning, optimizer hints, AWR/ASH analysis
  • Oracle Data Pump, SQL*Loader, External Tables
• Data Integration & ETL (Primary)
  • Design and implementation of ETL/ELT pipelines in Python (e.g., polars,pandas, pySpark, dbt)
  • Knowledge of messaging/streaming (Kafka, RabbitMQ) for realtime ingestion
  • Data orchestration platforms: Apache Airflow, Prefect, or Azure Data Factory
• Big Data & Distributed Processing (Secondary)
  • Working knowledge of Apache Spark (PySpark) and its integration with Oracle
  • Experience with cloudbased bigdata services (AWS EMR, Azure Synapse, GCP Dataproc)
• Cloud & DevOps (Secondary)
  • Oracle Cloud Infrastructure (OCI) services: Autonomous DB, Object Storage, Functions
  • Containerization (Docker) and orchestration (Kubernetes) for scalable pipelines
  • CI/CD pipelines (Git, Jenkins, GitHub Actions) for automated testing and deployment
  • InfrastructureasCode tools (Terraform, OCI Resource Manager)
• Data Quality & Governance (Primary)
  • Implementing data validation, profiling, and cleansing in Python
  • Familiarity with data lineage, metadata management, and catalog tools (Apache Atlas, Collibra)
  • Understanding of GDPR, CCPA, and other dataprivacy regulations
• Testing & Monitoring (Primary)
  • Unit & integration testing frameworks (pytest, unittest) for data pipelines
  • Monitoring & alerting (Prometheus, Grafana, OCI Monitoring) of ETL jobs and database health
• Version Control & Collaboration (Secondary)
  • Proficient with Git (branching, pullrequests, code reviews)
  • Agile methodologies (Scrum/Kanban) and ticketing systems (Jira, Azure Boards)