Data Engineer

CoreTek Labs; Bengaluru; On-site; INR 4,500,000 - 6,500,000; Full time

Job summary

CoreTek Labs seeks a Lead Data Engineer to drive enterprise EDW modernization toward Google BigQuery and GCP. The role requires 9+ years in Data Engineering, with 4+ years in a lead role, and hands-on expertise in Spark, Python, and BigQuery.

You will mentor teams, establish data standards, and lead AI-augmented data workflows. Ideal candidates have Oracle/Exadata background, experience with CDC and streaming architectures, and a strong CI/CD mindset.

Qualifications

• 9+ years in Data Engineering/Enterprise Data Warehousing
• 4+ years in a lead/principal capacity
• Deep expertise in BigQuery — architecture, partitioning, clustering, cost governance & security
• Strong Oracle & SQL background — Exadata, PL/SQL, data modeling
• Advanced hands-on with Spark (PySpark/Scala) and Python
• Experience with Oracle GoldenGate, Kafka, or CDC streaming architectures
• Proven track record automating data workflows and CI/CD pipelines
• Hands-on use of AI/LLMs in development — code/SQL generation, testing, and refactoring

Responsibilities

• Lead migration and modernization of Oracle EDW/Exadata workloads to BigQuery
• Design and build batch and real-time ETL/ELT pipelines with Spark (PySpark/Scala) and Python
• Implement CDC and replication from Oracle to BigQuery
• Establish DataOps automation — CI/CD, data quality checks, IaC (Terraform)
• Champion AI/LLM-driven development for code generation, testing, and debugging
• Set engineering standards, lead code/architecture reviews, mentor teams on data engineering and AI workflows

Skills

BigQuery

Spark

Python

Airflow

CI/CD

Terraform

Oracle SQL

Data modeling

DataOps

AI/LLMs

Tools

GitHub Actions

Jenkins

Terraform

Oracle GoldenGate

Kafka

Cloud Composer

Airflow

Job description

We're hiring a Lead Data Engineer to drive our enterprise Data Warehouse modernization to Google BigQuery & GCP!

Experience: 9+ years in Data Engineering/EDW, with 4+ years in a Lead/Principal role

Short joiners preferred

What you'll do:

• Lead migration & modernization of legacy Oracle EDW/Exadata workloads to BigQuery and modern Lakehouse architectures
• Design, build, and optimize large-scale batch & real-time ETL/ELT pipelines using Apache Spark (PySpark/Scala), Python, and Cloud Composer (Airflow)
• Implement CDC and continuous replication from Oracle to BigQuery using Oracle GoldenGate, Kafka, and CDC frameworks
• Establish end-to-end DataOps automation — CI/CD, data quality/reconciliation checks, and Infrastructure-as-Code (Terraform)
• Champion AI/LLM-driven development — using tools like GitHub Copilot, Claude, ChatGPT, and Gemini for code generation, SQL refactoring, testing, and debugging
• Set engineering standards, lead code/architecture reviews, and mentor teams on data engineering and AI-augmented workflows

What we’re looking for:

• 9+ years in Data Engineering/Enterprise Data Warehousing, incl. 4+ years in a lead/principal capacity
• Deep expertise in BigQuery — architecture, partitioning, clustering, slot optimization, cost governance & security (policy tags, row/column access policies)
• Strong Oracle & SQL background — Exadata, advanced PL/SQL, complex data modeling (Star/Snowflake, Data Vault, SCDs), query tuning
• Advanced hands-on experience with Apache Spark (PySpark/Scala) and Python for distributed data processing
• Strong experience with Oracle GoldenGate, Kafka, or CDC streaming architectures
• Proven track record automating data workflows and CI/CD pipelines (GitHub Actions, Jenkins, Terraform)
• Demonstrated hands-on use of AI/LLMs in daily development — prompt engineering for code/SQL generation, automated testing, and code modernization