Senior Data Engineer

Bidgely, Inc

India

On-site

INR 1,200,000 - 2,400,000

Full time

11 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Looker
Tableau

Job summary

Bidgely, Inc is seeking a seniorData Engineer to own central data platform features on Databricks and Snowflake. You will design, build, and run scalable batch and streaming pipelines, and contribute to data governance, cost control, and data quality.

Collaborating with product managers, BI engineers and data scientists, you will translate requirements into robust analytics solutions, applying Spark, Delta Lake, and AI-enabled data products in production.

Qualifications

  • 5+ years of data engineering experience building central data platforms.
  • Hands-on Databricks and Snowflake proficiency required.
  • Strong Spark using Python/Scala on Databricks; SQL on Databricks/Snowflake.
  • Experience with Delta Lake and data governance concepts.

Responsibilities

  • Build and run central data platform features on Databricks and Snowflake.
  • Design and maintain scalable batch and streaming ELT/ETL pipelines.
  • Create AI agents and data products using LangGraph, Databricks, Snowflake.
  • Implement data quality checks, lineage, RBAC, and governance.
  • Monitor costs and optimize compute/storage in cloud platforms.
  • Collaborate with product managers, BI, and data scientists.
  • Support Kimball/3NF dimensional modelling for AWB analytics.
  • Contribute to CI/CD for data workflows with version control and tests.
  • Review code and promote best practices; produce production-ready components.

Skills

Databricks
Snowflake
Apache Spark
Python/Scala
SQL
Delta Lake
Kafka
Airflow
LangGraph
LangChain
RBAC
CI/CD
DevOps
Data governance
AWS
Azure
GCP
Kimball modelling
3NF normalization
Looker
Tableau

Tools

Airflow
dbt
Looker
Tableau

Job description

ROLES & RESPONSIBILITIES
  • Play a key hands‑on role in building and running central data platform features on Databricks and Snowflake, supporting the architects and managers who own the design and delivery.
  • Design, build, and maintain scalable batch and streaming ELT/ETL data pipelines and large‑scale datasets using Apache Spark with Python or Scala, Delta, Kafka and Airflow.
  • Build AI agents and agentic data products on Databricks (for example Genie), Snowflake (for example Cortex) and LangGraph, and expose governed data to agents through MCP tools built with FastMCP.
  • Implement automated data quality checks, end‑to‑end lineage tracking and RBAC security policies for the datasets you build, following the team’s data governance framework.
  • Monitor and optimize Databricks and Snowflake compute and storage costs, and work within the team’s budget and spend‑alert policies.
  • Work closely with product managers, BI engineers, and data scientists to translate business requirements into scalable, well‑modelled analytical solutions.
  • Support data modelling initiatives using dimensional (Kimball) and normalized (3NF) techniques to serve AWB reporting and analytics needs.
  • Contribute to CI/CD pipeline development for data workflows, including version control, automated testing, and deployment standards.
  • Review code and share best practices on Databricks, Snowflake and data quality with the team. This is an individual contributor role with no direct reports.
  • Translate architectural designs into high‑quality, production‑ready code by applying software design principles, design patterns, SOLID principles, and clean code practices. Build modular, reusable, loosely coupled, and testable components that support long‑term scalability and maintainability.
WHAT YOU’LL DELIVER IN YOUR FIRST 12 MONTHS
  • Deliver core data platform features end to end in production on Databricks and Snowflake, working with the architects and managers who own them.
  • Deliver agent‑ready datasets and tools that LangGraph agents and MCP tools use in production.
  • Have data quality checks, lineage and RBAC in place for every dataset you build.
  • Bring the compute and storage cost of the pipelines you build within agreed budgets.
WHAT YOU’LL BRING
  • 5+ years of data engineering experience building central data platforms, large‑scale pipelines, and production datasets; strong, hands‑on proficiency in both Databricks and Snowflake is strictly required.
  • Strong proficiency in Apache Spark using Python or Scala on Databricks, and in SQL on both Databricks and Snowflake, for core data processing workloads. Strong coding skills are required. This is a hands‑on engineering role.
  • Hands‑on experience with Delta Lake as the primary table format, including MERGE, schema evolution, OPTIMIZE, VACUUM and performance tuning.
  • Hands‑on experience building streaming pipelines with Kafka and Spark Structured Streaming, including duplicates, late data, checkpoints and delivery guarantees.
  • Production experience orchestrating batch and streaming workflows with Apache Airflow (mandatory), including retries, backfills, dependencies and alerting.
  • Hands‑on experience building AI agents with LangGraph (or LangChain) and with the agent features of Databricks and Snowflake, such as Genie and Cortex. Uses AI code‑generation tools in day‑to‑day engineering work.
  • Solid understanding of data governance concepts including data lineage, quality monitoring, regulatory compliance, RBAC, and governance frameworks.
  • Experience building production‑grade data pipelines on cloud platforms (AWS, Azure, or GCP) with a focus on reliability and observability.
  • Good understanding of data modelling concepts including Kimball dimensional modelling, star/snowflake schemas, and 3NF normalization.
  • Familiarity with CI/CD pipelines, Git‑based workflows, and applying DevOps practices to data engineering projects.
  • Strong communication and stakeholder management skills with an ownership mindset and ability to work independently across time zones.
BONUS SKILLS / NICE TO HAVE
  • Exposure to Databricks and Snowflake Native Applications, data clean rooms, or Marketplace integrations.
  • Experience implementing, testing and documenting dbt models for data reliability, lineage tracking and governance.
  • Experience with other orchestration tools such as Prefect or dbt Cloud.
  • Experience supporting BI/Analytics teams through semantic layer development (e.g., Looker, Tableau, or similar tools).
  • Experience working in Agile/product‑based development environments with sprint planning, backlog management, and iterative delivery.
  • Databricks and Snowflake cost optimization experience: compute usage analysis, query profiling, clustering and partitioning design, materialized views, search optimization, resource monitors, budgets and spend alerts.
  • Background in the energy, utilities, or IoT analytics domain is a strong differentiator.
  • Exposure to MLOps concepts and production deployment workflows for data science models integrated with Databricks and Snowflake.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

Bidgely, Inc • Bengaluru

On-site
INR 1,400,000 - 2,800,000
Senior Data Engineer
Senior Data Engineer

Bidgely • India

On-site
INR 4,500,000 - 7,500,000
Senior Data Engineer
Senior Data Engineer

Bidgely • Bengaluru

On-site
INR 900,000 - 1,400,000
Senior Data Engineer
Senior Data Engineer

HashedIn by Deloitte • Bengaluru

On-site
INR 4,000,000 - 7,000,000
Technical Lead
Technical Lead

Carelon Global Solutions • Gurugram District, Bengaluru

Hybrid
INR 3,000,000 - 5,500,000
Senior Manager
Senior Manager

ExlService Holdings, Inc. • Bengaluru

Hybrid
INR 2,400,000 - 3,600,000
Senior Data Engineer
Senior Data Engineer

Amgen Technology • Hyderabad

Hybrid
INR 2,500,000 - 4,000,000
Data Engineer, Data Platforms
Data Engineer, Data Platforms

S&P Global Market Intelligence • Hyderabad, Chennai

On-site
INR 2,500,000 - 4,500,000
Senior Data Engineer
Senior Data Engineer

CoffeeBeans • Bengaluru

On-site
INR 1,800,000 - 2,400,000
Senior Data Engineer
Senior Data Engineer

Wissen Technology Pte Ltd • Bengaluru

Hybrid
INR 3,000,000 - 5,400,000