Senior Data Engineer - Databricks

Intetics; Town of Poland (NY); On-site; USD 120,000 - 160,000; Full time

Job summary

Intetics Inc. is seeking a data engineer to own Databricks production support across our data platform. You will monitor, alert, and respond to incidents while maintaining SLA performance for data pipelines in Azure and AWS environments.

The role emphasizes migrating legacy ETL/ELT to Databricks, optimizing costs, and building scalable, secure, multi-tenant data pipelines. Collaboration with data science, product, and infrastructure teams is essential near daily operations.

Qualifications

• 4+ years of data engineering experience.
• 2+ years with Databricks or Spark in Azure/AWS.
• Proficiency in PySpark, SQL, and Python on production pipelines.
• Hands-on experience with Delta Lake and ACID transactions.
• Experience with pipeline performance tuning and compute optimization.
• Knowledge of PostgreSQL in production data pipelines.
• Experience supporting multi-tenant architectures and tenant isolation.

Responsibilities

• Own Databricks production support across the data platform, including monitoring and incident response.
• Maintain and report SLA performance metrics for data pipeline delivery.
• Identify and implement pipeline optimizations to reduce compute costs and improve throughput.
• Migrate legacy ETL/ELT pipelines to Databricks and build automation tooling.
• Support onboarding by provisioning and hardening tenant data pipelines.
• Design and build high-performance Databricks pipelines ingesting ERP and CRM data at scale.
• Own Delta Lake architecture including schema design, partitioning, data quality enforcement, and incremental processing.
• Enforce data security best practices across Databricks environments (RBAC, secrets management).
• Implement data quality monitoring and observability across pipelines and ML inputs.
• Apply multi-tenant data isolation patterns and ensure reliable data delivery across customers.
• Collaborate with Enterprise Architecture to integrate pipelines with AI/analytics ecosystem.
• Participate in on-call rotation for global operations and incident response.
• Maintain technical documentation and runbooks for operational readiness.
• Apply CI/CD best practices to data pipeline development.

Skills

Databricks

Apache Spark

PySpark

SQL

Python

Delta Lake

PostgreSQL

Multi-tenant

Tools

SSIS

Informatica

Terraform

Azure

AWS

Job description

Intetics Inc. is a global technology company specializing in custom software development, AI-powered solutions, cloud technologies, and digital transformation. With over 30 years of experience, we help organizations worldwide build scalable, innovative, and data-driven solutions across a wide range of industries. We are looking for talented professionals who are passionate about solving complex technical challenges and building high-quality data platforms.

Impact You Will Make in the Role

• Own Databricks production support for the company's data platform, including monitoring, alerting, and incident response across all production data flows.
• Maintain and report on SLA performance metrics for data pipeline delivery, ensuring visibility into platform health and accountability across internal and external stakeholders.
• Identify and implement pipeline optimizations that reduce Databricks compute costs, improve throughput, and reduce processing windows while tracking impacts through measurable KPIs.
• Migrate legacy ETL/ELT pipelines to Databricks, building automation tooling to reduce manual intervention and ensure uninterrupted data delivery during transitions.
• Support new customer onboarding by provisioning, validating, and hardening tenant data pipelines that deliver reliable, isolated data from day one.
• Design and build high-performance Databricks pipelines that ingest, transform, and serve ERP and CRM data at scale across both Azure and AWS environments.
• Own the Delta Lake architecture, including schema design, partitioning strategies, data quality enforcement, and incremental processing patterns.
• Enforce data security best practices across Databricks environments, including role-based access control, secrets management, and compliance requirements for enterprise business data.
• Implement data quality monitoring and observability across pipeline health and ML model inputs, ensuring data integrity that directly supports predictive analytics.
• Apply and enforce multi-tenant data isolation patterns, ensuring reliable and secure data delivery across enterprise customers.
• Partner with the Enterprise Architecture team to ensure data pipelines integrate seamlessly with the broader AI and analytics ecosystem.
• Support a globally distributed operation through on-call rotation and after-hours incident response, meeting SLAs across multiple time zones.
• Maintain technical documentation, runbooks, and architectural decision records, contributing to team knowledge sharing and operational readiness across on-call and incident response scenarios.
• Apply CI/CD best practices to data pipeline development, including version control, automated testing, and deployment tooling to ensure reliable and repeatable pipeline delivery.

Requirements

What You Will Bring

• 4+ years of data engineering experience.
• At least 2 years of experience with Databricks or the Apache Spark ecosystem across Azure and/or AWS.
• Proficiency in PySpark, SQL, and Python with a strong track record of building and operating production‑grade pipelines under SLA constraints.
• Hands‑on experience with Delta Lake, including schema evolution, ACID transactions, optimize/vacuum lifecycle, and both incremental and streaming processing patterns.
• Hands‑on experience with pipeline performance tuning and compute optimization in production Databricks environments.
• Solid working knowledge of PostgreSQL, including query optimization, schema design, and use as a source or sink in production data pipelines.
• Experience supporting and maintaining legacy ETL tooling (SSIS, Informatica, custom Python/SQL pipelines, or similar) in production.
• Experience supporting large‑scale multi‑tenant architectures with a focus on tenant isolation, per‑tenant performance, and data privacy, including navigating tools and platforms that default to single‑tenant assumptions.
• Proven ability to work collaboratively across data science, product, and infrastructure teams, owning end‑to‑end delivery in a cross‑functional environment.
• Strong understanding of data governance, security, and compliance principles, including access control, data privacy, and protection of sensitive enterprise data across multi‑tenant environments.

Preferred Qualifications / Experience

• Experience operating Databricks workspaces across both Azure and AWS, including cost governance, cluster management, and cross‑cloud data access.
• Experience optimizing Databricks workloads in a Serverless environment, including compute cost governance and performance tuning for serverless compute.
• Experience with Microsoft SQL Server in a data engineering or ETL context.
• Exposure to ML feature engineering or feature stores (Databricks Feature Store, Feast, or similar) supporting predictive analytics.
• Experience with customer onboarding automation or Infrastructure as Code (IaC) patterns for provisioning tenant data pipelines at scale.
• Databricks Certified Data Engineer Associate or Professional certification