Senior Software Engineer

Mphasis

New York (NY)

Vor Ort

USD 60.000 - 118.000

Vollzeit

vor 14 Stunden
Sei unter den ersten Bewerbenden

Erhalte eine Antwort von diesem Arbeitgeber

Dein Lebenslauf und dein Anschreiben, genau auf das zugeschnitten, was gesucht wird.

Schaffe es an den ATS-Filtern vorbei

Zusammenfassung

Mphasis in New York seeks a hands-on Senior Databricks Engineer with 8–10 years of data engineering experience. You will design, build, test, optimize and operationalize scalable batch and streaming pipelines, Delta Lake data products and governed Lakehouse solutions for analytics, BI and AI workloads within the Insurance and Benefits Management domain.

You will use Azure Databricks, Spark, PySpark, Unity Catalog and Lakeflow to implement robust pipelines, data quality rules, lineage and cost

Qualifikationen

  • 8-10 years of data engineering experience.
  • Hands-on Azure Databricks experience and big-data pipelines.
  • Bachelor's degree in CS or related field; Databricks/Azure cert preferred.

Aufgaben

  • Develop and optimize ETL/ELT pipelines with Azure Databricks, Spark, PySpark and Python.
  • Build Bronze, Silver and Gold layers using Delta Lake with schema enforcement.
  • Implement API, DB, file and streaming ingestion using Auto Loader, Lakeflow and Azure tools.
  • Create curated tables and data products for Databricks SQL, BI and AI workloads.

Kenntnisse

Azure Databricks
Spark
PySpark
Delta Lake
Lakehouse
Unity Catalog
Python
SQL

Ausbildung

Bachelor's degree in Computer Science or related field

Tools

Auto Loader
Lakeflow
Git

Jobbeschreibung

Role Description
JOB DESCRIPTION - SENIOR DATABRICKS ENGINEER
Azure Databricks | Spark & PySpark | Delta Lake | Lakeflow | Unity Catalog
Experience

8-10 years

Role Level

Senior Databricks Engineer

Primary Platform

Microsoft Azure and Azure Databricks

Domain

Insurance and Benefits Management

Role Summary

Seeking a hands-on Senior Databricks Engineer with 8-10 years of data engineering experience and strong expertise in Azure Databricks. The role will design, build, test, optimize and operationalize scalable batch and streaming pipelines, Delta Lake data products and governed Lakehouse solutions for analytics, BI and AI workloads, supported by Insurance and Benefits Management domain experience.

Key Responsibilities
  • Develop and optimize ETL/ELT pipelines using Azure Databricks, Spark, PySpark, Spark SQL and Python.
  • Build Bronze, Silver and Gold data layers using Delta Lake, schema enforcement, incremental processing and reusable engineering patterns.
  • Implement ingestion from APIs, databases, files, cloud storage and streaming sources using Auto Loader, Lakeflow and Azure orchestration services.
  • Develop batch and near-real-time pipelines using Spark Structured Streaming, checkpoints, watermarking and resilient recovery patterns.
  • Create curated tables, views and data products for Databricks SQL, Power BI, Genie, analytics and downstream AI/ML consumption.
  • Apply Unity Catalog controls, lineage, data classification, access policies, masking and employer-level data isolation.
  • Implement data-quality rules, validation, reconciliation, error handling, audit controls and operational observability.
  • Optimize Spark jobs, Delta tables, queries, partitioning, file sizes, caching, Photon workloads, compute utilization and cloud cost.
  • Build automated tests and CI/CD pipelines for notebooks, jobs, Lakeflow pipelines and data assets using Git and Azure DevOps.
  • Participate in design and code reviews, troubleshoot production issues, maintain technical documentation and mentor junior engineers.
Mandatory Technical Skills

Microsoft Azure; Azure Databricks; Apache Spark; PySpark; Spark SQL; Python; advanced SQL; Delta Lake; Lakehouse and Bronze/Silver/Gold architecture; Auto Loader; Lakeflow Connect, pipelines and jobs; Spark Structured Streaming; Databricks Workflows; Databricks SQL and Photon; Unity Catalog; ADLS Gen2; Azure Data Factory or equivalent orchestration; REST APIs; JSON, CSV, Parquet and Avro; data modelling; ETL/ELT; data quality and reconciliation; Git; Azure DevOps or GitHub Actions; CI/CD; Terraform awareness; monitoring, logging, ing, security, performance tuning and cost optimization.

Experience And Qualifications
  • 8-10 years of experience in data engineering, big-data development, ETL/ELT, data warehousing or cloud analytics.
  • At least 4 years of strong hands-on experience developing production solutions on Azure Databricks or an equivalent Databricks environment.
  • Proven experience building and supporting scalable Spark and Delta Lake pipelines for structured and semi-structured data.
  • Experience with Unity Catalog, data lineage, access controls, data-quality frameworks and production deployment practices.
  • Experience working with data architects, modellers, business analysts, QA, DevOps, security and BI teams in Agile delivery.
  • Bachelor’s degree in Computer Science, Information Technology, Data Engineering or a related discipline; Databricks or Azure certification is preferred.
Domain Experience
  • Insurance or employee-benefits experience involving employer, client, plan, member, enrollment, carrier, claims or renewal data is preferred.
  • Experience handling privacy-sensitive and regulated information with traceability, auditability and role-based access controls.
  • Familiarity with EDI, HRIS, benefits-administration, CRM or document-derived data is advantageous.
Core Competencies

Hands-on engineering depth, structured problem-solving, production ownership, data-quality discipline, performance tuning, collaboration, mentoring and the ability to balance scalability, reliability, governance, maintainability and cost.

Other Details

Salary range $60000 to $117500

Hol dir deinen kostenlosen, vertraulichen Lebenslauf-Check.

oder ziehe deine Datei hierhin.

Similar jobs

Ähnliche Jobs, die dir auch gefallen könnten

Databricks Developer
Databricks Developer

Vservice It Solutions • USA

Remote
USD 140.000 - 190.000
Senior Data Engineer, Azure Databricks
Senior Data Engineer, Azure Databricks

Nextech Solutions • USA

Remote
USD 130.000 - 170.000
Sr Azure Databricks Engineer
Sr Azure Databricks Engineer

ITMC Systems, Inc • USA

Remote
USD 140.000 - 190.000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Virtue America • New York (NY)

Vor Ort
USD 140.000 - 190.000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Fa Ewjt Saasfaprod1 • New York (NY)

Vor Ort
USD 140.000 - 170.000
Senior Data Engineer
Senior Data Engineer

Tata Consultancy Services Limited • Irvine (CA)

Vor Ort
USD 120.000 - 180.000
Databricks Data Architect
Databricks Data Architect

NewVision • San Francisco (CA)

Hybrid
USD 140.000 - 190.000
Hybrid work model
Databricks Data Architect
Databricks Data Architect

NewVision Software • USA

Remote
USD 120.000 - 180.000
Senior Azure Databricks Data Engineer
Senior Azure Databricks Data Engineer

EXL • Detroit (MI)

Vor Ort
USD 120.000 - 190.000
Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

UNAVAILABLE • McLean (VA)

Vor Ort
USD 120.000 - 160.000