Data Engineer: Azure Databricks Pipelines & Data Governance

Intellibee; Indianapolis (IN); Presencial; USD 120.000 - 160.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Long-Term Stability

Comprehensive Health Coverage

Future Planning

GC Assistance

Descripción de la vacante

IntelliBee is seeking a data engineering specialist to build batch and streaming data pipelines in Azure Databricks, optimize data warehouse workloads, and enforce governance for PHI/PII data under HIPAA.

You will collaborate with data architects, scientists, analysts, and project managers to translate business needs into scalable technical designs and maintain documentation and best practices across the data platform.

Formación

• Master's degree in data science, analytics, informatics, CS, math, engineering, or related field.
• 4+ years of hands-on data engineering or 2+ years with a relevant degree.
• Certifications may substitute for formal education/experience requirements.
• Proficient with Databricks notebooks in PySpark, Spark SQL, and R.
• Experience with ACID, MERGE, schema evolution, and time travel concepts.

Responsabilidades

• Assist with data integration strategy, data quality, and real-time data workflows.
• Design and implement batch and streaming pipelines in Azure Databricks.
• Develop data models and SQL workloads in Azure Synapse Analytics.
• Orchestrate workflows with Databricks Workflows and Synapse pipelines.
• Apply data governance controls to protect PHI/PII per HIPAA and policy.
• Document solutions using flowcharts, diagrams, and clear code comments.

Conocimientos

Databricks

PySpark

Spark SQL

Delta Lake

Azure Synapse Analytics

Auto Loader

Unity Catalog

Structured Streaming

Git/GitHub

Azure Data Factory

Azure Databricks Workflows

Educación

Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field

Bachelor's degree plus 2+ years of relevant experience; or 4+ years hands-on data engineering experience

Databricks and Azure certifications may substitute for formal education/experience

Herramientas

Azure Databricks

Spark

Delta Lake

Azure Synapse

ADF / Synapse Pipelines

Databricks Workflows

Descripción del empleo

IntelliBee is seeking a data engineering specialist to build batch and streaming data pipelines in Azure Databricks, optimize data warehouse workloads, and enforce governance for PHI/PII data under HIPAA.

You will collaborate with data architects, scientists, analysts, and project managers to translate business needs into scalable technical designs and maintain documentation and best practices across the data platform.