Data Scientist - Azure Databricks, Pipelines & Governance

Intellibee Inc; Indianapolis (IN); On-site; USD 110,000 - 170,000; Full time

Benefits offered by this job

Long-Term Stability

Comprehensive Health Coverage

401(k) Program

Green Card assistance

Job summary

IntelliBee seeks a Data Scientist/Data Engineer to drive data integration, pipeline development, governance, and modeling for high-quality data products. You will lead batch and streaming pipelines in Azure Databricks using PySpark, Spark SQL, and Delta Lake, while optimizing warehouse workloads in Azure Synapse Analytics and enforcing HIPAA-aligned data governance.

Collaboration across architects, analysts, and PMs is essential.

Qualifications

• Master's degree or 4+ years data engineering experience
• Bachelor's + 2+ years in data engineering
• Certifications may substitute for formal requirements
• Databricks notebooks in PySpark, Spark SQL, and R
• ACID, MERGE, schema evolution and maintenance
• Medallion architecture and Auto Loader ingestion
• Unity Catalog for permissions and lineage
• GPU/cluster tuning and SQL warehouse optimization
• ADF Pipelines and Databricks Workflows management
• Synapse/ADF pipelines and CETAS/COPY INTO
• Monitoring with Azure Monitor and Log Analytics
• Strong written and verbal communication
• Lakehouse patterns between Synapse and Databricks
• Databricks/Data Engineer certifications desirable
• Power BI connectivity to Synapse
• 4 Years in related roles

Responsibilities

• Assist data strategy and governance through data integration efforts
• Build batch and streaming pipelines in Databricks
• Develop dimensional models and SQL workloads in Synapse
• Enforce data governance and security controls for PHI/PII
• Collaborate with data architects, scientists, analysts, PMs
• Document solutions with diagrams and comments
• Support enterprise data operations and policy improvements
• Translate business needs into technical designs
• Mentor junior staff as needed
• Monitor and optimize data workflows and pipelines

Skills

Databricks

PySpark

Spark SQL

Delta Lake

Azure Synapse Analytics

Azure Data Factory

Unity Catalog

Databricks Workflows

OPENROWSET / Parquet / Delta

Power BI

Git / Repos

Education

Master's degree in data science or related field

Bachelor's degree + 2+ years experience or 4+ years data engineering

Databricks / Azure certifications desirable

Tools

Azure Databricks

Azure Synapse Analytics

Azure Data Factory

Databricks Workflows

Delta Lake

Job description

IntelliBee seeks a Data Scientist/Data Engineer to drive data integration, pipeline development, governance, and modeling for high-quality data products. You will lead batch and streaming pipelines in Azure Databricks using PySpark, Spark SQL, and Delta Lake, while optimizing warehouse workloads in Azure Synapse Analytics and enforcing HIPAA-aligned data governance.

Collaboration across architects, analysts, and PMs is essential.