Senior Data Scientist: Databricks & Synapse Expert (Hybrid)

Career Soft Solutions Inc; Indianapolis (IN); Hybrid; USD 110,000 - 150,000; Full time

Job summary

Career Soft Solutions Inc. is seeking a senior data engineer to design and optimize large-scale data pipelines in a hybrid Indianapolis office setup (1 day in-office, 4 days remote).

You will work with Databricks, Delta Lake, and Unity Catalog to build robust ingest, transformation, and data-consumption layers for enterprise analytics. The role requires advanced SQL, Spark (PySpark), and cloud data warehousing experience, plus excellent communication.

Qualifications

• Advanced degree in a data-related field is preferred.
• Bachelor's plus 2+ years of relevant experience or 4+ years hands-on data engineering work.
• Certifications or coursework may substitute for formal requirements.

Responsibilities

• Develop Databricks notebooks in PySpark, Spark SQL, and R.
• Handle ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM).
• Work with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT.
• Use Unity Catalog for workspace catalogs, schemas, permissions, lineage, and storage credentials.
• Right-size clusters and SQL warehouses; diagnose partitioning, shuffle, skew, and caching issues via Spark UI.
• Schedule and manage dependencies with ADF Pipelines, Databricks Workflows, and Git repositories for source control.
• Design MPP warehouse tables with distribution/hash, partitioning, and columnstore indexes; build star schemas.
• Query Parquet, Delta, and CSV in ADLS Gen2 using OPENROWSET and external tables; create views and stored procedures.
• Build ETL pipelines with COPY INTO, CETAS, and parameterized Synapse/ADF pipelines (copy, data flows, triggers).
• Monitor pipelines with Azure Monitor/Log Analytics and troubleshoot failed jobs.
• Demonstrate strong written and verbal communication skills.

Skills

Data engineering

SQL proficiency

Spark / PySpark

Cloud data pipelines

Communication skills

Education

Master's degree in data science / related field

Bachelor's degree + 2+ years OR 4+ years hands-on experience

Certifications substitute for education/experience

Tools

Databricks

Azure Synapse

ADF Pipelines / Databricks Workflows

Unity Catalog

Git / GitHub / Azure DevOps

Delta Lake

OPENROWSET / Parquet / Delta formats

Job description

Career Soft Solutions Inc. is seeking a senior data engineer to design and optimize large-scale data pipelines in a hybrid Indianapolis office setup (1 day in-office, 4 days remote).

You will work with Databricks, Delta Lake, and Unity Catalog to build robust ingest, transformation, and data-consumption layers for enterprise analytics. The role requires advanced SQL, Spark (PySpark), and cloud data warehousing experience, plus excellent communication.