Data Scientist

Career Soft Solutions Inc; Indianapolis (IN); Hybride; USD 110 000 - 150 000; Plein temps

Résumé du poste

Career Soft Solutions Inc. is seeking a senior data engineer to design and optimize large-scale data pipelines in a hybrid Indianapolis office setup (1 day in-office, 4 days remote).

You will work with Databricks, Delta Lake, and Unity Catalog to build robust ingest, transformation, and data-consumption layers for enterprise analytics. The role requires advanced SQL, Spark (PySpark), and cloud data warehousing experience, plus excellent communication.

Qualifications

• Advanced degree in a data-related field is preferred.
• Bachelor's plus 2+ years of relevant experience or 4+ years hands-on data engineering work.
• Certifications or coursework may substitute for formal requirements.

Responsabilités

• Develop Databricks notebooks in PySpark, Spark SQL, and R.
• Handle ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM).
• Work with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT.
• Use Unity Catalog for workspace catalogs, schemas, permissions, lineage, and storage credentials.
• Right-size clusters and SQL warehouses; diagnose partitioning, shuffle, skew, and caching issues via Spark UI.
• Schedule and manage dependencies with ADF Pipelines, Databricks Workflows, and Git repositories for source control.
• Design MPP warehouse tables with distribution/hash, partitioning, and columnstore indexes; build star schemas.
• Query Parquet, Delta, and CSV in ADLS Gen2 using OPENROWSET and external tables; create views and stored procedures.
• Build ETL pipelines with COPY INTO, CETAS, and parameterized Synapse/ADF pipelines (copy, data flows, triggers).
• Monitor pipelines with Azure Monitor/Log Analytics and troubleshoot failed jobs.
• Demonstrate strong written and verbal communication skills.

Connaissances

Data engineering

SQL proficiency

Spark / PySpark

Cloud data pipelines

Communication skills

Formation

Master's degree in data science / related field

Bachelor's degree + 2+ years OR 4+ years hands-on experience

Certifications substitute for education/experience

Outils

Databricks

Azure Synapse

ADF Pipelines / Databricks Workflows

Unity Catalog

Git / GitHub / Azure DevOps

Delta Lake

OPENROWSET / Parquet / Delta formats

Description du poste

Indianapolis, IN (One day in office 4 days remote)

Master's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; ...or

Bachelor's degree in a related field plus 2+ years of relevant experience; or 4+ years of hands‑on experience in a similar data engineering role; OR..

Relevant certifications and coursework may substitute for formal education/experience requirements

Developing with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions

ACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM)

Working with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT

Using Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials

Right-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI

Scheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control

Designing MPP warehouse tables, including distribution (hash, round-robin, replicated), column store indexes, and partitioning

Querying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views

Building star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2

Developing with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers)

Workload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing

ADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault

Git and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles

Monitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs

Strong written and verbal communication skill

Using Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks

Power BI connectivity to Synapse (Import vs. DirectQuery) and how model design affects dashboard performance