Databricks Engineer

Ventures Unlimited Inc; O’Fallon (MO); On-site; USD 90,000 - 120,000; Full time

Job summary

A renowned IT consulting firm is seeking a Recruitment Manager to lead IT recruitment in O’Fallon, Missouri. This mid-senior level role focuses on building and optimizing data pipelines using Databricks, along with ensuring data observability and effective recruitment strategies. The ideal candidate will have at least 8+ years of experience in IT recruitment and software development, with a strong emphasis on Python, real-time data processing, and CI/CD practices. Join a dynamic team and impact the future of data-driven solutions.

Qualifications

• 8+ years of experience in software development and delivery.
• Strongly prefer experience with real-time data processing and data streaming.
• Experience with CI/CD and agile methodologies.

Responsibilities

• Design, develop, and optimize data pipelines on Databricks.
• Build data transformation workflows using dbt and TDD.
• Implement data observability practices across platforms.

Skills

Data observability tools

Databricks

Python

Spark SQL

Power BI

CI/CD

Real-time data processing

SQL development

Cloud infrastructure (AWS, Azure, GCP)

Agile methodologies

Education

Bachelor's degree in Computer Science or related field

Tools

Git

Jenkins

Kafka

Job description

Recruitment Manager @ Ventures Unlimited Inc | IT Recruitment, Team Management

Client :- TCS

Duration:- Full Time Position

Job Description:-

Experience Required - 8+ Years

Must Have Technical/Functional Skills

• Tenured in the fields of Computer Science/Engineering or Software Engineering.
• Bachelor's degree in computer science, or a related technical field including programming with 15+ years of demonstrated experience in software development and delivery.
• Deep hands‑on experience with Databricks and Python.
• Spark and SQL Expertise: Writing performant Spark SQL and PySpark transformations. Optimizing joins, window functions, and aggregations. Handling large-scale data with partitioning and caching strategies
• Experience with real‑time data processing and streaming pipelines.
• Proficient in both Delta Live Tables (DLT) and Workflows – Understanding of Lakeflow Declarative Pipelines is an asset.
• DevOps and CI/CD - Using Databricks Asset Bundles (DABs) for deployment - Integrating with Git, Bitbucket, and Jenkins for version control and automation
• Proven track record using dbt for building robust, testable data transformation workflows following TDD.
• Strong grasp of cloud‑based database infrastructure (AWS, Azure, or GCP).
• Skilled in developing insightful dashboards and scalable data models using Power BI.
• Expert in SQL development and performance optimization.
• Demonstrated success in building and maintaining data observability tools and frameworks.
• Proven ability to plan and execute deployments, upgrades, and migrations with minimal disruption to operations.
• Focus on standing up/adopting tools that will enable both internal and external customers. Key leader/contributor in the data and software and platform and data warehouse development life cycle (i.e. design, development, documentation, testing, deployment, support)
• Work efficiently within a high security, PII and PCI‑DSS environments
• Key leader/contributor in the data and software and platform and data warehouse development life cycle (i.e. design, development, documentation, testing, deployment, support)
• Support architects and engineers as they design and build effective, agile applications and platforms
• Design, develop, and optimize batch and real‑time data pipelines using Medallion Architecture, preferably on Databricks cloud data platforms
• Build data transformation workflows using dbt, with a strong focus on Test‑Driven Development (TDD) and modular design.
• Implement and manage CI/CD pipelines using GitLab and Jenkins, enabling automated testing, deployment, monitoring of data workflows.
• Drive DataOps practices by integrating testing, monitoring, versioning, and collaboration into every phase of the data pipeline lifecycle.
• Build scalable and reusable data models that support business analytics and dashboarding in Power BI.
• Develop and support real‑time data streaming pipelines (e.g., using Kafka, Spark Structured Streaming) for near‑instant data availability.
• Establish and implement data observability practices, including monitoring data quality, freshness, lineage, and anomaly detection across the platform.
• Plan and own deployments, migrations, and upgrades across data platforms and pipelines to minimize service impacts, including developing and executing mitigation plans.
• Collaborate with stakeholders to understand data requirements and deliver reliable, high‑impact data solutions.
• Document pipeline architecture, processes, and standards, promoting consistency and transparency across the team.
• Apply exceptional problem‑solving and analytical skills to troubleshoot complex data and system issues.
• Demonstrate excellent written and verbal communication skills when collaborating across technical and non‑technical teams.

Seniority level

Mid‑Senior level

Employment type

Full‑time

Job function

Consulting