Data Engineer

Business Needs Inc; Austin (TX); Hybrid; USD 110,000 - 160,000; Full time

Job summary

Business Needs Inc. in Austin, TX is seeking a Data Engineer to design, develop, and maintain scalable data pipelines, ingestion frameworks, and reusable data products using Azure Databricks, PySpark, SQL, and Delta Lake.

You will implement Bronze, Silver, and Gold patterns, support batch and streaming workloads, lead technical reviews, mentor engineers, and collaborate with data architects to ensure secure, governed, cost-conscious solutions.

Qualifications

• Design, build, and maintain scalable data pipelines and products.
• Experience with Azure Databricks, PySpark, SQL, and Delta Lake.
• Lead reviews, mentor engineers, and promote coding standards.

Responsibilities

• Design, develop, and maintain scalable data pipelines and platforms.
• Implement Bronze/Silver/Gold data architecture patterns.
• Support batch, near-real-time, and streaming data workloads.

Skills

Data engineering

Data pipeline design

Mentoring

Observability

Tools

Azure Databricks

PySpark

SQL

Delta Lake

Azure DevOps

Git

Job description

Data Engineer
Duration: Long term
Location: Austin, TX- 3 days a week Hybrid
Details:

Design, develop, and maintain scalable data pipelines, ingestion frameworks, transformation processes, and reusable data products using Azure Databricks, PySpark, SQL, and Delta Lake.

Implement Bronze, Silver, and Gold architecture patterns supporting enterprise reporting, analytics, AI, and self-service data consumption.

Build reusable frameworks, utilities, and platform components that improve engineering productivity, quality, consistency, and deployment speed.

Develop and support batch, near-real-time, and streaming integration solutions, including modernization of legacy warehouse and ETL workloads.

Serve as technical lead for complex initiatives; develop solution designs, lead technical reviews, recommend tools and approaches, and guide work through production implementation.

Partner with data architects and platform leaders to ensure solutions are scalable, secure, governed, cost-conscious, and operationally supportable.

Mentor engineers and promote standards for coding, testing, documentation, performance, and production readiness.

Implement data quality, validation, reconciliation, monitoring, metadata, and lineage capabilities; support RBAC and enterprise security controls.

Build and maintain CI/CD, automated testing, deployment, and release processes using Azure DevOps and Git-based practices.

Contribute to platform observability, alerting, operational dashboards, health metrics, performance tuning, and cost optimization.

Participate in production support, incident response, pager, and on-call rotations; troubleshoot issues, lead root cause analysis, and implement durable remediation.