Senior Data Engineer (Azure Databricks)

EmergenceTek Group Inc.; United States; Hybrid; USD 140,000 - 210,000; Full time

Benefits offered by this job

Remote or hybrid work

Job summary

EmergenceTek Group, a boutique IT consulting firm, is seeking aSenior Data Engineer to design, build, and optimize Azure Databricks data pipelines for aerospace/defense clients.

You’ll collaborate with a team of data engineers, analysts, and architects, contributing immediately in a hands-on senior role from a remote or hybrid setup in EST hours.

Qualifications

• 5+ years of professional data engineering experience building production data pipelines and data platforms.

Responsibilities

• Design, build, and maintain batch and streaming ingestion and transformation pipelines on Azure Databricks using a medallion (bronze/silver/gold) Lakehouse architecture with Delta Lake.

Skills

Azure Databricks

Python/PySpark

SQL

Data modeling

Generative AI tooling

Azure Data Factory

Unity Catalog

CI/CD

ETL/ELT

Cloud security

Tools

Databricks

Azure

Delta Lake

Job description

We're hiring a Senior Data Engineer to support a manufacturer serving aerospace and defense markets as it builds out its Azure Databricks data platform. This is a full-time W2 position with ETG.

You will work with a team of data engineers, analysts, and architects to accelerate a data platform backlog already underway. This is a hands-on, senior-level role for a self-directed engineer who can contribute from day one.

• Remote or hybrid, EST working hours. Proximity to Buffalo / East Aurora, NY is a plus
• Competitive compensation
• U.S. citizenship required as role involves U.S. export-controlled information
• This is a W2 position only. No C2C, no third-party submissions, and no visa sponsorship available.

What you'll do

• Design, build, and maintain batch and streaming ingestion and transformation pipelines on Azure Databricks using a medallion (bronze/silver/gold) Lakehouse architecture with Delta Lake
• Build and use AI agents and LLM-assisted tooling to discover source systems, profile data, propose models and mappings, and generate pipeline code, with appropriate human review and validation
• Maximize low-code and no-code ingestion where it fits (e.g., Databricks-native ingestion, declarative pipelines, metadata-driven frameworks) and reserve custom code for the cases that need it
• Register, secure, and document data assets in Unity Catalog, including lineage, access controls, catalog metadata, and tagging for sensitive and export-controlled data
• Prepare curated, well-described datasets that power Databricks AI/BI Genie spaces, dashboards, and other natural-language and self-service analytics
• Build data quality checks, monitoring, and alerting into pipelines; conduct root cause analysis, troubleshoot and tune for performance, reliability, and cost, and provide post-implementation support
• Integrate data from multiple disparate ERP and business systems (e.g., SAP and other ERP platforms), integration-hub and Azure Data Factory pipelines, and MES, PLM, SCADA, quality management, supply chain, and shop-floor/IoT sources
• Use CI/CD and infrastructure-as-code practices (e.g., Git-based workflows and automated deployment pipelines) to deploy and promote work across environments
• Partner with the data modeler and architect to implement conceptual, logical, and physical models in the Lakehouse, following enterprise modeling standards, data dictionaries, and data definitions
• Participate in sprint planning, estimation, and code review; mentor peers on the client team and share reusable patterns

Required Experience

• 5+ years of professional data engineering experience building production data pipelines and data platforms
• 3+ years of hands-on experience with Azure Databricks (Spark, Delta Lake, Workflows/Jobs, Unity Catalog)
• 3+ years of experience with Lakehouse, medallion architecture, data warehousing, and/or data vault techniques
• Advanced proficiency in Python/PySpark and SQL
• Demonstrated hands-on experience building with generative AI or agentic tooling for data engineering work (e.g., Databricks Genie, Databricks AI/agent tooling, or comparable LLM and agent frameworks), with examples you can walk through in an interview
• Experience with Azure data services, including Azure Data Factory, and with Azure cloud storage, security, and identity concepts (e.g., SSO/Entra ID)
• Experience integrating data from complex enterprise source systems with medium-to-large scale datasets
• Experience in a manufacturing, aerospace, defense, or other regulated industrial environment

Required Skills

• Solid understanding of data modeling principles (relational, star, and snowflake schemas), Medallion Architecture, and enterprise integration patterns
• Working knowledge of data security, access control, and handling of sensitive or regulated data
• Sound engineering practices: version control, automated testing, code review, and CI/CD
• Experience working on blended or distributed teams of contractors and employees within an Agile/sprint delivery model
• Senior-level ability to work independently and deliver without extensive onboarding or training; a self-starter who is energized by a large backlog
• Strong analytical and problem-solving skills with excellent attention to detail
• Ability to challenge the status quo and find a better way to solve a problem, with the ability to influence others in this regard
• Strong communication (verbal and written) and interpersonal skills; able to explain technical trade-offs in business terms
• Strong organization, planning, and time management skills

Preferred Skills

• Databricks Data Engineer (Associate or Professional) or Microsoft Azure data certifications
• Experience with manufacturing data domains such as bill of materials, work orders, routings, inventory, quality/non-conformance, and supplier data
• Experience with ERP data structures (SAP experience a plus) and familiarity with MES, PLM, SCADA, and quality management systems
• Experience working from data models maintained in enterprise modeling tools (e.g., SqlDBM, Erwin)
• Familiarity with data contracts, including schema evolution standards and producer-consumer data quality expectations
• Experience building integration-hub or enterprise integration patterns with Azure Data Factory or similar tools
• Experience as or supporting a government aerospace/defense contractor, including awareness of ITAR/EAR, CUI handling, NIST SP 800-171, CMMC, or DFARS requirements
• Familiarity with AS9100 quality systems or program/contract data such as earned value management (EVM)
• Experience with Databricks platform tooling for ingestion, deployment automation, monitoring, and cost visibility
• Experience with Power BI, SSRS, or other BI tools that consume Lakehouse data
• Experience migrating workloads from legacy data platforms and on-premises warehouses to Databricks

About ETG

EmergenceTek Group is a boutique IT consulting firm that has worked with healthcare, manufacturing, and other high-trust industries since 2012. Work alongside senior engineers on long-term engagements, supported by a leadership team that stays in the work. You'll build real domain expertise working on meaningful problems.