Data Engineering Architect, Senior

Bloomberg BNA; Arlington (VA); On-site; USD 180,000 - 220,000; Full time

Job summary

Bloomberg Industry Group is seeking an experienced Data Engineer to lead end-to-end data pipelines in a Databricks Lakehouse, translating product analytics needs into scalable, governed datasets.

You will design future-state architectures, drive data quality, and mentor engineers while collaborating with analytics and product teams to power analytics, experimentation, and decision-making.

Qualifications

• Bachelor's degree in computer science or related discipline or equivalent experience.
• 10+ years of experience in data engineering or data architecture.
• Hands-on production experience with Databricks in cloud environments.
• Experience designing scalable data pipelines and governance patterns.
• Ability to translate business needs into reliable data products.

Responsibilities

• Own end-to-end data pipelines from ingestion to serving in a Databricks Lakehouse.
• Define and evolve data architectures, models, and production patterns.
• Lead data quality, monitoring, SLAs, and incident resolution for datasets.
• Collaborate with product analytics to translate KPIs into datasets.
• Mentor engineers and drive adoption of reusable data frameworks.

Skills

SQL

Python

Databricks

Spark

Data modeling

Data governance

Architectural design

Stakeholder collaboration

CI/CD

Education

Bachelor's degree in Computer Science or related

Master's degree preferred

Tools

Databricks

Spark/PySpark

SQL engines

Job description

Primary Responsibilities:

You are a data engineer who thrives in a highly collaborative environment, partnering with product, analytics, and engineering teams to deliver high-quality, trusted data. You're motivated by building scalable data systems and shaping how data is modeled, governed, and consumed across a modern cloud platform. You bring deep, hands-on data engineering experience and are equally comfortable designing future-state architecture, building production solutions, and establishing the patterns and standards that allow others to build effectively. You bring recent, hands-on production experience with Databricks and will play a leading role in evolving our Databricks-based Lakehouse architecture, including the modernization and migration of existing data workloads. You enjoy translating complex product and user behavior into well-structured, reliable datasets that power analytics, experimentation, and decision-making. You can move comfortably between technical implementation and strategic architecture, communicating complex decisions clearly and influencing technical direction across teams. You bring recent, hands-on production experience with Databricks and will play a leading role in evolving our Databricks-based Lakehouse architecture, including the modernization and migration of existing data workloads. You enjoy translating complex product and user behavior into well-structured, reliable datasets that power analytics, experimentation, and decision-making. You can move comfortably between technical implementation and strategic architecture, communicating complex decisions clearly and influencing technical direction across teams.

Partner with product analytics stakeholders to translate business-defined KPIs and data requirements into scalable, production-grade datasets. Own the design, build, and operation of scalable data pipelines end-to-end (ingestion → transformation → serving). Define and evolve the architecture of the Product Analytics Lakehouse, making technical decisions that improve scalability, performance, reliability, governance, and consistency across datasets and workloads. Build and maintain production-grade, well-modeled datasets (Gold layer) that power analytics and AI use cases. Define, implement, and drive adoption of reusable data engineering patterns, frameworks, standards, and guardrails that reduce duplication, improve engineering leverage, and make the right development patterns easier to adopt. Own data quality and reliability for production datasets, including validation, monitoring, SLAs, and incident resolution. Productionize and scale prototype datasets and logic developed by analytics partners into reliable, maintainable data pipelines. Build governed, purpose-built datasets to support AI/ML use cases while enforcing controlled and secure data access patterns. Lead the technical evolution of workloads into Databricks, evaluating existing architecture and determining appropriate migration, modernization, and coexistence strategies. Make and communicate architectural tradeoffs across performance, cost, reliability, governance, maintainability, and developer experience. Provide technical leadership and architectural guidance across Product Analytics, helping engineers and analytics partners make sound data architecture, modeling, and platform decisions. Mentor and provide technical guidance to engineers and other technical contributors, raising engineering standards through hands-on leadership rather than formal authority.

Job Requirements:

Strong experience building and operating data pipelines using SQL and Python in a modern cloud environment. Deep expertise in SQL, including complex transformations, data modeling, query optimization, and performance tuning at scale. 2+ years of recent, hands-on production experience with Databricks, including designing, building, optimizing, and operating production data workloads. Strong hands-on experience with Spark/PySpark and distributed data processing in a production environment. Strong understanding of modern data architecture patterns, including Lakehouse architecture, ELT, and layered data models (bronze/silver/gold). Proven experience designing data models for analytics, including dimensional or domain-oriented approaches. Experience driving database and data engineering best practices, including schema design, migrations, and performance optimization. Demonstrated ability to own consequential architecture and engineering decisions and drive them from design through production in environments with limited structure or support. Demonstrated experience establishing reusable data frameworks, standards, and guardrails that have been successfully adopted beyond an individual project or pipeline. Experience owning production data systems, including monitoring, debugging, and resolving data pipeline failures. Experience working closely with business stakeholders or analysts to translate ambiguous requirements into scalable data solutions. Strong technical judgment in evaluating technologies and architecture patterns based on scalability, reliability, cost, maintainability, governance, and developer experience—not technology novelty alone. Experience evolving or migrating legacy data platforms and workloads while maintaining continuity for downstream consumers. Strong understanding of data governance, including metadata, lineage, access controls, data quality, and the role of governance in creating trusted, reusable data products. Strong record of project execution and completion with experience with agile development practices. Excellent written and verbal communication skills, with demonstrated ability to translate complex technical concepts for both technical and non-technical audiences, influence technical direction across teams, and build alignment around architecture and engineering standards. Experience with developing source-controlled pipelines in a CI/CD environment. Experience working with product analytics data (event tracking, user behavior, experimentation) is a plus.

Education and Experience:

Bachelor's degree in computer science or a related discipline, or equivalent experience. Master's degree preferred. 10+ years of experience in data engineering, data architecture, software engineering, or related fields, including significant experience designing and owning production data systems and architecture. Hands-on experience building and maintaining scalable data pipelines and datasets in cloud-based environments with recent production experience in Databricks and cloud infrastructure. Proven ability to design and evolve data architectures, data models, and production pipelines that support analytics at scale, including performance, reliability, governance, and cost optimization. Experience taking data solutions from concept to production, including monitoring, debugging, and ongoing maintenance. Demonstrated ability to operate independently at an architectural level, make high-impact technical decisions, influence engineering practices across teams, and raise data platform standards in environments with limited structure.

Compensation Range:

$180,000.00-$220,000.00

Equal Opportunity Bloomberg Industry Group maintains a continuing policy of non-discrimination in employment. It is Bloomberg Industry Group's policy to provide equal opportunity and access for all persons, and the Company is committed to attracting, retaining, developing, and promoting the most qualified individuals without regard to age, ancestry, color, gender identity or expression, genetic predisposition or carrier status, marital status, national or ethnic origin, race, religion or belief, sex, sexual orientation, sexual and other reproductive health decisions, parental or caring status, physical or mental disability, pregnancy or maternity/parental leave, protected veteran status, status as a victim of domestic violence, or any other classification protected by applicable law (“Protected Characteristic”). Bloomberg prohibits treating applicants or employees less favorably in connection with the terms and conditions of employment, in all phases of the employment process, because of one or more Protected Characteristics (“Discrimination”). Bloomberg Industry Group, an affiliate of Bloomberg L.P., is a leading source of legal, tax, regulatory, government, and business information for professionals.