About the Role:
Grade Level (for internal use):
11
Key Responsibilities
Data Pipeline Transition and Platform Delivery
- Partner with the core Databricks team to support the transition of existing data pipelines to the target enterprise data platform.
- Provide hands‑on technical guidance for enhancing, stabilizing, and optimizing pipelines after transition, with a focus on performance, scalability, maintainability, operational reliability, and secure AWS deployment patterns.
- Implement and promote repeatable engineering patterns for ingestion, transformation, testing, deployment, monitoring, and support across onboarded datasets.
- Ensure pipelines are designed and operated in a way that supports long‑term platform consistency, reliability, observability, and ease of maintenance.
- Support the design and operation of cloud‑native data pipelines using AWS services such as:
- Amazon S3 for durable data lake storage
- AWS Glue for data integration, cataloging, and processing
- AWS Lambda for event‑driven processing
- Amazon Kinesis for streaming use cases
- AWS Lake Formation for governed data lake controls
- AWS IAM for secure access control
- Promote the use of encryption, environment‑level controls, access policies, and platform guardrails to support secure access to platform resources and data products.
- Apply lakehouse technologies and concepts such as Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata‑driven pipelines, and governed data access patterns.
AI‑Assisted Engineering and Context Engineering
- Use AI‑assisted engineering tools such as Claude, GitHub CoPilot, LLMs, and similar platforms to accelerate data engineering delivery while maintaining enterprise quality and security standards.
- Apply AI tooling to support common engineering activities, including:
- Code generation
- Code refactoring
- SQL and PySpark optimization
- Unit test generation
- Documentation creation
- Pipeline analysis
- Troubleshooting and debugging
- Data quality rule suggestions
- Runbook and operational support documentation
- Apply prompt engineering and context engineering practices to improve the accuracy, relevance, and reliability of AI‑generated outputs.
- Structure technical context for AI‑assisted work, including requirements, schemas, data contracts, architecture constraints, coding standards, logs, error messages, acceptance criteria, and governance requirements.
- Create or contribute to reusable prompt templates, context packs, and engineering playbooks for common data platform activities such as Databricks migration, AWS data engineering, pipeline troubleshooting, and test generation.
- Review, validate, and refine AI‑generated code, recommendations, documentation, and analysis before applying them to production‑grade engineering work.
- Ensure responsible use of AI tooling by following enterprise standards for security, privacy, sensitive data protection, compliance, and human review.
Data Onboarding and Asset‑Agnostic Enablement
- Support asset‑agnostic data onboarding patterns that can accommodate a broad range of data assets, domains, and source systems without requiring bespoke platform redesign for each use case.
- Work with platform, data engineering, architecture, governance, and business‑aligned teams to simplify and standardize how data is ingested, transformed, governed, and published to the enterprise platform.
- Help offshore and global teams adopt common onboarding frameworks, technical standards, and delivery practices that improve speed, consistency, and reduce friction.
- Establish reusable ingestion and processing patterns across batch, streaming, and event‑driven use cases.
- Create or contribute to reusable technical assets such as:
- Design patterns
- Reference implementations
- Onboarding templates
- Pipeline frameworks
- Technical documentation
- Operational runbooks
- AI‑assisted engineering templates
- Support metadata‑driven and configuration‑driven pipeline patterns to improve scalability, consistency, and maintainability.
Data Mastering Platform Integration
- Support integration of platform pipelines and datasets with the enterprise data mastering platform.
- Collaborate with upstream and downstream stakeholders to ensure mastered data can be consumed reliably through standardized interfaces and governed data flows.
- Help implement data quality controls, reconciliation processes, metadata alignment, and stewardship workflows required to support trusted mastered data in the platform.
- Contribute to issue resolution and continuous improvement related to mastering‑related ingestion, enrichment, and distribution workflows.
- Support data mastering capabilities aligned with platforms such as NeoXam DataHub, including:
- Data acquisition
- Cleansing
- Enrichment
- Mastering
- Reconciliation
- Golden copy generation
- Downstream distribution of trusted data products
- Work with business and platform stakeholders to support golden record / golden copy generation and maintenance for key financial datasets, including reference data, instruments, business entities, market data, investment data, risk data, and related mastered domains.
Technical Leadership and Engineering Excellence
- Serve as a senior technical individual contributor for data platform engineering, with hands‑on involvement across pipeline migration, lakehouse architecture, AWS‑native data engineering, governance, AI‑assisted engineering, and mastering integrations.
- Provide technical guidance to engineers, delivery partners, and platform stakeholders without direct people management responsibility.
- Contribute to architecture discussions, design reviews, code reviews, implementation planning, technical standards, and production readiness reviews.
- Translate architectural direction into practical engineering tasks, implementation patterns, and reusable technical assets.
- Promote engineering best practices including:
- Version control
- Automated testing
- CI/CD
- Infrastructure as code
- Release automation
- Monitoring and alerting
- Incident response
- Documentation
- Production support readiness
- Help establish cloud engineering standards for infrastructure automation, release management, and environment promotion using tools and services such as AWS CodePipeline, AWS CodeBuild, and infrastructure automation frameworks.
- Drive operational discipline across production data pipelines, including observability, logging, telemetry, support models, service ownership, and incident management.
Collaboration, Governance, and Platform Standards
- Partner effectively with global platform, architecture, governance, security, Databricks‑aligned, and data mastering teams to ensure offshore delivery aligns with enterprise standards.
- Act as a technical bridge between platform direction and engineering execution.
- Support governance requirements through appropriate controls around:
- Data lineage
- Schema consistency
- Data quality
- Metadata
- Retention
- Access control
- Encryption
- Auditability
- Secure data distribution
- Ensure monitoring and operational health practices are in place using logging, alerting, telemetry, dashboards, and AWS-native operational tooling where appropriate.
- Help improve consistency and reusability across data onboarding and pipeline delivery by contributing to platform guardrails, standards, engineering playbooks, and reusable implementation patterns.
Required Qualifications
Professional Experience
- 6–8+ years of experience in data engineering, data platforms, cloud data architecture, software engineering, or related engineering domains.
- Demonstrated experience operating as a senior individual contributor, technical lead, senior data engineer, lead data platform engineer, or equivalent technical role.
- Proven ability to support technical initiatives and influence engineering outcomes in a complex, execution‑focused environment.
- Experience partnering with global or distributed teams to deliver platform and pipeline initiatives across time zones.
- Ability to provide technical guidance and improve delivery quality without direct people management authority.
Technical Expertise
- Strong hands‑on experience with modern data engineering and pipeline development, including batch and/or streaming data workflows.
- Experience working with Databricks‑based data platforms and supporting migration or transition of pipelines into a lakehouse‑oriented architecture.
- Strong familiarity with AWS cloud‑native data engineering, including services such as:
- Amazon S3
- AWS Glue
- AWS Lambda
- AWS Lake Formation
- Amazon Kinesis
- AWS IAM
- AWS‑native monitoring, logging, and security capabilities
- Working knowledge of technologies and concepts such as:
- Delta Lake
- Apache Iceberg
- Databricks Unity Catalog
- Metadata‑driven pipelines
- Data lake governance
- Lakehouse architecture
- Catalog‑driven processing
- Experience supporting data integration patterns involving mastering, MDM, reference data, market data, investment data, risk data or trusted data distribution workflows.
- Solid understanding of data quality, schema management, lineage, metadata, access control, encryption, and operational support for production data pipelines.
- Experience with AWS IAM, data lake governance, policy‑based access controls, and secure platform operations.
- Familiarity with engineering best practices such as version control, automated testing, CI/CD, infrastructure automation, release management, and monitoring.
- Experience implementing data platform observability and reliability practices, including alerting, monitoring, telemetry, dashboards, and production support procedures.
AI Tooling and LLM‑Enabled Engineering
- Practical experience using AI tooling such as Claude, GitHub CoPilot, LLMs, or similar AI‑assisted development platforms in an engineering context.
- Experience applying LLMs to data engineering or software engineering workflows, including code generation, refactoring, test creation, documentation, debugging, troubleshooting and pipeline analysis.
- Understanding of prompt engineering and context engineering, including how to structure technical context, examples, constraints, schemas, logs, coding standards, and acceptance criteria to improve AI‑generated outputs.
- Ability to use AI‑assisted workflows to support Databricks development, AWS data engineering, SQL, PySpark, pipeline migration, data quality analysis, and operational support activities.
- Ability to evaluate, validate, and refine AI‑generated code or recommendations before applying them to production‑grade engineering work.
- Understanding of responsible AI usage in enterprise environments, including security, privacy, compliance, human review, and protection of sensitive or regulated information.
Collaboration and Execution
- Strong communication and stakeholder management skills, with the ability to work effectively across offshore and global teams.
- Ability to translate architectural direction into actionable technical deliverables, implementation plans, and reusable engineering patterns.
- Comfortable operating in a role that blends hands‑on engineering, technical guidance, and delivery enablement.
- Strong problem‑solving skills and ability to navigate ambiguity in complex enterprise data platform environments.
- Ability to partner with engineers, architects, governance stakeholders