We are seeking an autonomous, self-starting data engineering professional with exceptional English communication skills to help us complete a conversion to the Azure Databricks Lakehouse environment for AI-enabled Business Intelligence, by designing, building, and optimizing enterprise data ingestion and integration pipelines.
The successful candidate will be responsible for establishing connections to and extracting data from a diverse ecosystem of on-premise and cloud-based enterprise platforms, including SAP S/4HANA, SAP BW, SAP SuccessFactors, Oracle, SQL Server, and various other external systems and applications.
The role will involve ingesting a broad range of structured and unstructured source data, including raw database tables, application data, documents, files and images, into our Azure Databricks Lakehouse, primarily within the Bronze and Silver layers.
Beyond core ETL/ELT engineering, the candidate must possess strong expertise in implementing secure API integrations (e.g., OData and REST) and industry-standard security and authentication protocols. The candidate will be expected to work with a variety of connectivity mechanisms and source technologies to establish reliable, secure, and maintainable ingestion processes across both cloud and on-premises environments.
Key Responsibilities:
- Data Extraction & Ingestion: Design, implement, and maintain scalable pipelines to ingest and synchronize data from a wide range of enterprise transactional systems, databases, applications, file repositories, and other data sources into Azure Databricks.
- Enterprise Application Connectivity: Establish and maintain secure connections to on-premises and cloud-based applications and platforms, using appropriate technologies such as APIs, database connectivity, file transfer mechanisms, and other application-specific integration methods.
- API Integration & Security: Architect and implement robust, secure API-driven data integrations, including OData, REST and SOAP endpoints, adhering to enterprise security protocols, authentication mechanisms, network requirements, and credential management standards.
- Lakehouse Ingestion & Processing: Build reliable and performant ingestion and processing pipelines within the Databricks medallion architecture, with particular focus on landing raw source data in Bronze and creating cleaned, standardized and reusable data in Silver, using Spark Declarative Pipelines and Unity Catalog.
- Structured & Unstructured Data: Develop ingestion solutions for a range of data types, including relational database tables, application data, flat files, documents and images, ensuring appropriate metadata, lineage, security and traceability are maintained.
- Collaboration & Delivery: Work closely with cross-functional architecture, infrastructure, security and analytics stakeholders to deliver production-grade data integrations and assets with minimal oversight.
Required Skills & Qualifications:
- Core Data Engineering: Proven experience engineering cloud-based data platforms and ingestion pipelines using Azure Databricks, SQL, PySpark, Delta Lake, and Unity Catalog.
- API & Integration Expertise: Expert-level experience implementing APIs and other integration mechanisms to connect with external systems, including technologies such as OData, REST and SOAP, alongside strong familiarity with enterprise security protocols, encryption, authentication and authorization standards.
- Cloud & On-Premises Integration: Demonstrated experience establishing secure data connectivity between on-premises systems, cloud applications and cloud data platforms, including understanding of common network, firewall, proxy, authentication and credential-management considerations.
- Heterogeneous Source Systems: Demonstrated experience extracting data from diverse enterprise databases and platforms, including Oracle, SQL Server, file-based sources and legacy/modern transactional environments.
- SAP Platform Experience: Hands-on integration experience extracting data from SAP S/4HANA and SAP SuccessFactors.
- Experience working with SAP BDC / SAP BW, including knowledge of SAP data structures and ABAP, is considered an advantage.
- Experience ingesting and managing unstructured or semi-structured data, including documents, images, JSON, XML and other file-based content.
- Understanding of and experience with streaming architectures, including use of Kafka and MQTT in combination with Spark, is considered an advantage.
- Databricks Lakehouse: Strong practical understanding of the Databricks medallion architecture, particularly the design and implementation of Bronze and Silver ingestion and transformation pipelines.
- Autonomy: A proven \"self-starter\" mindset capable of taking ownership of end-to-end data integration initiatives with minimal supervision.
- Communication: Excellent written and verbal English communication skills, with the ability to document technical designs and articulate architectural decisions clearly to technical and non-technical stakeholders.