This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Data Engineer (Databricks) – Contractor based in the United States.
As a Senior Data Engineer, you will play a key technical role in designing and delivering modern, scalable data platforms using Databricks and cloud-based lakehouse architectures. You will work with forward-thinking organizations to solve complex data challenges through advanced data engineering, distributed processing, and real-time data solutions. Collaborating closely with Solution Architects and cross-functional engineering teams, you will transform technical requirements into reliable and efficient data pipelines. You will also contribute to the development of reusable solution accelerators and innovative engineering practices within a specialized data engineering community. This remote contract opportunity offers the chance to work with modern data technologies, influence enterprise data architecture, and deliver measurable business impact.
Accountabilities
- Lead the development and implementation of advanced data engineering solutions on the Databricks platform, ensuring scalability, reliability, and performance across batch and streaming workloads.
- Work alongside Solution Architects and multidisciplinary teams to deliver client data platform components while contributing to technical innovation and shared engineering standards.
- Design, develop, and implement data solutions using Databricks and its core platform capabilities, including Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
- Build and optimize scalable data pipelines supporting batch, micro-batch, and streaming data ingestion and processing.
- Implement configuration-driven data pipelines using YAML and modern ingestion technologies such as Auto Loader.
- Develop robust data processing and transformation solutions using distributed computing frameworks and cloud lakehouse methodologies.
- Design and maintain complex job orchestration and workflow patterns using Databricks Jobs and Workflows or tools such as Apache Airflow and dbt.
- Implement data quality, pipeline auditing, monitoring, and observability solutions using native Databricks capabilities and frameworks such as Great Expectations, Collibra, and dbt.
- Collaborate with Solution Architects to translate architectural designs into technical components and deliver high-quality implementations.
- Participate in technical design sessions, break down complex development requirements into manageable tasks, and deliver assigned work within agreed timelines.
- Contribute to Databricks Center of Excellence initiatives by developing reusable solution accelerators and promoting innovative engineering practices.
- Explore emerging technologies, new Databricks features, and modern data engineering approaches to continuously improve platform capabilities.
- Share technical knowledge, insights, and best practices with engineering colleagues and the broader technical community.
Requirements
The ideal candidate is an experienced data engineering professional with strong hands-on Databricks expertise, a solid understanding of modern cloud data architectures, and the ability to independently deliver complex technical solutions. You should combine advanced programming and distributed processing skills with a proactive mindset, strong problem-solving abilities, and a commitment to continuous learning.
- At least 2 years of hands-on experience designing and developing data pipelines using Databricks.
- Strong practical knowledge of Databricks platform features, including Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
- Proven experience building robust, scalable, YAML configuration-driven data pipelines for batch, micro-batch, and streaming data processing.
- Experience with data ingestion technologies such as Databricks Auto Loader.
- Strong understanding of workflow orchestration and experience with Databricks Jobs and Workflows or alternative tools such as Apache Airflow and dbt.
- Experience implementing data quality, pipeline auditing, monitoring, and observability solutions using Databricks-native features or tools such as Great Expectations, Collibra, and dbt.
- Proficiency with big data technologies and distributed processing platforms, including Apache Spark, Presto, and Amazon EMR.
- Experience with cloud data warehouses such as Amazon Redshift, Snowflake, and Google BigQuery.
- Strong programming skills in SQL, stored procedures, and object-oriented languages such as Java and Python, including PySpark.
- Familiarity with DevOps practices and CI/CD pipeline development, particularly using Databricks Asset Bundles with automated testing and validation.
- Understanding of Infrastructure as Code (IaC) tools such as Terraform is a plus.
- Familiarity with NoSQL databases and container management systems.
- Exposure to AI and machine learning technologies such as MLflow, prompt engineering, and modern AI-agent workflows.
- Ability to work independently with minimal supervision while collaborating effectively with architects and cross-functional engineering teams.
- Strong analytical thinking, problem-solving skills, and the ability to break down complex technical challenges into practical solutions.
- Curiosity, adaptability, and enthusiasm for exploring emerging data technologies and sharing knowledge with others.
Preferred Qualifications
- Databricks Data Engineer Associate and/or Professional certification.
- Experience delivering multiple successful Databricks implementation projects.
- Previous involvement in developing reusable data engineering frameworks, accelerators, or enterprise-scale data platform solutions.
Benefits
- Remote work: Fully remote contract opportunity available to professionals based in the United States.
- Contract duration: Initial six-month engagement, with the possibility of extension.
- Flexible start date: Expected start in December or January.
- Technical innovation: Opportunity to work with modern Databricks technologies and advanced cloud data lakehouse architectures.
- Professional development: Exposure to complex enterprise data challenges and opportunities to expand expertise across data engineering, AI, and distributed processing.
- Collaborative environment: Work alongside experienced engineers, Solution Architects, and cross-functional technical teams.