Databricks Architect

Tiger Analytics Inc.; United States; Remote; USD 150,000 - 220,000; Full time

Job summary

Tiger Analytics is seeking a Databricks Architect to design and deploy scalable data solutions on the Databricks platform for enterprise clients. The role combines architecture, hands-on development, and governance across AWS services.

You will lead data engineering efforts, optimize Spark pipelines, and guide clients on best practices for secure, cost-efficient data platforms while working in a fast-growing, entrepreneurial environment.

Qualifications

• 12+ years of experience in architecture and data engineering of large-scale data ecosystems.
• Databricks design and architecture focusing on scalable, secure solutions.
• Hands-on Databricks/DataOps architecture to improve performance and monitoring.
• Good exposure to AWS and leadership skills; able to work independently.
• Experience with Hadoop environments and Spark streaming pipelines.

Responsibilities

• Design and implement scalable end-to-end data solutions on Databricks for ETL, streaming, and ML workloads.
• Provide technical leadership on Databricks adoption and reference architectures.
• Tune Spark jobs and optimize data pipelines for performance and cost efficiency.
• Architect, develop, and debug PySpark jobs; integrate Databricks with AWS services.
• Implement data governance, security, and access controls in the Databricks environment.
• Collaborate with clients to translate business problems into data engineering solutions.
• Drive performance improvement initiatives and monitoring tooling.

Skills

Databricks

PySpark

AWS

Data governance

Security

Leadership

Performance tuning

Big data

Tools

Unity Catalog

EMR

Lambda

S3

Hadoop

Job description

Tiger Analytics is pioneering what AI and analytics can do to solve some of the toughest problems faced by organizations globally. We develop bespoke solutions powered by data and technology for several Fortune 100 companies. We have offices in multiple cities across the US, UK, India, and Singapore, and a substantial remote global workforce.

If you have a passion for tackling business challenges through the use of both structured and unstructured data at scale, Tiger Analytics wants to connect with you. We are looking for a skilled and proactive Databricks Architect to take a pivotal role in designing and deploying resilient data solutions that address our clients' complex business issues.

Responsibilities

• Design and implement scalable, end-to-end data solutions for ETL, real-time streaming, and machine learning workloads on the Databricks platform

• Provide technical leadership and strategic guidance on Databricks adoption, including developing reference architectures and best practices.

• Ensure optimal performance by tuning Spark jobs, optimizing data pipelines, and resolving complex technical issues.

• Architect, develop, and debug PySpark jobs and data models, integrating Databricks with cloud services such as AWS S3, EMR, and Lambda

• Implement and enforce data governance, security, and access control policies (e.g., encryption, auditing) on the Databricks environment

• 12+ years of experience in Architecture, design and data engineering development of large-scale data ecosystems.

• Databricks Design and Architecture: Design and architect scalable and secure Databricks solutions platform, optimizing performance, cost, and reliability.

• Hands on Databricks/Dataops Architect that can help reduce consumption, and improve performance issues. Can help build monitoring tools.

• Good exposure to AWS. Strong comm and leadership skills and Self driven and work independently.

• Experience in Performance initiatives and drive recommendations (identify tuning opportunities, creation of performance environment etc.

• Experience with hadoop environment.

• Core Skills:-Databricks platform knowledge, Unity Catalog migration implementation exposure, Spark Streaming pipeline design exposure

Significant career development opportunities exist as the company grows. The position offers a unique opportunity to be part of a small, fast-growing, challenging, and entrepreneurial environment, with a high degree of individual responsibility.

Tiger Analytics provides equal employment opportunities to applicants and employees without regard to race, color, religion, age, sex, sexual orientation, gender identity/expression, pregnancy,
national origin, ancestry, marital status, protected veteran status, disability
status, or any other basis as protected by federal, state, or local law.