This presentation introduces Tune and Fugue, frameworks for intuitive and scalable hyperparameter optimization (HPO). Tune supports both non-iterative and iterative HPO problems. For non-iterative problems, Tune supports grid search, random search, and Bayesian optimization. For iterative problems, Tune generalizes algorithms like Hyperband and Asynchronous Successive Halving. Tune allows tuning models both locally and in a distributed manner without code changes. The presentation demonstrates Tune's capabilities through examples tuning Scikit-Learn and Keras models. The goal of Tune and Fugue is to make HPO development easy, testable, and scalable.

1. Intuitive & Scalable HPO
With Spark+Fugue
Han Wang

2. Agenda
Introduction
Non-Iterative HPO
Demo
Iterative HPO Demo

3. pip install tune
https://github.com/fugue-project/tune
pip install fugue
https://github.com/fugue-project/fugue

4. Introduction

5. Questions
● Is parameter tuning a machine learning problem?
● Are there common ways to tune both classical models and deep
learning models?
● Why is it so hard to do distributed parameter tuning?

6. Tuning Problems In General
General Parameter Tuning
Hyperparameter Tuning (for Machine Learning)
Some Classical
Models
Deep Learning Models
Some Classical
Models
Non-Iterative Problems Iterative Problems

7. Distributed Parameter Tuning
● Not everything can be parallelized
● The tuning logic is always complex and tedious
● Popular tuning frameworks are not distributed environment
friendly
● Spark is not suitable for iterative tuning problems

8. Distributed Parameter Tuning
Tune SQL Validation

9. Our Goals
● For non-iterative problems:
○ Unify grid and random search, make other plugable
● For iterative problems:
○ Generalize SOTA algos such as Hyperband and ASHA
● For both
○ Tune both locally and distributedly without code change
○ Make tuning development iterable and testable
○ Minimize moving parts
○ Minimize interfaces

10. Non-Iterative Problems

11. Grid Search
a: Grid(0,1)
b: Grid(“a”, “b”)
c: 3.14
a:0, b:”a”, c:3.14
a:0, b:”b”, c:3.14
a:1, b:”a”, c:3.14
a:1, b:”b”, c:3.14
Search Space Candidates
Pros: determinism, even coverage, interpretable
Cons: complexity can increase exponentially

12. Random Search
a: Rand(0,1)
b: Choice(“a”,“b”)
c: 3.14
a:0.12, b:”a”, c:3.14
a:0.66, b:”a”, c:3.14
a:0.32, b:”b”, c:3.14
a:0.94, b:”a”, c:3.14
Search Space Candidates
Pros: complexity and distribution are controlled, good for continuous variables
Cons: by luck, not deterministic, large number of samples are normally needed

13. Bayesian Optimization
objective: a^2
a: Rand(-1,1)
-0.66 -> 0.76 -> -0.18
-> 0.75 -> 0.90
-> 0.07 -> 0.00
-> 0.41 -> 0.12 -> 0.66
Search Space Candidates
Pros: less compute to guess the optimal parameters
Cons: sequential operations may require more time

14. Hybrid Search Space
Distributed Hybrid Search
Model 1 Model 2
Grid Random Bayesian

15. Live Demo
Space Concept & Scikit-Learn Tuning

16. Iterative Problems

17. Challenges
● Realtime asynchronous communication
● The overhead for checkpointing iterations can be significant
● Single iterative problem can’t be parallelized
● A lot of boilerplate code

18. Successive Halving (SHA)
Rung 1
Rung 2
Rung 3
Rung 4

19. Fully Customized Successive Halving
8, [(4,6), (2,2), (6,1)]

20. Hyperband

21. Asynchronous Successive Halving (ASHA)

22. Live Demo
Keras Model Tuning

23. Summary
Space Monitoring
Dataset
Distributed
Execution
Abstraction
Non-Iterative
Random, Grid, BO
Iterative
SHA, HB, ASHA, PBT ...
Specialization
Scikit-Learn
Specialization
Keras, TF, PyTorch

24. Let’s Collaborate!
● Create specialized higher level APIs for major tuning cases so
users can do tuning with minimal code and without learning
distributed systems
● Enable advanced users to create fully customized, platform
agnostic and scale agnostic tuning pipelines with tune’s lower
level APIs

25. pip install tune
https://github.com/fugue-project/tune
pip install fugue
https://github.com/fugue-project/fugue

26. Feedback
Your feedback is important to us.
Don’t forget to rate and review the sessions.