Optimization cheat sheet

Introduction

Which optimization algorithm should I use? This might be the question in many applications or theoretical works. Here, I provide my insights as a flowchart.



Effective optimization is about a right balance between exploration and exploitation. Exploration is the act of searching for regions in the search space (candidate solutions) where potentially high-quality solutions can be found (i.e., global search). Exploitation, however, focuses on the local information.

I came up with the following guideline to choose an appropriate optimization method for different problems.

A: an Evolutionary Algorithm
B: Gradient descend
C: Use A for exploration and then B
D: Finite Difference method to calculate derivatives and use gradient descend
E: Use A for exploration and then D

A higher order of derivatives can be also helpful for better exploitation in combination with the gradient descent.

See also Evolutionary Algorithms.



Like always, this is only my opinion based on my experience and only a point of discussion. I would love to discuss this topic further in the comments.


By R. Bonyadi