Optimization is not just about finding a good answer.
It is about finding a better answer than the one you have now.
The hard part is knowing whether your current “best” is truly best, or just locally good.
Optimization is the process of improving a solution according to some objective.
You start with a candidate solution.
You measure how good it is.
Then you search for a better one.
That sounds simple.
But the search space can be huge.
And the best nearby solution may not be the best overall solution.
A simple optimization loop looks like this:
Candidate Solution → Evaluate Score → Explore Neighbor → Accept or Reject → Repeat
More compactly:
Optimization = search space + objective function + update strategy
The objective function tells you what “better” means.
The update strategy tells you how to move.
At a high level, optimization search works like this:
start with an initial solution
evaluate its score
while stopping condition is not met:
generate candidate solutions
compare scores
choose whether to move
update the current solution
return the best solution found
This structure appears in many algorithms.
Hill Climbing.
Simulated Annealing.
Genetic Algorithms.
Even mathematical optimization methods follow the same broad idea:
evaluate, improve, repeat.
Imagine tuning a route, schedule, or model setting.
You start with one possible solution.
Then you slightly modify it.
If the new solution improves the score, you keep it.
If not, you reject it.
That is the basic intuition behind Hill Climbing.
But there is a problem.
What if the current solution is better than all nearby options, but not the best possible solution overall?
That is the local optimum problem.
This is the first concept to understand.
A local optimum is the best solution in its nearby region.
A global optimum is the best solution across the entire search space.
Local optimum:
Global optimum:
If you do not understand this distinction, optimization algorithms feel like random strategy names.
Hill Climbing is the most intuitive optimization strategy.
It repeatedly moves to a better neighboring solution.
The idea is simple:
If the next move improves the score, take it.
If not, stop or try another neighbor.
Hill Climbing is easy to understand and implement.
But it has a major weakness.
It can get stuck at a local optimum.
That is why more advanced strategies exist.
Simulated Annealing extends Hill Climbing.
Hill Climbing usually accepts only better moves.
Simulated Annealing sometimes accepts worse moves.
That sounds strange at first.
But it helps the algorithm escape local optima.
Hill Climbing:
Simulated Annealing:
So the key difference is exploration.
Hill Climbing exploits.
Simulated Annealing explores before settling.
Genetic Algorithms take a different approach.
Instead of improving one solution, they work with a population of solutions.
The algorithm repeatedly selects, mixes, and mutates candidates.
A simple view:
Population → Selection → Crossover → Mutation → New Population
This is useful when the search space is complex.
Instead of betting on one path, the algorithm explores multiple directions at once.
That makes Genetic Algorithms very different from Hill Climbing.
Hill Climbing improves one candidate.
Genetic Algorithms evolve many candidates.
Newton-Raphson is more mathematical.
It uses information about the function’s slope and curvature.
When the function is smooth and the structure is known, this can move quickly toward a solution.
But it depends more heavily on mathematical assumptions.
Compared with search-based methods, Newton-Raphson is less like exploring a landscape blindly.
It is more like using the shape of the landscape to jump efficiently.
This comparison helps organize the field.
Search-based optimization:
Mathematical optimization:
So the right method depends on the problem.
If you know the structure, use it.
If you do not, search.
If optimization strategies feel scattered, learn them in this order:
This order works because you first understand the problem.
Then you learn the simplest strategy.
Then you learn how algorithms escape limitations.
Then you compare broader approaches.
Optimization is search under an objective.
The shortest version is:
Optimization = find better solutions according to a score
Hill Climbing moves toward local improvement.
Simulated Annealing sometimes accepts worse moves to escape traps.
Genetic Algorithms explore with many candidates.
Newton-Raphson uses mathematical structure to move faster.
If you remember one idea, remember this:
Every optimization method is a trade-off between exploitation, exploration, and assumptions about the problem.
When solving optimization problems, do you usually start with a simple greedy method like Hill Climbing, or jump directly to broader strategies like Simulated Annealing or Genetic Algorithms?
Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/optimization-and-search-strategies-hub-en/
GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai