Mathematics - Optimization

Thu, 27 Jun 2024 23:14:15 +0800

Branches of Optimization Research #

Convex Optimization #

Convex optimization focuses on problems where the objective function and constraints are convex, ensuring a single global optimum. This field is foundational in machine learning, signal processing, and control systems due to its guaranteed convergence and efficient algorithms.

Convex Optimization by Boyd and Vandenberghe - PDF
Convex Optimization Theory by Dimitri P. Bertsekas - PDF

Discrete, Combinatorial, and Integer Optimization #

This branch deals with optimization problems involving discrete variables, such as integers or combinatorial structures, often encountered in scheduling, network design, and logistics. Bayesian optimization, a subset, is particularly useful for optimizing expensive black-box functions.

Bayesian Optimization In Action by Quan Nguyen - Amazon
Experimentation for Engineers by David Sweet - Amazon

Operations Research #

Operations research applies mathematical modeling and optimization to complex decision-making in logistics, supply chain, and resource allocation. It integrates techniques like linear programming, simulation, and heuristic methods to optimize real-world systems.

Operations Research An Introduction by Hamdy A. Taha - Pearson
Introduction to Operations Research by Frederick Hillier and Gerald Lieberman - McGraw Hill
Julia Programming for Operations Research by Changhyun Kwon - PDF - code
Mathematical Programming and Operations Research: Modeling, Algorithms, and Complexity. Examples in Python and Julia. Edited by Robert Hildebrand - PDF
A First Course in Linear Optimization by Jon Lee - PDF
Decomposition Techniques in Mathematical Programming by Conejo , Castillo , Mínguez , and García-Bertrand - Springer
Algorithms for Optimization by Mykel J. Kochenderfer and Tim A. Wheeler - PDF
Model Building in Mathematical Programming - Introductory modeling book by H. Paul Williams - Wiley

Meta-heuristics #

Meta-heuristics are high-level strategies for solving complex optimization problems where exact methods are computationally infeasible. They include nature-inspired algorithms like genetic algorithms and simulated annealing, widely used in engineering and data science.

Metaheuristics by Patrick Siarry - Springer (open access)
Essentials of Metaheuristics by Sean Luke - link
Handbook of Metaheuristics by Michel Gendreau and Jean-Yves Potvin - Springer (open access)
An Introduction to Metaheuristics for Optimization by Bastien Chopard , Marco Tomassini - Springer (open access)
Metaheuristic and Evolutionary Computation: Algorithms and Applications by Hasmat Malik, Atif Iqbal, Puneet Joshi, Sanjay Agrawal, and Farhad Ilahi Bakhsh - Springer (open access)
Clever Algorithms: Nature-Inspired Programming Recipes by Jason Brownlee - GitHub
Metaheuristics: from design to implementation by El-Ghazali Talbi - Wiley

Dynamic Programming and Reinforcement Learning #

Dynamic programming and reinforcement learning address sequential decision-making problems, breaking them into subproblems or learning optimal policies through interaction with environments. These methods are critical in robotics, finance, and AI.

Various tiltes on Dynamic Programming, Optimal Control and Reinforcement Learning by Dimitri Bertsekas. - List
Reinforcement Learning: An Introduction (2nd Edition) by Richard Sutton and Andrew Barto - PDF
Decision Making Under Uncertainty: Theory and Application by Mykel J. Kochenderfer - PDF
Algorithms for Decision Making by Mykel J. Kochenderfer, Tim A. Wheeler, and Kyle H. Wray - PDF

Constraint Programming #

Constraint programming solves problems by defining constraints that must be satisfied, often used in scheduling, planning, and configuration tasks. It excels in problems with complex logical constraints and discrete variables.

Handbook of Constraint Programming by Francesca Rossi, Peter van Beek and Toby Walsh - Amazon
A Tutorial on Constraint Programming by Barbara M. Smith (University of Leeds) - PDF

Combinatorial Optimization #

Combinatorial optimization focuses on finding optimal solutions in discrete structures, such as graphs or sets, often using algorithms for problems like the traveling salesman or graph coloring, with applications in logistics and network design.

Combinatorial Optimization: Algorithms and Complexity by by Christos H. Papadimitriou and Kenneth Steiglitz - Amazon
Combinatorial Optimization: Theory and Algorithms by Bernhard Korte and Jens Vygen - Springer
A First Course in Combinatorial Optimization by Jon Lee - Amazon

Stochastic Optimization and Control #

Stochastic optimization handles problems with uncertainty or randomness, using probabilistic models to optimize objectives. It is widely applied in machine learning, finance, and operations research for robust decision-making.

Lectures on Stochastic Programming Modeling and Theory (SIAM) - by Shapiro, Dentcheva, and Ruszczynski - PDF
Introductory Lectures on Stochastic Optimization by John C. Duchi - PDF

Useful Resources #

Prof. Nguyen Mau Nam, Convex Analysis - An introduction to convexity and nonsmooth analysis
Ben Recht, arg min
Prof. Dimitri P. Bertsekas, Convex Analysis and Optimization
Prof. Dimitri P. Bertsekas, Nonlinear Programming: 3rd Edition
Off the convex path

Discrete, Combinatorial, and Integer Optimization on Nam Le