Five ways to combine Mathematical Optimization and Machine Learning
Practical examples of combining two forces.
Are you a data scientist? And do you know how to use mathematical optimization and machine learning? Did you ever combine them in a project? This post is written to give you some inspiration, and to encourage you to try it the next time you encounter a problem that can use them both. For every data scientist, it’s important to know at least a little bit about mathematical optimization. Because you shouldn’t create a complex machine learning solution if the actual problem is a perfect optimization use case.
Mathematical Optimization & Machine Learning
Mathematical optimization (or mathematical programming) is a powerful decision making tool. By formulating the goals in the objective and specifying constraints and variables, mathematical optimization can help making the best decisions given the current real-world circumstances. It has proven its value in many different industries, like airlines, logistics, electrical power and finance. If you’re dealing with a problem that has a clear goal and constraints, mathematical optimization might be a good fit.
Machine learning is a subfield of artificial intelligence. A computer can recognize patterns in data and learns to predict the future, can make clusters, detects anomalies or generate new music or images. The possibilities are endless. The three types of machine learning, (supervised, unsupervised and reinforcement learning) can be applied in all industries, like in healthcare and even in art. Machine learning models are all about probabilities and predict the probability of what will happen.
Both methods have their advantages and disadvantages. Machine learning models become useless when the data changes too much, and models need to be retrained or rebuild from scratch. Mathematical optimization needs clear instructions and a good mathematical description. It can’t handle unstructured data like machine learning can. Besides, if the problem becomes too large, you might need a commercial solver to solve the problem, this can be quite expensive.
Some problems are more suitable for machine learning, and in other cases mathematical optimization is a better fit. When you want to discover patterns in the data, find similar data samples or predict the weather, you should use machine learning. If you want to create a schedule, find the optimal placement of facilities or minimize costs for a problem, mathematical optimization is the better choice. It’s ineffective when the wrong method is applied, and it can save you a lot of time and frustration starting with the right one.
How to combine ML & MO?
It can be useful to combine mathematical optimization and machine learning. They have different strengths and weaknesses, and some problems are too complex to use just one of the two. They can complement each other. Here are five ways with practical examples to show you how you can combine them.
1. Use machine learning predictions as constraints in an optimization model
This is the most easy and straight forward way to combine ML and MO. First you predict something using machine learning, and these predictions are used as input for an optimization problem. You can set constraints using the output of the machine learning model.
Example: Predict viewing figures with ML, use them as input to create an optimal schedule with optimizationYou are a data scientist at a broadcasting company, and you sell advertising space to companies. Advertisers buy air time and the broadcasting company makes an educated guess about how many people will see the commercial of the advertiser. You, as data scientist, want to use the advertising space in the best way possible. You want to give the advertisers the exact amount of viewers, so you don’t waste airtime you could’ve sold to someone else. First, you predict the viewing figures based on historical data using machine learning. Then, you create an optimization model that uses the viewing figures as inputs. You optimize the schedule by giving every advertiser the exact amount of viewing figures they want. By doing this, you maximize profit for the company.
2. Use optimization decisions as training features in a machine learning model
This is the other way around compared to method 1: First the optimization model makes decisions, and the decisions are used as features in a machine learning model. In reality, this method is less common, because mostly the decisions (MO) follow the predictions (ML). But it is possible and might be useful in specific projects.
Example: Use transportation decisions in a machine learning modelMathematical optimization is widely used in logistics. It can save a lot of time, money and resources if you use optimization to decide how many supplies you need to ship from production plants to markets. The results of the model contain the shipment quantities between plants and markets. After you have obtained these results you can use them in a machine learning problem, for example to predict how many employees you need in every plant on a given day.
3. Use machine learning output to determine the scope of the mathematical optimization model
Besides using machine learning output directly in an optimization problem, you can choose to combine them more separately. You use them in the same project, but not in the same process. You can use machine learning output to make the mathematical optimization problem smaller: you determine the scope of an optimization model with machine learning. An extra benefit here is that the optimization model can be solved in a shorter amount of time.
Example: Use predictive maintenance and clustering to determine the scope of routing problemsFor this example, we look at a company that fixes power boxes. They have a limited number of repairmen available and want to use them in the best way possible. First, you can use predictive maintenance (machine learning) to decide which power boxes have the highest risk of failure. Then, you use clustering (machine learning) to cluster the high risk power boxes. You cluster because you want groups of power boxes close to each other. You can choose the number of clusters equal to the number of repairmen available. Finally, with mathematical optimization, you create optimal routes between the power boxes of every cluster, one route for every repairman.
4. Use optimization to solve machine learning research problems
This is a research area where optimization is used to help solve challenges in machine learning. You can use optimization to find an optimal set of parameters for a machine learning problem. ML and MO are tightly integrated here, because you use optimization inside the ML problem. The example below shows that mixed integer programming (MIP) has proven its value in solving a classical research problem.
Example: Solving the best subset selection problem in linear regressionWhen building a regression model, removing irrelevant features will make the model easier to interpret and less prone to overfit the data. It’s hard to find the optimal subset of features, known as the best subset selection problem. Mixed integer programming (mathematical optimization) has significantly improved in speed for the last decades, this makes it useful to test it on existing problems. It was tested on this problem, with great results.
5. Let a machine learning model learn from many similar optimization problems
The final way you can use machine learning and optimization together is the following: If you have an optimization model that solves the same type of problem over and over again, it’s possible to let machine learning help you. By storing the data and solutions every time the optimization model runs, you build up an interesting database for applied machine learning. This paper suggests different ways to use machine learning. It’s possible to predict the optimal objective value, to predict values for the decision variables (see example), or to accelerate the performance of an algorithm. This combination of machine learning and mathematical optimization is still an active research area.
Example: Predicting the optimal transshipment of blood units in a network of hospitalsFor hospitals, it’s really important to have good inventory levels of blood. Every day, blood is transported between hospitals close to each other to keep the levels high enough. The variables of this problem are about shipping instructions: Shipping a certain amount of a certain blood type between hospitals. In mathematical optimization, this problem can be solved faster if you provide the model with good initial values of the decision variables. This is called a warm start. It’s not guaranteed (at all) that the final solution will use these values for the variables. Still everything is possible, it’s just an okay first solution. If you store data together with the optimal solutions, which are found using mathematical optimization, a machine learning model can try to find patterns between the data and the target. The model will try to predict the optimal values of the decision variables. If the machine learning model finds good values, this can speed up the optimization process. In a perfect world, it would be possible to skip the optimization part completely. The results of this study are promising. The machine learning model that predicted the blood transportation case had up to 98% of similarity between the decision variables, and the constraints were almost always enforced.
Conclusion
Hopefully, this article inspires you to try the interesting combination of mathematical optimization and machine learning! They are suitable for different types of problems and can complement each other. Besides the obvious ways (use the output from one as the input to the other), there are other ways you can combine them. You can loosely combine them, like with the third method, where you use machine learning to determine the scope of the optimization problem. Or you can tightly integrate them to solve research problems, like the fourth example shows.
For any data scientist, it’s beneficial to know at least a little bit about mathematical optimization. Here are some related articles to get you started:
Why Every Data Scientist Should Learn Mathematical Optimization
Share This Article
Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.
Write for TDS
