Is deep learning really going to be able to do everything? 

Opinions on deep learning’s true potential vary. Geoffrey Hinton, awarded for pioneering deep learning, is not entirely unbiased,  but others, including Hinton’s deep learning collaborator Yoshua Bengio, are looking to infuse deep learning with elements of a domain still under the radar: operations research, or an analytical method of problem-solving and decision-making used in the management of organizations.

Machine learning and its deep learning variety are practically household names now. There is a lot of hype around deep learning, as well as a growing number of applications using it. However, its limitations are also becoming better understood. Presumably, that’s the reason Bengio turned his attention to operations research.

In 2020, Bengio and his collaborators surveyed recent attempts, both from the machine learning and operations research communities, to leverage machine learning to solve combinatorial optimization problems. They advocate for pushing further the integration of machine learning and combinatorial optimization and detail a methodology.

Until now, however, there was no publicly visible operations research renaissance to speak of and commercial applications remain few compared to machine learning. 

Nikolaj van Omme and Funartech want to change that.

 

Operations research leverages domain knowledge to optimize

While the birth of operations research (OR) is usually identified as occurring during WWII, its mathematical roots may go back even further to the 19th century. 

In OR, problems are broken down into basic components and then solved in defined steps by mathematical analysis. Van Omme self-identifies as a mathematician, as well as a computer scientist. After his postgraduate studies, he started noticing the similarity and complementarity between machine learning and OR. After failing to get the attention he was looking for in order to pursue the exploration of this potential synergy, in 2017 he launched Funartech to make it happen himself.

For van Omme, there were several reasons why combining machine learning and OR seemed like a good idea. First, machine learning is data-hungry and in the real world, there are cases in which there is not enough data to go by.

It’s also a matter of philosophy: “If you are only using data, you’re hoping your algorithms will get some patterns out of the data,” van Omme said. “You’re hoping to find some constraints, some knowledge out of the data. But actually, you’re not sure you will be able to do that.” 

In OR, he added, knowledge can be modeled. “You can talk to the engineers and they can tell you what they do, what they think and how they proceed,” he explained. “You can transform this into mathematical equations, so you can have that knowledge and use it. If you combine both data and domain knowledge, you’re able to go further.” 

OR is all about optimization and using it can result in 20% to 40% optimized results, according to van Omme. Like Bengio, he referred to the traveling salesman problem (TSP) – a reference problem in computer science. In TSP, the goal is to find the optimal route to visit all cities in a traveling salesman’s assigned district once.

If you approach the TSP with OR, it is possible to produce exact solutions for 100,000 cities, according to van Omme. By using machine learning, on the other hand, the best you can do for an exact solution is to solve the same problem with 100 cities. This is an order of magnitude of difference, so it begs the question: Why isn’t OR used more often? 

For van Omme, the answer is multifaceted: “Machine learning was considered a subfield of OR a few years ago, so I wouldn’t say that OR is not applied, although now people tend to put machine learning on one side and OR on the other,” he said. “There are some fields where OR is really used extensively –transportation, for instance, or manufacturing.” 

However, machine learning had so much success in some fields that it overshadowed all the other approaches, he explained. 

3 ways to combine operations research and machine learning

1. Van Omme is not out to bash machine learning. What he is advocating for is an approach that combines machine learning and OR, in order to have the best of both worlds. Usually, van Omme said, first you use machine learning so that you get some estimates and then you use those estimates as input for your OR algorithm to optimize.
2. Machine learning and OR can also be used in conjunction, to help the other. Machine learning can be used to improve OR algorithms and OR can be used to improve machine learning algorithms. OR is mainly rule-based and when the rules apply, that’s hard to beat, van Omme noted.
3. Construct new algorithms. If you understand fundamentally the strengths and weaknesses of machine learning and OR, there are ways to combine both so that one’s weaknesses are leveled by the other’s strengths. Van Omme mentioned graph neural networks as an example of this approach.

 

 

Drawbacks

OR is not without its issues and van Omme acknowledges that. The problem, in his words, is that “most of the time the rules don’t apply. You don’t know exactly how to apply them. And there is some probability that if you take one direction or another, you will get completely different outcomes.”

This cannot be approached in “traditional” ways with one model that can solve the whole problem, because it is a very complex problem at a massive scale, van Omme noted. After working on this for four months, Funartech was able to optimize by 53%. However, it turned out that they didn’t have the right data for some parts of the problem.

So, when Funartech tried to figure out whether their solution made sense or not, they quickly discovered that some estimations for the data they didn’t have were actually not very good. When the right data was provided, then the optimization dropped to 30%.

“The thing is, our algorithms are so tailored to the instance that when they gave us the right data, they stopped working,” he said. “They couldn’t produce anything. So, we had to backtrack and we had to simplify our approach a little bit. And because it was the end of the project, we didn’t want to invest as much time as we did.” 

 

Scaling operations research up

Van Omme also explained that Funartech spends a lot of time with customers, aiming to bring a tailored approach to each problem. This seems like a blessing and a curse at the same time. Even though van Omme mentioned Funartech is working on developing a platform, at this point it’s hard to imagine how this service-oriented approach could scale.

Part of what has made the machine learning approach succeed to the extent that it has is the fact that there are algorithms and platforms that people can use without having to develop everything from scratch. On the other hand, van Omme pointed out that Funartech has a 100% success rate, while 85% of machine learning and 87% of data science projects fail.

But there is another, perhaps unexpected, obstacle that OR practitioners have to deal with, according to van Omme: learning to get along with each other. The “no Ph.D. required to make this work” narrative has been an integral part of machine learning’s push to the mainstream. In OR, things are not there yet.

The fact that OR practitioners are highly skilled also means that they tend to be highly opinionated, according to van Omme. People skills, as in learning to listen and compromise, are therefore essential.

All in all, OR – and the various ways it can be combined with machine learning – seems like a double-edged sword. It has the potential to produce highly optimized results, but at this point, it also looks brittle, resource- and skills-intensive and difficult to apply. 

But then again, the same could probably be said about machine learning a few years ago. Perhaps cross-fertilizing the two disciplines with techniques and lessons learned could help lift both of them up.

 © 2022 LeackStat.com