TransWikia.com

Are tabular reinforcement learning methods obsolete (or getting obsolete)?

Artificial Intelligence Asked on November 12, 2021

While learning RL, I came across some problems where the Q-matrix that I need to make is very very large. I am not sure if it is ever practical. Then I research and came to this conclusion that using the tabular method is not the only way, in fact, it is a very less powerful tool as compared to other methods such as deep RL methods.

Am I correct in this understanding that with the increasing complexity of problems, tabular RL methods are getting obsolete?

One Answer

Am I correct in this understanding that with the increasing complexity of problems, tabular RL methods are getting obsolete?

Individual problems don't get any more complex, but the scope of solvable environments increases due to research and discovery of better or more apt methods.

Using deep RL methods with large neural nets can be a lot less efficient for solving simple problems. So tabular methods still have their place there.

Practically, if your state/action space (number of states times number of actions) is small enough to fit a Q table in memory, and it is possible to visit all relevant state/action pairs multiple times in a relatively short time, then tabular methods offer guarantees of convergence that approximate methods cannot. So tabular approaches are often preferred if they are appropriate.

Many interesting, cutting edge problems that are relevant to AI, such as autonomous robots acting in the real world, do not fit the tabular approach. In that sense, the approach is "obsolete" in that it no longer provides challenging research topics for practical AI (there are still unanswered theoretical questions, such as proof of convergence for Monte Carlo control).

It is still worth understanding tabular value-based methods in detail, because they form the foundations of the more complex deep learning methods. In some sense they represent ideal solutions that deep RL tries to approximate, and the design of tabular solutions can be the inspiration for changes and adjustments to neural-network methods.

Answered by Neil Slater on November 12, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP