An exponential lower-bound for the Policy iteration algorithm on discounted MDPs
The question of knowing whether the Policy Iteration algorithm (PI) for solving Markov Decision Processes (MDPs) has exponential or (strongly) polynomial complexity has attracted much attention in the last 50 years. Recently, an example on which PI requires an exponential number of iterations to converge was proposed for the total-cost and the average-cost criteria. On the other hand, it was shown that PI runs in strongly polynomial time on discounted-cost MDPs, yet only when the discount factor is fixed. In this work, we show that PI needs an exponential number of steps to converge on discounted-cost MDPs with a general discount factor. For that purpose, we show that it is always possible to find a discount factor such that PI behaves the same when applied either to the discounted or the undiscounted version of the exponential complexity example for PI in totalcost MDPs.
