Publications

2026

1. TMLR

   Computing Monetary Risk Measures in Linear Time

   Gersi Doko^*, Palash Agrawal^*, Maeve Burwell, and 1 more author

   Transactions on Machine Learning Research, Oct 2026

   Abs arXiv Bib HTML PDF Code

   Monetary risk measures have gained popularity for expressing decision-makers’ risk aversion. Value-at-Risk (VaR) and Conditional-Value-at-Risk (CVaR), in particular, are used commonly for this purpose. This paper proposes new efficient algorithms to compute these risk measures for a discrete random variable in expected linear time with respect to the size of its domain. First, we propose QuickVaR, an algorithm that computes the VaR of a discrete random variable. Then, we leverage QuickVaR to propose QuickDivergence, an algorithm for computing a class of φ-divergence risk measures, including the popular CVaR risk measure. The QuickVaR algorithm adapts the well-known quickselect algorithm, while QuickDivergence builds on polymatroid optimization algorithms. Numerical results show that our new algorithms offer speedups of up to nearly an order of magnitude for large domains. A library implementation is available at https://github.com/RiskAverseRL/RiskMeasures.jl.

   @article{doko2026monetary,
     title = {Computing Monetary Risk Measures in Linear Time},
     author = {Doko, Gersi and Agrawal, Palash and Burwell, Maeve and Petrik, Marek},
     journal = {Transactions on Machine Learning Research},
     issn = {2835-8856},
     year = {2026},
     month = oct,
   }

2025

1. NeurIPS

   Risk-Averse Total-Reward Reinforcement Learning

   Xihong Su, Jia Lin Hau, Gersi Doko, and 2 more authors

   In Advances in Neural Information Processing Systems, Oct 2025

   Abs arXiv Bib HTML PDF

   Risk-averse total-reward Markov Decision Processes (MDPs) offer a promising framework for modeling and solving undiscounted infinite-horizon objectives. Existing model-based algorithms for risk measures like the entropic risk measure (ERM) and entropic value-at-risk (EVaR) are effective in small problems, but require full access to transition probabilities. We propose a Q-learning algorithm to compute the optimal stationary policy for total-reward ERM and EVaR objectives with strong convergence and performance guarantees. The algorithm and its optimality are made possible by ERM’s dynamic consistency and elicitability. Our numerical results on tabular domains demonstrate quick and reliable convergence of the proposed Q-learning algorithm to the optimal risk-averse value function.

   @inproceedings{su2025riskaversetotalrewardreinforcementlearning,
     title = {Risk-Averse Total-Reward Reinforcement Learning},
     author = {Su, Xihong and Hau, Jia Lin and Doko, Gersi and Panaganti, Kishan and Petrik, Marek},
     booktitle = {Advances in Neural Information Processing Systems},
     year = {2025},
     volume = {38},
     pages = {93221--93249},
   }

2. RCR

   Deep Reinforcement Learning-Based Optimization of an Island Energy-Water Microgrid System

   Roozbeh Ghasemi, Gersi Doko, Marek Petrik, and 4 more authors

   Resources, Conservation and Recycling, Aug 2025

   Bib HTML PDF

   @article{deeprl-island,
     title = {Deep Reinforcement Learning-Based Optimization of an Island
                Energy-Water Microgrid System},
     author = {Ghasemi, Roozbeh and Doko, Gersi and Petrik, Marek and Wosnik, Martin and Lu, Zhongming and Foster, Diane and Mo, Weiwei},
     year = {2025},
     month = aug,
     journal = {Resources, Conservation and Recycling},
     publisher = {Elsevier},
     volume = {222},
     pages = {108440},
     doi = {10.1016/j.resconrec.2025.108440},
   }

2024

1. RLC

   ROIL: Robust Offline Imitation Learning

   Gersi Doko, Guang Yang, Daniel Brown, and 1 more author

   Reinforcement Learning Journal, May 2024

   Abs Bib PDF

   We study the problem of imitation learning via inverse reinforcement learning where the agent attempts to learn an expert’s policy from a dataset of collected state, action tuples. We derive a new Robust model-based Offline Imitation Learning method (ROIL) that mitigates covariate shift by avoiding estimating the expert’s occupancy frequency. Frequently in offline settings, there is insufficient data to reliably estimate the expert’s occupancy frequency and this leads to models that do not generalize well. Our proposed approach, ROIL, is a method that is guaranteed to recover the expert’s occupancy frequency and is efficiently solvable as an LP. We demonstrate ROIL’s ability to achieve minimal regret in large environments under covariate shift, such as when the state visitation frequency of the demonstrations does not come from the expert.

   @article{roil,
     title = {ROIL: Robust Offline Imitation Learning},
     author = {Doko, Gersi and Yang, Guang and Brown, Daniel and Petrik, Marek},
     year = {2024},
     month = may,
     institution = {RLC},
     journal = {Reinforcement Learning Journal},
     publisher = {RLJ},
     volume = {1},
     number = {1}
   }