reinforce learning

actor-critic

origin

• $s_t$: state at time t
• $V(s_t)$: value function predict reward with $s_t$
• $A(s_t)$: action function return action probability base on $s_t$
• top1: select the action with max probability
• $R(a_{[0:t]})$: reward function input a sequence of action out float
• advantages value: if the can get more reward then positive value else negative value

$$\begin{array}{rl}\text{action probability}& =A(s_{[0:t-1]})\\ a_{[0:t-1]}& =\text{top1}(\text{action probability})\\ \text{reward}& =R(a_{[0:t-1]})\\ L_{\text{critic}}& =MSE(reward,V(s_{t-1}))\\ \text{advantages value}& =\text{reward}-V(s_{t-1})\\ L_{\text{actor}}& =\text{CrossEntropy}(\text{action probability},a_{[0:t-1]})\times \text{advantages value}\end{array}$$

with Temporal Difference error(TD-error)

• Hope $V(s_t)$ can predict reward that may get in future with proportion $\gamma$
• note: you should add stop gradient to $T{D}_{\text{target}}$ (aka detach in pytorch)

$$\begin{array}{rl}T{D}_{\text{target}}& =\text{reward}+\gamma V(s_t)\\ T{D}_{\text{error}}& =T{D}_{\text{target}}-V(s_{t-1})\\ L_{\text{critic}}& =MSE(T{D}_{\text{error}},0)=MSE(T{D}_{\text{target}},V(s_{t-1}))\\ \text{advantages value}& =T{D}_{\text{error}}\\ L_{\text{actor}}& =\text{CrossEntropy}(\text{action probability},a_{[0:t-1]})\times \text{advantages value}\end{array}$$