四、MC ε-Greedy（MC without exploring starts ）1、Soft Policies（软策略）什么是Soft policies？如果一个策略采取的每一个action的概率是positive，也就是这些action都有可能被采取，那么这个策略就称为soft policy。为什么要引入软策略？使用软策略，一些足够长的回合就能够足够多次地访问每个状态-动作对。【soft policy，a few episodes that are sufficiently long can visit every state-action pair for sufficiently many times.】因此，我们不需要必须将每个state-action pair作为出发点进行始大量的episodes。因此，exploring starts的前提要求可以被removed。那么我们使用什么样的 soft policies 呢？答案是ϵ-greedy policies。2、ϵ\epsilonϵ-greedy policiesπ(a∣s)={ 1−ϵ∣A(s)∣(∣A(s)∣−1),forthegreedyaction,ϵ∣A(s)∣,fortheother∣A(s)∣−1actions,\left.\pi(a|s)=\left\{\begin{array}{ll}1-\dfrac{\epsilon}{|\mathcal{A}(s)|}(|\mathcal{A}(s)|-1),\text{for the greedy action,}\\\\\dfrac{\epsilon}{|\mathcal{A}(s)|},\text{for the other }|\mathcal{A}(s)|-1\text{ actions,}\end{array}\right.\right.π(a∣s)=⎩⎨⎧​1−∣A(s)∣ϵ​(∣A(s)∣−1),∣A(s)∣ϵ​,​forthegreedyaction,fortheother∣A(s)∣−1actions,​ε-Greedy algorithm在statesss，有一个greedy actiona∗a^*a∗，该action 对应的action valueqπ(s,a∗)q_π(s,a^*)qπ​(s,a∗)是最大的，这时，根据ε-Greedy算法：给greedy actiona∗a^*a∗的选择概率为1−ε∣A(s)∣(∣A(s)∣−1)1-\cfrac\varepsilon{|\mathcal{A}(s)|}(|\mathcal{A}(s)|-1)1−∣A(s)∣ε​(∣A(s)∣−1)；给其他不是greedy actiona∗a^*