> 1) With an on-policy algorithm we use the current policy (a regression model with weights W, and ε-greedy selection) to generate the next state's Q.

Yes. To avoid confusion, it may be better to use the terms "behaviour policy" for the the policy that controls current actions and "target policy" for the policy being evaluated and/or learned.

> 2) With an off-policy algorithm we use a greedy version of the current policy to generate the next state's Q.

Sort of. The only *requirement* for an algorithm to be *off-policy* is that the policy that the target policy is different to the behaviour policy. The usual target policy in Q-learning is not necessarily a greedy version of the behaviour policy, but is the maximising policy over Q. However, if the behaviour policy is ε-greedy over Q, and adapting to updates in Q, then yes your statement holds.

> 3) If an exploration constant ε is set to 0, then the off-policy method becomes on-policy, since Q is derived using the same greedy policy.

This is true when comparing SARSA with Q learning, but may not hold when looking at other algorithms. This greedy-only action selection would not be a very efficient learner in all environments.

> 4) However, on-policy method uses one sample to update the policy, and this sample comes from on-line world exploration since we need to know exactly which actions the policy generates in current and next states. While off-policy method may use experience replay of past trajectories (generated by different policies) to use a distribution of inputs and outputs to the policy model.

Experience replay is about online vs offline methods, and not *directly* related to on-policy vs off-policy learning.

However, you will see experience replay used more often with off-policy methods, since off-policy learners that boot-strap (i.e. use Q value of next state/action to help estimate current Q value) are less stable when used with function approximators. Experience replay helps to address that problem.