# Representing similar states in reinforcement learning?

Let's say I'd like to design a Q learning algorithm that learns to play poker. The number of different possible States is very large, but a lot are very similar: for example, if the initial state having 10 spades, 4 hearts, 6 clubs on the table and holding King and Queen of hearts had already been visited, I would like it to affect the weights of similar states, like the same cards with different suits. How do I accomplish this?

• What you are asking about is not really transfer learning, but efficient/clever state representations. Although it's a blurred lines perhaps. Commented Aug 6, 2018 at 17:51
• this problem is better suited to combinatorics. There are only 52 cards in a deck, you know what hand beats what other hand, and you know what's left in the deck (or in other players' hands). Write a program to count cards. Not everything needs an ML solution Commented Sep 5, 2018 at 19:55
• Most poker games do not rank suits, thus there is no reason to track their value. Commented Jan 4, 2019 at 20:16

Define your biggest suits as suit1 and if lowest is different suit2. Then do the same with the ground.

In your example, it would be king and queen of suit1 in your hand and 4 suit1, and 10 suit2 and 6 suit3 on the ground.

• I thought of that, but that doesn't solve the issue, for example: I have 10 hearts and 10 spades, table has three Jacks of hearts, spades and clubs. Also, this answer is very specific to this case, I was hoping for something a little more general Commented Aug 6, 2018 at 11:35
• if the suit in your hand is same then select suit1 from the ground. And about your example, hearts and spades would be suit1 and suit2 and clubs will be suit3, so for each similar case, you will have an equal case. please give two example that will not be equal with this solution. Commented Aug 6, 2018 at 12:16
• I think I see where I got it wrong.. thanks. Still, is there a more general solution? Commented Aug 6, 2018 at 12:27

I like your use of the word "like". It means "having the same characteristics or qualities as; similar to." It means that in some ways it is the same, but implies that in some ways it is different. For this problem I am going to hear your like as if you were saying "similar in a general sense, but dissimilar in significant enough ways to drive my current approach".

One paraphrase to your main question: how do I connect similar or effectively identical states in the state-space so that the training rate and training quality are maximized without having to rely on a-priori knowledge such as combinatorics.

If I had to do this, I would use a graph-network to represent the transition paths, find connection groups that had similar statistics, and preferentially explore then as a paired-test. If the weighted connections in the similar subgraphs align within a tolerance band, then we could call them an approximate isomorphism, then set something like a file-link, so that any attempt to perform q-learning in the isomorphous domain is only operated on the non-copy. As long as "close enough" is well specified, this could truncate the search space substantially.

(still working) To do:

• set up a poker-analog and use the graph-centric approach to handle isomorphous regions
• compare with classic Q-learning
• compare with combinatoric (expert) speedup
• perhaps find a "like" question such that combinatoric method is not viable, and apply graph-based search-space reduction.