First off, I don't really know much about machine learning.

In a virtual world, such as a video game like minecraft or an application like Google Street view, a user can navigate the world using the keyboard and mouse.

I'm looking to predict what interaction the user will perform. For example, the user might hold down w for some period of time (to move forward) and then press and hold a for a period of time to move left. Alternatively, the user may hold w and use the mouse to turn and look up/down.

I've looked into N-Grams, and I can predict interaction to a certain degree, but I am not considering how long a user is not performing an action. I think performing no action is an important indicator for correct interaction prediction as "no action" could itself be classified as an interaction. The duration of an interaction should also be important, but again, my not sure how to consider that in a model. My Ngram model simply considers the past 3 or 5 inputs.

I'd like to develop something which is very fast to predict interaction (this is key) and learns rapidly from mistakes. Ideally, it would be "Online" in that it constantly "Learns".

I'd be interested in hearing which models you think would be well suited to my task (are ngrams appropriate?), any examples which closely match my task and I can therefore adopt to suit my needs, any datasets containing user interaction and what I should look at to predict the mouse interaction.


1 Answer 1


Interesting question. I haven't worked with this sort of data much, but it seems to me that the bulk of the job is likely to be feature engineering. Every "supervised" statistical method that I know of requires that you shoe-horn your data into "outcomes" and "covariates." $\mathbf{Y}$ and $\mathbf{X}$. Once your data is in this form you simply find an appropriate algorithm that can estimate

$$ \mathbf{Y} = f(\mathbf{X}) $$

But taking data on how people play a video game and turning it into a 2-dimensional matrix isn't necessarily obvious. One of the most difficult things is that you'll need to make observations of different users comparable.

Say you're playing quake 3 and you're trying to predict whether the user is going to go for the BFG. How do you define $\mathbf{Y}$? Is it got BFG? Or should it be came within some distance of BFG? (maybe people messed up the rocket jump, for example). What if they got fragged on the way to the BFG? The point is that defining the outcome involves choices.

Likewise what are the covariates? Say I get spawned in some location, and I know the map, and immediately move toward the BFG. I'll have a different set of w, a, s, and d keystrokes than someone who got spawned elsewhere on the map. So is it n-grams that you want to look at? Maybe rather it is change in euclidian distance between spawning location and BFG location over some time interval. if so, what is the appropriate time interval?

In general, what you're doing is taking raw data and turning it into abstractions of that data. This is actually what a neural net does. It takes raw data $\mathbf{X}$ and forms a lower-dimensional representation of that data, $\mathbf{V}$, where $\mathbf{V}$ has fewer columns than $\mathbf{X}$. When neural nets work well, those columns of $\mathbf{V}$ can be uncannily similar to things that a human would pick out of a stream of raw data. Akin to this: https://www.youtube.com/watch?v=3vAnuBtyEYE

Those $\mathbf{V}$ are then just related to your outcome $\mathbf{Y}$ in a linear model or a logistic regression.

But you don't turn raw keystrokes into blonde, brunette and redhead without a lot of data and long training times. It'll go faster if you can prespecify functions of your raw data, which requires domain knowledge.


Not the answer you're looking for? Browse other questions tagged or ask your own question.