0
$\begingroup$

Background

I would consider myself to have a decent level in R and virtually 0 skills in any other programming language - something quite common in my field. Note also that my background is not in data science, I consider myself an interested amateur in that regard.

Working on a new project I took the opportunity to dive into Reinforcement Learning. The final problem I'm hoping to solve is, as far as I can tell with my limited knowledge right now, quite complex (continuous state- and action space, high stochasticity in the environment etc.).

I started out by working through recommended ressources (Sutton and Barto's book, lectures by D. Silver...) and constructed some small, domain-specific problems as exercises along the way. I'm able to solve those with various algorithms, at least as long as working with tabular solution methods. However, I struggle to implement methods using function approximation. It feels like I'm hitting the limits of what I'm able to implement in R, at least given my current knowledge.

Problem

Searching for guideance online I found surprisingly few implemented examples for RL-algorithms using function approximation in R. Almost all examples are using Python, and many algorithms seem to be fully implemented in pytorch and/or tensorflow.

However, I'm reluctant to switch to Python, since this would mean to learn a whole new language (while learning a whole new method). To add on that, I did all of my previous work in R, some parts of which I'd like to integrate into my RL-problem. At the same time I have the fear that I won't be able to tackle my final problem in R (given my limited skills and the lack of available examples) and I will have to switch anyways later on.

Question

Given the information above, should I stick to R and work on a deeper understanding of the underlying methods in order to implement those, or should I switch to Python to take advantage of the available resources, on the cost of learning a whole new language?

Edit: When talking about 'function approximation' I refer to the approximation of state- and/or action-value functions, be it using linear combinations of features or more advanced methods such as neural networks.

$\endgroup$
2
  • $\begingroup$ Can you clarify what kind of function approximation you are talking about? $\endgroup$
    – Valentas
    Jul 20, 2023 at 16:20
  • $\begingroup$ @Valentas: Sure, I edited my question to clarify. As long as I can represent my states and actions in matrices I can follow along quite well, but I struggle to bridge the gap to very large or continuous state spaces. $\endgroup$
    – ionatura
    Jul 21, 2023 at 8:18

1 Answer 1

0
$\begingroup$

The problem is that deep RL (= deep learning + RL) is a whole world sometimes quite different from classical RL (e.g. tabular or linear fn approx.)

From one side, Python and RL libraries are usually built from popular numerical frameworks such as tensorflow and pytorch. The benefit is that there are many RL libraries that already implement SOTA algorithms that you can directly plug in and solve your problem (indeed, assuming you define it in a suitable environment.) Also, if you want to do research you may want to start from some already avaiable implementation, which is likely to be in Python.

The drawback is that you would need to learn: a new programming language that is also quite different from R, but also a Rl lib and even a ML framework (TF or PT.)

But if you stick with R, you probably need to find a good DL library that allows to build and train neural networks, and then implement the RL algo from scratch, which can get quite tricky, technical and complex.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.