1
$\begingroup$

EDITED: See below for additional information..

TL;DR: How can I add missing data in a dataset like the sample in a way that it doesn't deviate much from the original dataset.

ORIGINAL:

I have a question about the mice package.

I am looking at a dataset like the example code:

set.seed(1337)
df <- data.frame(x = c(seq(100, 10, -4), seq(100, 20, -3), seq(100, 60, -3), 
seq(80, 40, -3), seq(100, 10, -3))) %>% data.table()
df[sample(seq_len(nrow(df)), size = nrow(df) * 0.3), x := NA]
df$y <- seq(1,nrow(df),1)

Tmp <- mice(df, m = 1, maxit = 30, seed = 1337, print = FALSE)
Completed <- mice::complete(Tmp, 1)

plot(df$y, df$x, col = "blue")
points(Completed[is.na(df$x), ]$y, Completed[is.na(df$x), ]$x, col = "red")
legend(70, 30, legend = c("actual", "pred"), col = c("blue", "red"), lty = 1, cex = 0.5)

I am trying to fit the missing data points, but as you can see in the image below, the fit (red) isn't like I want it to be, because I would expect it to appear in between the blue line. Would anyone know how to 'fix' this?

For clearance: The data on the x-axis is interpreted as a datetime object while the value on the right is the current battery level (in percentage) of ONE device. The points where the blue dotted line stops and starts at a higher number is when the battery is charged.

Example Plot

UPDATE: I have looked at the DMwR-package and especially at knnImputation. I was able to fit it to the sample set without any issue, see image below. enter image description here

However it gave an issue when I tried to implement it with the real data set, because it threw an error:

Error in scale.default(xcomplete, dm[i, ], FALSE) : length of 'center' must equal the number of columns of 'x'

See sessionInfo below:

> sessionInfo()
R version 3.4.4 (2018-03-15)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS release 6.9 (Final)

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] lubridate_1.3.3     zoo_1.7-12          tidyr_0.6.0         pROC_1.13.0         randomForest_4.6-14 DMwR_0.4.1          lattice_0.20-35     stringr_1.0.0      
 [9] openxlsx_3.0.0      tictoc_1.0          data.table_1.10.4-3 wlhive_1.1.9003     ggplot2_1.0.1       dplyr_0.5.0   
$\endgroup$
  • $\begingroup$ What are you doing with your original data (e.g. fit a regression, do classification, something else)? Do you have reason to believe that it's drawn from an underlying deterministic process so those missing data points really "should" fall on those lines without noise? What determines the lines--does each line correspond to a different subgroup within your data-generating population? Multiple imputation of this form may not be the solution you need here. $\endgroup$ – plagueheart Nov 28 '18 at 16:56
  • $\begingroup$ @plagueheart: the interpretation of this 'issue' is that df$y is a datetime object and df$x is the batterylevel at a given time. Therefore it must fall within the blue 'lines'. However using MICE doesn't work this way. $\endgroup$ – Tunder250 Nov 28 '18 at 17:07
  • 1
    $\begingroup$ Right, I thought that might be the problem. So the issue here is you have a set of distinct time series (the battery's charge during a given observation period) that you know should be monotonically decreasing over the duration of the observation period. That's why MICE isn't fitting data into the lines--it's attempting to impute using all the data available from all the time series as if it were one (possibly periodic) function. MICE is not the tool you want here. (I'll write up a fuller answer when I have a moment to do so.) $\endgroup$ – plagueheart Nov 28 '18 at 17:38
  • $\begingroup$ You'll also want to edit your original question to add that information into it. Are these all done on the same battery at different times? $\endgroup$ – plagueheart Nov 28 '18 at 17:39
  • $\begingroup$ Thak you for your answer! That makes sense. I was thinking that MICE would do that (in a limited way). I have edited the original question to make it (hopefully) clearer. I have also added an update, as I did some testing with different packages that would perform in a different way. I don't want to make it more difficult, but in my original dataset MICE does fit quite well on one column with the same characteristics but it is plain wrong in the other one, which I am questioning for. $\endgroup$ – Tunder250 Nov 29 '18 at 10:26
1
$\begingroup$

After a lucky google search, I found the function na.interpolation from the imputeTS-package. After changing the dataset to a time series (XTS), I was able to impute the right data.

library(xts)
library(imputeTS)

set.seed(1337)
df <- data.frame(x = c(seq(100, 10, -4), seq(100, 20, -3), seq(100, 60, -3), 
                       seq(80, 40, -3), seq(100, 10, -3))) %>% data.table()
df[sample(seq_len(nrow(df)), size = nrow(df) * 0.3), x := NA]
df$y <- seq(1,nrow(df),1)

ts <- xts(df[, -c("y")], as.Date(df$y))
head(ts)
#>              x
#> 1970-01-02 100
#> 1970-01-03  96
#> 1970-01-04  NA
#> 1970-01-05  88
#> 1970-01-06  NA
#> 1970-01-07  80

full <- na.interpolation(ts)
plot(full, type = "p")

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.