Memory profiling

[wlandau]

I received a request to support memory profiling in proffer (r-prof/proffer#32) and I am trying to figure out why proffer::pprof(memory.profiling = TRUE) does not show memory profiling in the pprof dashboard. I am wondering if profile is already capturing the memory data. I have a small example:

path <- tempfile()
Rprof(filename = path, memory.profiling = TRUE)
n <- 1e3
x <- data.frame(x = rnorm(n), y = rnorm(n))
for (i in seq_len(n)) {
  x[i, ] <- x[i, ] + 1
}
Rprof(filename = NULL)

It looks like Rprof() is correctly recording memory data.

head(summaryRprof(filename = path, memory = "both")$by.total)
#>                      total.time total.pct mem.total self.time self.pct
#> "Ops.data.frame"           0.10     83.33      89.6      0.00     0.00
#> "as.data.frame.list"       0.08     66.67      58.4      0.02    16.67
#> "as.data.frame"            0.08     66.67      58.4      0.02    16.67
#> "data.frame"               0.08     66.67      58.4      0.00     0.00
#> "<Anonymous>"              0.06     50.00      58.4      0.00     0.00
#> "do.call"                  0.06     50.00      58.4      0.00     0.00

But I am having trouble locating memory data in the data from profile::read_rprof(), which proffer uses to convert to pprof format..

packageVersion("profile")
#> [1] ‘1.0.3.9019’

samples <- profile::read_rprof(path)
#> Warning message:
#> Removing unexpected incomplete sampling information.

str(samples)
#> List of 6
#>  $ meta        : tibble [1 × 2] (S3: tbl_df/tbl/data.frame)
#>   ..$ key  : chr "version"
#>   ..$ value: chr "1.0"
#>  $ sample_types: tibble [1 × 2] (S3: tbl_df/tbl/data.frame)
#>   ..$ type: chr "samples"
#>   ..$ unit: chr "count"
#>  $ samples     : tibble [6 × 2] (S3: tbl_df/tbl/data.frame)
#>   ..$ value    : int [1:6] 1 1 1 1 1 1
#>   ..$ locations:List of 6
#>   .. ..$ 1: tibble [1 × 1] (S3: tbl_df/tbl/data.frame)
#>   .. .. ..$ location_id: int 4
#>   .. ..$ 2: tibble [13 × 1] (S3: tbl_df/tbl/data.frame)
#>   .. .. ..$ location_id: int [1:13] 5 11 16 12 14 9 7 6 13 8 ...
#>   .. ..$ 3: tibble [2 × 1] (S3: tbl_df/tbl/data.frame)
#>   .. .. ..$ location_id: int [1:2] 2 1
#>   .. ..$ 4: tibble [3 × 1] (S3: tbl_df/tbl/data.frame)
#>   .. .. ..$ location_id: int [1:3] 3 2 1
#>   .. ..$ 5: tibble [4 × 1] (S3: tbl_df/tbl/data.frame)
#>   .. .. ..$ location_id: int [1:4] 18 17 10 15
#>   .. ..$ 6: tibble [8 × 1] (S3: tbl_df/tbl/data.frame)
#>   .. .. ..$ location_id: int [1:8] 9 7 6 13 8 7 10 15
#>  $ locations   : tibble [18 × 3] (S3: tbl_df/tbl/data.frame)
#>   ..$ location_id: int [1:18] 1 2 3 4 5 6 7 8 9 10 ...
#>   ..$ function_id: int [1:18] 1 2 3 4 5 6 7 8 9 10 ...
#>   ..$ line       : int [1:18] 0 0 0 0 0 0 0 0 0 0 ...
#>  $ functions   : tibble [18 × 6] (S3: tbl_df/tbl/data.frame)
#>   ..$ function_id: int [1:18] 1 2 3 4 5 6 7 8 9 10 ...
#>   ..$ name       : chr [1:18] "[" "[_data_frame" "[[" "[<-" ...
#>   ..$ system_name: chr [1:18] "[" "[.data.frame" "[[" "[<-" ...
#>   ..$ filename   : chr [1:18] "" "" "" "" ...
#>   ..$ start_line : int [1:18] 0 0 0 0 0 0 0 0 0 0 ...
#>   ..$ .file_id   : int [1:18] 0 0 0 0 0 0 0 0 0 0 ...
#>  $ .rprof      :List of 3
#>   ..$ header: chr "memory profiling: sample.interval=20000"
#>   ..$ files : chr(0) 
#>   ..$ traces: chr [1:6] ":478394:774433:53618096:9090:\"[<-.data.frame\" \"[<-\" " ":391832:614469:36974056:19040:\"mode\" \"%in%\" \"deparse\" \"paste\" \"deparse1\" \"force\" \"as.data.frame.nu"| __truncated__ ":559003:984469:64613920:22356:\"length\" \"[.data.frame\" \"[\" " ":484278:820469:52243128:22499:\"[[.data.frame\" \"[[\" \"[.data.frame\" \"[\" " ...
#>  - attr(*, "class")= chr "profile_data"

[krlmlr]

Thanks. I never actually looked into memory profiling, it's unlikely that the data is captured. Would you like to take a stab at extending the data model we're using here?

[wlandau]

I'm not sure if I will have the capacity, but I made a note in case I have extra time.

[snowpong]

So the task would be to extend the internal format of profile with memory information, and find out how that could then be mapped to pprof file format?

[krlmlr]

I think so, yes.

[krlmlr]

For a clean solution, we also need to think about extending the validation to support different versions of profile objects, with and without memory data. I'd like the format to be type stable, with empty all-NA columns as needed, with a version tag that allows distinguishing between different object versions.

We'd also need sample data for .out , and perhaps infrastructure to generate them. I don't remember where the current sample in the package comes from.