left_join two dataframes out of memory #352

Activity

1. xiaodaigh commented on Sep 11, 2021

   

   xiaodaigh

   on Sep 11, 2021

   Last edited by xiaodaigh

   Collaborator

   More actions

   I suspect your left join has a lot of duplicate keys. For example, if i generate a and b as below

   library(disk.frame)
   setup_disk.frame()
   
   a = data.frame(
     id = sample(1:8, 2127625, replace=TRUE)
   )
   
   b = data.frame(
     id = sample(1:8, 73364, replace=TRUE)
   )
   
   
   library(dplyr)
   
   left_join(a, b)
   

   This would also fail to join on my computer and it's allocating 72.7GB of RAM.

   You can't improve the situation by having more workers, but you might be able to fix the issue by having fewer workers and working on smaller chunks like this.

   setup_disk.frame(workers = 4) # reduce to 1 if still not working since the more worker you have the more memory it uses
   a = as.disk.frame(a)
   b = as.disk.frame(b)
   
   nchunks_for_merge = max(nchunks(a), nchunks(b))
   
   # double the number of chunks and shard by the merge key which is "id" in my case
   a = rechunk(a, nchunks=nchunks_for_merge*2, shardby = "id")
   b = rechunk(b, nchunks=nchunks_for_merge*2, shardby = "id")
   
   nc = max(nchunks(a), nchunks(b))
   
   #setup_disk.frame(workers=1) # if necessary
   
   system.time(ab <- left_join(a, b, merge_by_chunk_id=TRUE))
   
2. DiskFrame

   locked and limited conversation to collaborators on Sep 11, 2021
3. xiaodaigh

   closed this as completedon Sep 11, 2021