TransWikia.com

For memory, what should be done when you need to constantly grow a vector to an unknown upper limit?

Stack Overflow Asked on November 15, 2021

Suppose that you are dealing with a potentially infinite amount of data. Suppose further that you do not have this data stored in memory, but can generate individual terms at will. Finally, suppose that you want to do some experiment on this data that will involve checking a large but unknown amount of terms in a way that necessitates keeping a great many of them in memory. Toy problems with Recamán’s sequence, like "find the minimum number terms needed in that sequence for the first 25 even numbers to have appeared", are what I have in mind as typical examples.

The obvious solution to this sort of problem would be to write some code like:

list<-c(first term)
while([not found enough terms yet])
{
  nextTerm<-Whatever
  if(this term worked){list<-c(list,nextTerm)}
}

However, building a big vector like this by adding one new term at a time is your memory’s worst nightmare. The alternative that I often see suggested is to pre-allocate a big vector in memory by making the first line of your code something like list<-numeric(10^6), but those solutions suppose that we have some rough idea of how many terms we need to check, which isn’t always the case. So what can we do when we are dealing with an ever-growing list of unknown required length?

One Answer

This is very popular subject in R check this answer: https://stackoverflow.com/a/45195098/5442527 Summing up: Do not use c() to bind as providing value by index [ is much faster. I know that it might seem surprising that you could grow pre-allocated vector. Make an iter variable before while loop and increase the index inside the if statement.

Normally like in Python you do not have to care about it when using append. Even starting with empty list is not an problem as the list (reserved memory) grows expotentialy (x2x2x1.5x1.2...) when you pass some perimeter number of elements. Link Over-allocating

Answered by polkas on November 15, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP