Friday, May 15, 2009

I couldn't resist... - a quote from Subversion's web site

When assessing the memory consumption of the Subversion (SVN) 1.6.1 client, I also found the following on SVN's web site:
"all the memory we allocate will be cleaned up eventually" (Source: "Hacker's Guide To Subversion", retrieved 15 May 2009)
I like that quote since I think it is true for all software! "Eventually" all memory will be cleaned up, and if necessary when the process terminates. On second thought, though, here might be an opportunity for a new business model! What if we had one-way memory? You can allocate it once, and once a process has consumed it you need to buy new DRAM (or maybe it could be called OW-DRAM as in "One Way"-DRAM). I'm sure Intel and other chips vendors would love it! But seriously (and I'm sure I've got some funny things somewhere in my published text and code as well. Tell me!): A memory leak is a memory leak is a memory leak. Using add and commit for adding large amounts of files consumes about 1 KByte per file. In one case I tried to add about 13,000 files and the process fell over when it reached 1.3 GByte having added only 10% of the files. So this approach doesn't work for version 1.6.1. All indications are that this is a memory leak. It is proportional to the number of files you add and try to commit. OS tools show how the process grows in size. It never shrinks (unless the process terminates one way or the other). Admittedly I didn't use a memory profiler, but what do you think the issue is when an error is reported on the command line saying "out of memory"? The better option for getting large sets of files into a repository - and that's what I learned by now - is to use SVN's import functionality. I have made several tests now with up to 65,000 files in one batch (several hundred MB) and the memory consumption of the client process grew only very slowly - from about 10 MB to 58 MB at most. This growth - I suspect - is probably related to memory fragmentation but definitely within acceptable limits. So the recommendation is: Don't use svn add and then svn commit. Instead use svn import if you have large sets of files to import. If they go into the wrong place you can always move them later using a repository browser such as the one that comes as part of TortoiseSVN.

0 comments:

Post a Comment

All comments, questions and other feedback is much appreciated. Thank you!