Friday, May 08, 2009

SVN Client: Out Of Memory

With the client SVN 1.6.1 (32 bit on Windows) I ran into an issue today. I tried to commit 13,000 files with about 100 MBytes of size. Most of them are text files and just a very small number of them are binary files. No matter what, this commit didn't work. I actually had to take small portions of it and commit one portion at a time. Admittedly this is not a typical change set, and so I don't want to complain too much. There is one observation, though, that made me think: When the client starts to send the content (file data) then the memory consumption goes up. I'm not sure why that is because the file is still available locally (since we are uploading the file) and hence it's not quite clear to me why the consumption needs to go up. The client sends one file at a time anyways. If a buffer is used to get the data into the appropriate format for the wire transmission I can understand that. But can't that buffer be reused once one file has been transmitted and the next one is started? Maybe I'm overlooking something about the inner workings of Subversion. In that case please comment on this post. Otherwise I have that gut feeling that there might be a memory leak in the current implementation (SVN client 1.6.1)? --- An update: There are a few open bugs related to memory leaks, even one with regards to committing large number of added files. That one is from 2004 and still open.

2 comments:

  1. Another update: I logged this as a bug on Subversion's web site.

    ReplyDelete
  2. Anonymous4:10 AM

    This bug is really anoing and opposite to the subversion people that commented on your bug report I think it is a newly introduced one.
    I use subversion to maintain a seperate repository of vendor packages for our own project and didn't encounter any problems with large packages until now.
    E.g. we maintain our own VTK branchs (8795 files) for quite some time now with no problems.
    Opposite to that, trying to add a Qt branch (only 2476 files) yesterday with a recent subversion client was a real mess.

    ReplyDelete

All comments, questions and other feedback is much appreciated. Thank you!