extensions doesn't seem to work with svn

[lilireli]

Hello,

I'm using cvsanaly to parse the logs of one subversion repository, and I tried to run differents extensions (CommitLOC and Metric), they both run at infinite, with no error message. I tried with subset of my repo and they run continuously whatever the size of the repo.

[andygrunwald]

Hey @lilireli,
can you offer your configuration (config file)?
And is this subversion repository public available that we can reproduce it?
And can you offer your command you executed?

[lilireli]

Hey,

I tried different ways. First I used the config file provided as exemple, changing the information I needed (as shown under). I was running the command:
cvsanaly2 http://my_url

####### config file ######
debug = False
quiet = False
profile = False

db_driver = 'mysql'
db_user = 'root'
db_password = 'kartoffel'
db_database = 'development'
extensions = ['CommitsLOC']
###########################

Then I tried to remove this file and rerun cvsanaly with the url still:
cvsanaly2 -u user -p my_password -d my_database --extensions=CommitsLOC http://url

Finally I tried to check out the project and run the following command in the right folder
cvsanaly2 -u user -p my_password -d my_database --extensions=CommitsLOC

Now, everything works fine without the extension, but when it comes to the CommitLOC extension either it creates an empty table, either it runs continuously, either it does both. I also tried with metrics, and I have the same.

Unfortunately, the repo I'm working on is not public and I cannot provide any access to it

[jgbarah]

Have you tried to run the same setup (same version of CVSAnalY, same environment, etc.) with some other (if possible, simple) Subversion repository? Just to discard that there is some trouble with the setup.

[lilireli]

Hi,

I tried with a public folder, the problem is the same. However I get the
following error:

Parsing log for /tmp/extjs (svn)
Executing extensions
Executing extension CommitsLOC
Traceback (most recent call last):
File "/usr/local/bin/cvsanaly2", line 5, in
pkg_resources.run_script('cvsanaly2==2.1.0', 'cvsanaly2')
File "build/bdist.linux-i686/egg/pkg_resources.py", line 528, in
run_script
dist.insert_on(self.entries, entry)
File "build/bdist.linux-i686/egg/pkg_resources.py", line 1394, in
run_script
(d>>9)+1980, (d>>5)&0xF, d&0x1F, # ymd
File
"/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/EGG-INFO/scripts/cvsanaly2",
line 37, in
retval = pycvsanaly2.main.main (sys.argv[1:])
File
"/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/main.py",
line 347, in main
emg.run_extensions(repo, path or uri, db)
File
"/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/ExtensionsManager.py",
line 105, in run_extensions
self.run_extension(name, extension, repo, uri, db)
File
"/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/ExtensionsManager.py",
line 66, in run_extension
extension.run(repo, uri, db)
File
"/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/extensions/CommitsLOC.py",
line 305, in run
(added, removed) = counter.get_lines_for_revision(revision)
File
"/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/extensions/CommitsLOC.py",
line 115, in get_lines_for_revision
printerr("Error running svn diff command: %s", (str(e)))
File
"/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/utils.py",
line 111, in printerr
str = str % tuple(to_utf8(arg) for arg in args)
TypeError: not all arguments converted during string formatting

By printing the error I get that cvsanaly cannot run the command:
svn diff -r 131:132 http://extjs-public.googlecode.com/svn

When I try to run it myself, it works. However I'm using a proxy to get
access to internet and maybe the problem comes from this. Do you know if
the proxy settings are still handled correctly when using cvsanaly ?

Thanks for your help

[sduenas]

I think this is not a bug in CVSAnalY. From my point of view it's a problem related to SVN servers. Usuarlly, SVN servers are slow and commands like svn diff are even more slower.

svn diff is a time consuming operation (I don't really know why) in which the connection between client and server must be kept alive for a long amount of time. For instance, to get the result from the command svn diff -r 1:2 http://extjs-public.googlecode.com/svn, I needed like 30 minutes for downloading around 20MB. And this is just the first diff.

Right now, I have CVSAnalY running using your example. It's up since 314 minutes ago (~5 hours) and now is executing the fifth diff of the list.

sduenas@Guybrush:~$ ps aux |grep svn
sduenas   6203  0.0  0.2  22628  9032 pts/2    S+   00:18   0:04 svn diff -r 4:5 http://extjs-public.googlecode.com/svn

So, my impression is when you think CVSAnalY is doing nothing, it is really doing things but very slowly because of the problems with the SVN server. If an exception is raised it might be due to a connection error or similar problems.

I fixed the code to know exactly what it is happening when an exception is raised by CommitsLOC extension. I will keep you updated because I want to prove my supposition.

[lilireli]

I agree with you, the diff commands are really slow when computed by cvsanaly, I did the same and it was around one hour for one diff. However (and there I'm sorry not knowing more about how svn server works) when I run the same command by myself I can retrieve the result quite quickly (less than 1 minute). Is it normal ? Can't I do anything to speed a little bit up cvsanaly ?

[sduenas]

It's odd. I've been checking it again and it took the same time running the command in a terminal than by CVSAnalY. Maybe, when you run the command, the data was cached in memory but I'm not sure what SVN servers do on these cases.
Anyway, I'm going to have a look to the code and try to find a way to speed up the process.

[rodrigokuroda]

I'm having the same problem. I also noticed that the "svn diff" is a slow process.
Anyway, in this case, the parameter "--metrics-noerr" shouldn't work?

The output log of the respective command is as follow:

$  cvsanaly2 --debug --writable-path=./Camel --db-user=USER --db-password=PASS --db-database=DB --metrics-all --metrics-noerr --extensions=CommitsLOC,CommitsLOCDet https://svn.apache.org/repos/asf/camel

DBG: INSERT INTO repositories (id, uri, name, type) values (%s, %s, %s, %s)
Parsing log for https://svn.apache.org/repos/asf/camel (svn)
['svn', '-v', 'log', 'https://svn.apache.org/repos/asf/camel']

[... omitted a lot of logs ...]

[... omitted aprox. ~30 "svn diff" logs (as shown bellow) ...]

['svn', 'diff', '-r', '686458:686459', 'https://svn.apache.org/repos/asf']
Traceback (most recent call last):
  File "/usr/local/bin/cvsanaly2", line 5, in <module>
    pkg_resources.run_script('cvsanaly2==2.1.0', 'cvsanaly2')
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/EGG-INFO/scripts/cvsanaly2", line 37, in <module>
    retval = pycvsanaly2.main.main (sys.argv[1:])
  File "/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/main.py", line 383, in main
    emg.run_extensions(repo, path or uri, db)
  File "/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/ExtensionsManager.py", line 105, in run_extensions
    self.run_extension(name, extension, repo, uri, db)
  File "/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/ExtensionsManager.py", line 66, in run_extension
    extension.run(repo, uri, db)
  File "/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/extensions/CommitsLOC.py", line 305, in run
    (added, removed) = counter.get_lines_for_revision(revision)
  File "/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/extensions/CommitsLOC.py", line 115, in get_lines_for_revision
    printerr("Error running svn diff command: %s", (str(e)))
  File "/usr/local/lib/python2.7/dist-packages/cvsanaly2-2.1.0-py2.7.egg/pycvsanaly2/utils.py", line 111, in printerr
    str = str % tuple(to_utf8(arg) for arg in args)
TypeError: not all arguments converted during string formatting