Edgewall Software

Ticket #2611 (new defect)

Opened 3 years ago

Last modified 13 months ago

Problem with SVN bindings (SVN 1.3.0, Trac r2771)

Reported by: Manuzhai Owned by: cboos
Priority: high Milestone: not applicable
Component: version control Version: 0.10.4
Severity: major Keywords: svn13 svn14
Cc: joshland@…, manuzhai@…, odela01@…, pjenvey@…, shishz@…, james82@…, mark@…, dgeller@…, chasetheblues@…, brian@…, blaufalke@…, marc@…, neofutur@…, bock@…, boris@…, migs.ho@…, progrium@…

Description (last modified by cboos) (diff)

Every morning when I first try to get to Trac, I get this error on the first request:

Mod_python error: "PythonHandler trac.web.modpython_frontend"

Traceback (most recent call last):
...
  File "/home/manuzhai/dev/trac/trac/versioncontrol/svn_fs.py", line 31, in ?
    from svn import fs, repos, core, delta
  File "/usr/lib/python2.4/site-packages/svn/fs.py", line 19, in ?
    from libsvn.fs import *
  File "/usr/lib/python2.4/site-packages/libsvn/fs.py", line 29, in ?
    import core
  File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 3049, in ?
    svn_pool_create()
  File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 1098, in svn_pool_create
    return apply(_core.svn_pool_create, args)

TypeError: argument number 0: a 'apr_pool_t *' is expected, 'instance()' is received

Similar symptoms:

[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend: TypeError: argument number 0: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0xb6924c8c>)' is received

Subsequent requests seem to just work...

Note

This is now a well-known issue having to do with the way the SVN Python bindings behaves when multiple Python interpreters are used within the same process, as this can be the case with mod_python e.g. when more than one virtual host is used.

Most SVN binding versions are affected, though there's no confirmed report of this with 1.2.x (the bug started to be reported with 1.3.0) and reverting to SVN 1.2.4 is sometimes mentioned as a workaround.

There are other documented workarounds where one can keep using SVN 1.3.x and 1.4.x:

Related issue: #3371

Attachments

application_pool_race_condition_fix.txt (2.0 KB) - added by David James <james82@…> 3 years ago.
Fix race condition in application pool initialization

Change History

  Changed 3 years ago by cboos

  • priority changed from normal to high
  • severity changed from normal to major
  • milestone set to 0.9.4

See also #2472, where this error was also reported and discussed. As that other ticket was about another issue with the 1.3.0 bindings, it has been closed and this ticket should be used to focus on the 'apr_pool_t *' is expected issue.

  Changed 3 years ago by anonymous

  • cc odela01@… added

  Changed 3 years ago by pjenvey@…

  • cc pjenvey@… added

  Changed 3 years ago by anonymous

  • cc shishz@… added

  Changed 3 years ago by Manuzhai

This error seems to appear quite often on editing a wiki page. Refreshing the edit (resending POST data) gives the same error again, going back to the wiki page, then sending it again too, hard-refreshing the page then redoing the changes and submitting them still seems to work. Not sure whether this is always true...

  Changed 3 years ago by cboos

I've not been able to investigate and reproduce this myself, but could you try the following?

Index: svn_fs.py
===================================================================
--- svn_fs.py	(revision 2819)
+++ svn_fs.py	(working copy)
@@ -28,7 +28,7 @@
 from trac.versioncontrol.svn_authz import SubversionAuthorizer
 
 try:
-    from svn import fs, repos, core, delta
+    from svn import core, fs, repos, delta
     has_subversion = True
 except ImportError:
     has_subversion = False

  Changed 3 years ago by Manuzhai

I've made the change, and will report back to see if it helps.

  Changed 3 years ago by anonymous

  • cc james82@… added

I installed mod_python 3.1.4 / Python 2.3 / Subversion 1.3.0 on a single processor RHEL4 machine and I can't reproduce this issue. Ideas?

  Changed 3 years ago by pjenvey@…

I am seeing this problem with python 2.4. The original bug reported to the subversion-users list also occured under python 2.4

Myabe the fact that you're using python 2.3 is a factor, or possibly some other dependency version mismatch?

  Changed 3 years ago by Manuzhai

Yes, I'm also using Python 2.4. The change of import order doesn't solve all of the problem, though it does seem that it doesn't happen as often now. That might have to do with my usage pattern, though...

  Changed 3 years ago by cboos

Can you try to split the import in several lines, i.e.

try:
    from svn import core
    from svn import fs
    # etc.

  Changed 3 years ago by Manuzhai

Done...

  Changed 3 years ago by cboos

  • owner changed from cmlenz to cboos

I think I understand the issue: having all the imports in one line allows two threads (A and B) to start simultaneously the import process:

  • A: trac/versioncontrol/svn_fs.py: from svn import fs, repos, core, delta
  • B: trac/versioncontrol/svn_fs.py: from svn import fs, repos, core, delta
  • A: in svn/fs.py, line 19: from libsvn.core import *,
  • A: now in libsvn/core.py, is going through the > 3000 lines...
  • B: in svn/fs.py, line 19: from libsvn.core import *,
  • B: now in libsvn/core.py, is going through the > 3000 lines...
  • A: in libsvn/core.py, line 3049: does a _core.svn_pool_create
    and here, _core is really _core (i.e. the native extension)
  • A: in svn/fs.py, line 29: import svn.core as _core
  • B: in libsvn/core.py, line 3049: does a _core.svn_pool_create
    and now, _core is not _core but svn.core and therefore, _core.svn_pool_create is not the native function but the (deprecated) compatibility function, which expects a pool as first argument... hence the error.

Hopefully splitting the imports does solve the issue, but the bindings should really be more robust about that and not rename different things to a same name...

  Changed 3 years ago by mark@…

  • cc mark@… added

  Changed 3 years ago by james82@…

I have not been able to reproduce the problem (even with Python 2.4 and mod_python 3.1.3), but, I think I have figured out how to fix it. Please try the below patch (for Subversion 1.3.0).

  • subversion/bindings/swig/include/svn_types.swg

     
    701701 
    702702%exception { 
    703703#ifdef SWIGPYTHON 
    704     svn_swig_py_release_py_lock(); 
     704    int released_lock = 0; 
     705    if (_global_svn_swig_py_pool != NULL) { 
     706      released_lock = 1; 
     707      svn_swig_py_release_py_lock(); 
     708    } 
    705709#endif 
    706710    $action 
    707711#ifdef SWIGPYTHON 
    708     svn_swig_py_acquire_py_lock(); 
     712    if (released_lock) { 
     713      svn_swig_py_acquire_py_lock(); 
     714    } 
    709715#endif 
    710716} 
    711717 

  Changed 3 years ago by james82@…

Important note: In order for the above patch to take effect, you'll need to install SWIG 1.3.25, and run the following commands:

   make extraclean
   ./autogen.sh
   ./configure
   make swig-py
   make install-swig-py

Note that you need to make extraclean to remove any old headers shipped with the tarball.

  Changed 3 years ago by pjenvey@…

Manuzhai -

Did splitting the import into multiple lines solve the problem?

If not, I'll see if I can give james' patch a shot later today

  Changed 3 years ago by Manuzhai

Splitting the import didn't solve the problem.

James' fix is a little involved for me, I just install SVN & bindings through emerge, I'm afraid. I could try it, but I'm not sure when I get to it.

  Changed 3 years ago by pjenvey@…

james - I upgraded to swig 1.3.27, applied your patch and did the whole make extraclean/autogen. Now I am noticing a different, similar error:

Traceback (most recent call last):
  File "/usr/local/lib/python2.4/site-packages/trac/web/modpython_frontend.py", line 206, in handler
    dispatch_request(mpr.path_info, mpr, env)
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 139, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line 107, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/web_ui/browser.py", line 91, in process_request
    repos = self.env.get_repository(req.authname)
  File "/usr/local/lib/python2.4/site-packages/trac/env.py", line 157, in get_repository
    authname)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/api.py", line 61, in get_repository
    return self._connector.get_repository(repos_type, repos_dir, authname)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/svn_fs.py", line 205, in get_repository
    repos = SubversionRepository(dir, authz, self.log)
  File "/usr/local/lib/python2.4/site-packages/trac/versioncontrol/svn_fs.py", line 224, in __init__
    self.path = repos.svn_repos_find_root_path(path, self.pool())
  File "/usr/local/lib/python2.4/site-packages/libsvn/repos.py", line 43, in svn_repos_find_root_path
    return apply(_repos.svn_repos_find_root_path, args)
TypeError: argument number 2: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0x8b88f6c>)' is received

Instead of seeing the stack trace in my error log (where I saw the original apr_pool_t exception), it shows up on an actual trac "Oops... Trac detected an internal error:" error page. I can't tell if this is due to your patch, the swig upgrade, or both

This new error is intermittent like the old one. Occasionally I receive either 404 responses or blank pages instead of the error message. Nothing in the error_log

Also: I got the impression the make extraclean/reautogen would end up creating a brand new subversion/includes/svn_types.h file corresponding to your change to svn_types.swg. That was not the case -- the patch was successfully applied, but I ended up with the same svn_types.h file. Let me know if that is actually a problem

  Changed 3 years ago by David James <james82@…>

Hi pjenvey,

It looks like you applied the patch correctly, but my patch did not help. I am not surprised by your new error message, because it is the same as the one that was reported earlier on the users list.

Could you send the full details for your system, and the full details of your apache configuration including your httpd.conf and a description of which "mpm" you are using (e.g. the worker mpm, or the prefork mpm). I would like to try to reproduce the problem by copying your configuration. Are you using Linux, Mac OS, or something else?

Here's a question for the group: In mod_python, does the "global interpreter lock" lock just the current interpreter or does it lock all interpreters? If the "global interpreter lock" only locks the current interpreter, then it explains our problem. We will need to use a different locking system in order to prevent race conditions during pool initialization. Perhaps an APR lock would do the trick?

Cheers,

David

  Changed 3 years ago by Manuzhai

From http://docs.python.org/api/threads.html (and my understanding), the GIL just locks all but one threads within one interpreter. So if multiple interpreters are at work, we need a different locking mechanism (in which case the APR lock would seem like the best solution).

  Changed 3 years ago by David James <james82@…>

Hi pjenvey, Manzuhai:

After thinking about this problem some more, I have written a patch which uses Python locking to solve the race condition in application pool initialization. Could you give this patch a try?

  • subversion/bindings/swig/include/proxy_apr.swg

     
    6060 */ 
    6161 
    6262%pythoncode %{ 
     63import threading 
     64 
    6365application_pool = None 
     66application_pool_lock = threading.Lock() 
    6467class GenericSWIGWrapper: 
    6568  def __init__(self, this, pool): 
    6669    """Create new Generic SWIG wrapper object""" 
     
    98101      def set_parent_pool(self, parent_pool=None): 
    99102        """Create a new memory pool""" 
    100103        global application_pool 
     104 
     105        try: 
     106          application_pool_lock.acquire() 
     107 
     108          self._parent_pool = parent_pool or application_pool 
     109          self._mark_valid() 
     110     
     111          # Protect important functions from GC 
     112          self._apr_pool_destroy = _core.apr_pool_destroy 
     113          self._svn_swig_py_clear_application_pool = \ 
     114            _core.svn_swig_py_clear_application_pool 
     115     
     116          # If we are an application-level pool, 
     117          # then set this pool to be the application-level pool 
     118          if not self._parent_pool: 
     119            svn_swig_py_set_application_pool(self, self) 
     120            application_pool = self 
     121        finally: 
     122          application_pool_lock.release() 
    101123   
    102         self._parent_pool = parent_pool or application_pool 
    103         self._mark_valid() 
    104    
    105         # Protect important functions from GC 
    106         self._apr_pool_destroy = _core.apr_pool_destroy 
    107         self._svn_swig_py_clear_application_pool = \ 
    108           _core.svn_swig_py_clear_application_pool 
    109    
    110         # If we are an application-level pool, 
    111         # then set this pool to be the application-level pool 
    112         if not self._parent_pool: 
    113           svn_swig_py_set_application_pool(self, self) 
    114           application_pool = self 
    115    
    116124      def valid(self): 
    117125        """Check whether this memory pool and its parents 
    118126        are still valid""" 

Important note: Once again, in order for this patch to take effect, you'll need to type:

   make extraclean
   ./autogen.sh
   ./configure
   make swig-py
   make install-swig-py

  Changed 3 years ago by Manuzhai

Could you attach this patch to the ticket insteading of showing it inline? I'm having a hard time getting it applied correctly. Thanks!

Changed 3 years ago by David James <james82@…>

Fix race condition in application pool initialization

  Changed 3 years ago by David James <james82@…>

Hi Manzuhai,

The patch is attached. I've also posted it to the Subversion developers list, at http://svn.haxx.se/dev/archive-2006-01/0946.shtml

Cheers,

David

  Changed 3 years ago by pjenvey@…

David -

Unfortunately I'm getting the same error I reported last time with your latest patch. (scroll up for the entire thing)

 File "/usr/local/lib/python2.4/site-packages/libsvn/repos.py", line 43, in svn_repos_find_root_path
    return apply(_repos.svn_repos_find_root_path, args)
TypeError: argument number 2: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0x92633ac>)' is received

Here's my specs:

FreeBSD 4
python 2.4.2
trac 1.0dev (r2721)
subversion 1.3.0 (with or without your latest patch, and cboos's small __del__ patch in #2472 which shouldn't make a difference)
swig 1.3.27
apr 0.9.7 (from apache-2.0.55)
neon 0.24.7

  Changed 3 years ago by pjenvey@…

  • summary changed from Problem with SVN bindings (SVN 1.3.0, Trac r2771) to Problem with SVN bindings (SVN 1.3.0, Trac r2771)

Also --

prefork MPM

  Changed 3 years ago by cboos

  • summary changed from Problem with SVN bindings (SVN 1.3.0, Trac r2771) to Problem with SVN bindings (SVN 1.3.0, Trac r2771)
  • milestone changed from 0.9.4 to 0.10

I've been unable to actually reproduce the problem (apache-2.0.55, mod_python-3.1.4, Subversion-1.3.0 and Python-2.4.2 on Linux SLES9 64bits (did I say it's reasonably fast?)).

So I'm leaving this open for further investigation or feedback, but post-poning it.

  Changed 3 years ago by Manuzhai

This is still annoying. I'll see if I can try the patch tomorrow.

  Changed 3 years ago by Manuzhai

I thought, what the hell, I'll give it a whirl right now.

And it seems to work alright! We should probably give it some more time, but it looks good. I used a clean 1.3.x Subversion checkout from revision 18687 to apply the patch to, and then followed David's recipe.

(I still wonder why other people haven't been able to reproduce it, though.)

  Changed 3 years ago by mark@…

I have been able to reproduce this. See the previous bug reports on that topic.

Due to this we still use SVN 1.2.x ...

  Changed 3 years ago by Manuzhai

Well, sure, some have had the same errors, but not cboos or djames.

  Changed 3 years ago by anonymous

Manzuhai, can you still reproduce the issue without my patch? Is it clear that the patch fixes the problem?

Neither me nor cboos have been able to reproduce this bug, with or without the patches, so it's possible that the problem went away on your machine for an unrelated reason.

  Changed 3 years ago by Manuzhai

I changed nothing else except your patch, I've been seeing the problem for ages, and it is now gone. I'm fairly certain that it's your patch that fixed it.

  Changed 3 years ago by dg

Just 2 days ago I discovered the very same problem. It occurs only rarely, so I am not suprised not everyone has seen it. I have seen it on our sever machine (single CPU linux), and on my development machine (single CPU linux)

  • mod_python-3.2.7
  • python-2.4.2
  • subversion 1.3.0
  • swig 1.3.27
  • apache 2 with prefork MPM
  • linux

Everything upatched as yet.

Error from log:

[Sat Mar 04 13:47:20 2006] [error] Error(TypeError): argument number 0: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0x430bb02c>)' is received at /usr/local/lib/svn-python/libsvn/core.py line 1098
[Sat Mar 04 13:47:20 2006] [notice] child pid 15649 exit signal Segmentation fault (11)

I am attempting to use the python subversion bindings in a custom application to manage text document revisions - until I saw this problem, everything seemed just great.

Now I am not sure what to do - should I abandon using subversion at all for this?

  Changed 3 years ago by dg

  • cc dgeller@… added

  Changed 3 years ago by mark@…

We can exclude Apache's MPM: I've managed to reproduce this issue on MPM worker.

As version 1.3.0 is a constant, here, you could downgrade SVN to 1.2.3.

And well, none of the above patches worked for me.

  Changed 3 years ago by dg

Well, good news. I applied the patch according to the recipe (application_pool_race_condition_fix.txt), and the problem seems to have gone away!

(same configuration as before, using 1.3.0)

I have not tested this on our server yet, but on my local machine running 10 browser-emulation processes banging away at the local server app (each making 200 requests), not a single error! (2000 requests in about 10 minutes)

Will try this on our server late tonight.

  Changed 3 years ago by Manuzhai

mark@…: it would seem that yours is a different problem, then? As the patches fixes the problem both for me and dg.

  Changed 3 years ago by chasetheblues@…

  • cc chasetheblues@… added

I'm experiencing the same problems after upgrading

  • Bindings to SVN 1.3.0
  • Python from 2.3 to 2.4
  • SWIG from 1.3.2x(?) to 1.3.28.

Using

  • Trac 0.9.4 with
  • mod_python 3.2.8
  • Python 2.4.

  Changed 3 years ago by chasetheblues@…

forgot:

  • Python 2.4.2
  • apr 0.9.7 (from apache-2.0.55)
  • Debian Linux

  Changed 3 years ago by cboos

chasetheblues: have you tried to apply the attachment:application_pool_race_condition_fix.txt and rebuild your bindings as explained in the comment above?

  Changed 3 years ago by chasetheblues

yes, i applied it, and had the make procedure. but it doesn't help. i hence cannot except 100% the possibility of something went wrong at fixing. i got some compile errors, before the make procedure went through in the end.

on some small test projects, the error is shown 10 times in a row before the page is loaded correctly, on large test projects i have to try several dozen page request before any problem.

  Changed 3 years ago by Manuzhai

The same error does still turn up at times for me too, although it now seems much less frequent than before. *shrug* It's kinda weird, it seems really quite unpredictable.

  Changed 3 years ago by Manuzhai

Okay, I have obviously been too optimistic about the patch.

Subversion still dies on me regularly with the same error. djames, how can I help you diagnose this problem? I can probably provide you with access to one of my servers if that would help.

  Changed 3 years ago by dg

Just an update - after applying the patch March 4, we haven't seen the problem since. And we get pretty extensive use of the library, as our online editing system is based on this package.

Wish I could help with something more concrete....

  Changed 3 years ago by ctb

The Problem still exists, even after upgrading to svn 1.3.1 and downgrading to swig 1.3.25 and reinstalling trac.

  Changed 3 years ago by anonymous

Did you already report this to the Subversion's issue tracker?

  Changed 3 years ago by cboos

The problems seems to be there also for 1.3.1 on windows, see #2965.

  Changed 3 years ago by anonymous

  • cc brian@… added

  Changed 3 years ago by anonymous

  • cc blaufalke@… added

  Changed 3 years ago by cboos

  • keywords svn130 added

In order to get some progress on this topic, I'd like that the people who are still seeing the issue report here if they are using multiple virtual hosts, as #2965 suggests that this might trigger the problem.

Also mention if you're using:

  1. svn 1.3.1
  2. svn 1.3.0 + patch, built following those instructions
  3. something else (hm... you should have tried one of the above!)

  Changed 3 years ago by blaufalke@…

  • SVN 1.3.0 (no patching so far).
  • Several virtual hosts on Apache 2.2.0 (SuSE Linux 9.3).
  • Only one virtual host used for trac (via name based subdomains)
  • Subdomains configured like that:
     <Directory /path/to/httpsdocs>
      Satisfy Any
      Require valid-user
      AuthType Basic
      AuthName "Subversion@mydomain.com"
      AuthUserFile /path/to/.htpasswd
     </Directory>
    
     RedirectMatch permanent ^/(projects)$ https://repos.mydomain.com/$1/
     RedirectMatch permanent ^/(projects)/([^/]+)$ https://repos.mydomain.com/$1/$2/
    
     <Location /svn/>
      DAV svn
      SVNParentPath /path/to/repositories
      AuthzSVNAccessFile /path/to/.policy
     </Location>
    
     <Location /projects/>
      SSLRequireSSL
      setHandler mod_python
      PythonPath "sys.path + ['/usr/share/trac']"
      PythonHandler trac.web.modpython_frontend
      PythonOption TracEnvParentDir /path/to/projects/
      PythonOption TracUriRoot /projects/
      SetEnv PYTHON_EGG_CACHE /usr/lib/python2.4/site-packages
      SVNParentPath /path/to/repositories
      AuthzSVNAccessFile /path/to/.policy
     </Location>
    

  Changed 3 years ago by mark@…

Common settings:

  • Several virtual hosts.
    Every configured as described in TracModPython.
  • Trac 0.9.[2-5]
  • Python 2.4.2
  • SQLite 3
  • Apache: mpm_worker, PHP 5.1.x

Tested configurations at which the errors occurs:

  • Subversion 1.3.0 (with and without patch, no relief)
    • Apache 2.0.55, SWIG 1.3.21, APR 0.9.7
    • Apache 2.0.55, SWIG 1.3.21, APR 0.9.12
    • Apache 2.0.58, SWIG 1.3.25, APR 0.9.12
    • Apache 2.2.0, SWIG 1.3.21, APR 1.2.2
    • Apache 2.2.2, SWIG 1.3.25, APR 1.2.7
  • Subversion 1.3.1 (with/without)
    • Apache 2.0.58, SWIG 1.3.25, APR 0.9.12
    • Apache 2.2.0, SWIG 1.3.21, APR 1.2.2
    • Apache 2.2.0, SWIG 1.3.25, APR 1.2.2
    • Apache 2.2.0, SWIG 1.3.25, APR 1.2.7

Drop me a line if I got the "tester's award".

  Changed 3 years ago by mark@…

That's what I've forgotten to mention:

  • Apache 2.0.x: mod_python 3.1.4
  • Apache 2.2.x: mod_python 3.2.8

Everything compiled under Gentoo with GCC 3.4.x, -fomit-frame-pointer -pipe and either:

  • -Os -march=athlon-mp -ffast-math
  • -O2 -mtune=pentium-m -fforce-addr -frename-registers -falign-functions=64

  Changed 3 years ago by mark@…

SVN 1.3.1 has always been without patch - It didn't even apply.

  Changed 3 years ago by cboos

Actually, that patch was merged in Subversion 1.3.1, see TracSubversion for details.

Mark, just in order to verify the hypothesis that using multiple virtual hosts could trigger the problem, would it be possible for you to setup a test environment with one/no virtual hosts?

(of course, that will earn you even more points for the tester-of-the-year contest :) )

  Changed 3 years ago by mark@…

I've had only the time for one test, so here it is:

Environment

  • Apache 2.0.58, APR 0.9.12, mod_python 3.1.4 (r1)
  • Subversion 1.3.1, SWIG 1.3.21

one vhost:

Fire up Apache, click every of the items in mainnav (wiki, timeline...). No error, no matter how much items you invoke.

second vhost:

preparations

Stop Apache, copy 10_trac.conf to 10_something.conf and adjust path and hostname.
Create a new subversion repo, import something into it and run trac-admin /var/lib/trac/something initenv.

Add another line to /etc/hosts to match your new vhost:

127.0.0.1 something.localhost

invocation

Fire up Apache, click through the first installed vhost. No errors will appear, believe me.

And now has the time come to open up the second vhost. Type something.localhost in your browser, get the first page displayed and change into "timeline" or "browse source"! Voila, fatal error appears.

  Changed 3 years ago by cboos

Ok, thanks for the confirmation and for the steps to reproduce the issue.

  Changed 3 years ago by Manuzhai

There's something to the vhost diagnosis, but it's not everything.

On my old server, I had a single vhost containing different Location sections, for different projects, replicating the full setup for each config (e.g. SetHandler?, several PythonOption? directives, and authentication). I encountered the errors there, which is why I opened this ticket. On my new server, however, I have set it up differently, just using the multiple env support from Trac within a vhost, and I get no more errors.

  Changed 2 years ago by anonymous

with subversion 1.3.2,this problem still exists.

  Changed 2 years ago by athomas

I upgraded TracHacks to SVN 1.3.1 and was not able to get Apache to serve any Trac requests at all. Quite aggravating. Downgraded to 1.2.4 until fix is merged.

  Changed 2 years ago by Manuzhai

  • milestone changed from 0.10 to 0.11

This is really a Subversion problem. We should still gather more information here, but let's set the milestone to 0.11; it really doesn't block 0.10.

  Changed 2 years ago by maxb1@…

I have what seems to be a reliable reproduction recipe: You must have an apache setup which contains Trac instances in *multiple* mod_python subinterpreters. Restart apache. The first request to a svn-using page (i.e. "Browse Source") within the first subinterpreter to be accessed will SUCCEED. The first request to a svn-using page within a second subinterpreter will fail with the error.

Additionally, just as an interesting data point: I actually got this error running a totally non-Trac-related command line script, once (it was ViewVC's svndbadmin). Sadly, this was not reproducable.

  Changed 2 years ago by maxb1@…

If anyone has seen this error with SWIG 1.3.28 or 1.3.29, please comment!

The reliable reproduction I mentioned in the previous comment was using the pregenerated SWIG files shipped with subversion 1.3.2 (i.e. SWIG 1.3.25 IIRC). I retested with SWIG 1.3.28, and the problem seems to have disappeared!

Further, I have isolated the source of the TypeError to the following code segment in _wrap_svn_pool_create:

        if (_global_svn_swig_py_pool != NULL &&
            !PyObject_HasAttrString(_global_svn_swig_py_pool,
                                    (char *)"_mark_valid"))
        {
            int argnum = PyTuple_GET_SIZE(args);
            SWIG_Python_TypeError(SWIG_TypePrettyName(SWIGTYPE_p_apr_pool_t),
                                  _global_svn_swig_py_pool);
            SWIG_arg_fail(argnum);
            SWIG_fail;
        }

follow-up: ↓ 66   Changed 2 years ago by marc@…

  • cc marc@… added

I have the same problem, and I'm running

- apache 2.0.54 - mod_python 3.1.4 - subversion 1.3.2 - python 2.4.1 - trac 0.9.4

I have about 15 tracs running under a virtualhost each (on port 81), and get intermittent errors. I have an RSS reader configured to request the Timeline page for each trac and it always gets an error for at least one of the tracs.

When I tried the setup with a single virtualhost I got no errors, but I have not tested that extensively.

If there's anything else that I could do or provide just let me know and I will try.

in reply to: ↑ 65   Changed 2 years ago by anonymous

Replying to marc@guidance.nl:

I have the same problem, and I'm running - apache 2.0.54 - mod_python 3.1.4 - subversion 1.3.2 - python 2.4.1 - trac 0.9.4 I have about 15 tracs running under a virtualhost each (on port 81), and get intermittent errors. I have an RSS reader configured to request the Timeline page for each trac and it always gets an error for at least one of the tracs. When I tried the setup with a single virtualhost I got no errors, but I have not tested that extensively. If there's anything else that I could do or provide just let me know and I will try.

Instead of using virtual hosts, you can use another apache "instance" with another config file and on another TCP port or IP address. Something like this: /usr/sbin/apachectl -f /etc/httpd/conf/httpd.trac.conf

For Trac I have one apache (default) instance for intranet running on port 80, and another apache instance for internet running on port 81, and it works fine.

Before this I had only one apache instance running on ports 80 and 81, and it give this error.

  Changed 2 years ago by cboos

In light of #3455, it seems clear that the fix in attachment:application_pool_race_condition_fix.txt is not working in case of multiple Python subinterpreters (as this is the case by default with virtual hosts).

The svn_swig_py_set_application_pool(self, self) will be called for each sub-interpreter and therefore at the C level one pool object coming from one subinterpreter will be used by all the other subinterpreters. This probably leads to an invalid GC, which destroys that object prematurely, and that deletes the "_is_valid" string which triggers the message we're seeing, see comment:64).

So... the workaround for #3455 should most probably also work here.

It remains to be confirmed if the 1.3.2 bindings generated with SWIG >= 1.3.28 are handling the multiple sub-interpreter scenario correctly.

follow-up: ↓ 70   Changed 2 years ago by anonymous

We have mod_python-3.1.4, apache-2.0.58, swig-1.3.29, subversion-1.3.2, python-2.4.3, trac-0.10rc1. (gcc 4.1.1)

I have been working on converting a bunch of trac sites from 0.8 to 0.10rc1 (we will be moving to 0.10). We have hitting this since converting to gcc-4.1.1. That was a one-way change, but this is crazy. I have no idea what to do =(.

Apache is in mpm-prefork. We have disactivated about everything that we can, and I just implented the

PythonInterpreter "trac"

in my config. We have 5 sites.

  Changed 2 years ago by anonymous

  • cc joshland@… added

BTW: I guess I should mention - we are hitting this bug.

[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend: Traceback (most recent call last):
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/mod_python/apache.py", line 299, in HandlerDispatch\n    result = object(req)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/modpython_frontend.py", line 87, in handler\n    gateway.run(dispatch_request)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/wsgi.py", line 87, in run\n    response = application(self.environ, self._start_response)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 304, in dispatch_request\n    env = _open_environment(env_path, run_once=environ['wsgi.run_once'])
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 57, in _open_environment\n    env_cache[env_path] = open_environment(env_path)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/env.py", line 431, in open_environment\n    env = Environment(env_path)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/env.py", line 121, in __init__\n    load_components(self)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/loader.py", line 140, in load_components\n    __import__(module)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/trac/versioncontrol/svn_fs.py", line 56, in ?\n    from svn import fs, repos, core, delta
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/svn/fs.py", line 19, in ?\n    from libsvn.fs import *
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/libsvn/fs.py", line 29, in ?\n    import core
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 3057, in ?\n    svn_pool_create()
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend:   File "/usr/lib/python2.4/site-packages/libsvn/core.py", line 1098, in svn_pool_create\n    return apply(_core.svn_pool_create, args)
[Fri Sep 29 22:05:12 2006] [error] [client 127.0.0.1] PythonHandler trac.web.modpython_frontend: TypeError: argument number 0: a 'apr_pool_t *' is expected, 'instance(<libsvn.core.GenericSWIGWrapper instance at 0xb6924c8c>)' is received

in reply to: ↑ 68 ; follow-up: ↓ 71   Changed 2 years ago by cboos

Replying to anonymous:

We have mod_python-3.1.4, apache-2.0.58, swig-1.3.29, subversion-1.3.2, python-2.4.3, trac-0.10rc1. (gcc 4.1.1)

In case you regenerated the wrapping code yourself, i.e. really made use of swig-1.3.29, this could explain the problem. Subversion 1.3.x is only compatible with 1.3.24 and 1.3.25.

PythonInterpreter "trac"

Try "main_interpreter", as suggested in #3455.

in reply to: ↑ 70 ; follow-up: ↓ 72   Changed 2 years ago by joshland@…

Replying to cboos:

In case you regenerated the wrapping code yourself, i.e. really made use of swig-1.3.29, this could explain the problem. Subversion 1.3.x is only compatible with 1.3.24 and 1.3.25.

I think this worked. We are returning to mpm-worker mode. Our current software loadout is:

swig-1.3.25, subversion-1.3.2-r3, apache-2.0.58 (mpm-worker) and mod_python-3.1.4

Try "main_interpreter", as suggested in #3455.

my mod_python config is now