1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<TITLE> [Mageia-sysadm] Alamut memory problem
</TITLE>
<LINK REL="Index" HREF="index.html" >
<LINK REL="made" HREF="mailto:mageia-sysadm%40mageia.org?Subject=Re%3A%20%5BMageia-sysadm%5D%20Alamut%20memory%20problem&In-Reply-To=%3C1299582078.8638.13.camel%40akroma.ephaone.org%3E">
<META NAME="robots" CONTENT="index,nofollow">
<META http-equiv="Content-Type" content="text/html; charset=us-ascii">
<LINK REL="Previous" HREF="003021.html">
<LINK REL="Next" HREF="003022.html">
</HEAD>
<BODY BGCOLOR="#ffffff">
<H1>[Mageia-sysadm] Alamut memory problem</H1>
<B>Michael Scherer</B>
<A HREF="mailto:mageia-sysadm%40mageia.org?Subject=Re%3A%20%5BMageia-sysadm%5D%20Alamut%20memory%20problem&In-Reply-To=%3C1299582078.8638.13.camel%40akroma.ephaone.org%3E"
TITLE="[Mageia-sysadm] Alamut memory problem">misc at zarb.org
</A><BR>
<I>Tue Mar 8 12:01:18 CET 2011</I>
<P><UL>
<LI>Previous message: <A HREF="003021.html">[Mageia-sysadm] Alamut memory problem
</A></li>
<LI>Next message: <A HREF="003022.html">[Mageia-sysadm] Proposal for maintainers database API
</A></li>
<LI> <B>Messages sorted by:</B>
<a href="date.html#3027">[ date ]</a>
<a href="thread.html#3027">[ thread ]</a>
<a href="subject.html#3027">[ subject ]</a>
<a href="author.html#3027">[ author ]</a>
</LI>
</UL>
<HR>
<!--beginarticle-->
<PRE>Le dimanche 06 mars 2011 à 12:57 +0100, Michael Scherer a écrit :
><i> Hi,
</I>><i>
</I>><i> it seems that alamut memory usage is growing since a few days ( in
</I>><i> fact, we have seen it now that we monitor it thanks to xymon
</I>><i> installation, but it should be since a much longer time ).
</I>><i>
</I>><i> After some hours of debugging with dmorgan, I narrowed the problem down
</I>><i> to viewvc, after careful looking to logs and with the help of watch -d
</I>><i> &
</I>><i> ps .Each time we see a line IOError in error.log, that the sign
</I>><i> there is a memory leaked ( as we can see on graphs if we look carefully
</I>><i> ). I
</I>><i> suspect that the problem is on viewvc side, as explained on :
</I>><i>
</I>><i> <A HREF="http://viewvc.tigris.org/ds/viewMessage.do?dsForumId=4254&dsMessageId=2364399">http://viewvc.tigris.org/ds/viewMessage.do?dsForumId=4254&dsMessageId=2364399</A>
</I>><i>
</I>><i> Before, we did allowed wsgi web software to register custom signals
</I>><i> handler (
</I>><i> <A HREF="http://svnweb.mageia.org/adm/puppet/modules/apache/templates/mod_wsgi.conf?r1=908&r2=1051">http://svnweb.mageia.org/adm/puppet/modules/apache/templates/mod_wsgi.conf?r1=908&r2=1051</A>
</I>><i> ), but this caused apache to not restart, and so it was disabled again(
</I>><i> <A HREF="http://svnweb.mageia.org/adm/puppet/modules/apache/templates/mod_wsgi.conf?r1=1051&r2=1195">http://svnweb.mageia.org/adm/puppet/modules/apache/templates/mod_wsgi.conf?r1=1051&r2=1195</A>
</I>><i> ).
</I>><i>
</I>><i> So basically, when we see this :
</I>><i>
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66] Traceback
</I>><i> (most recent call last):
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66] File
</I>><i> "/usr/share/viewvc/bin/wsgi/viewvc.wsgi", line 40, in application
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66]
</I>><i> viewvc.main(server, cfg)
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66] File
</I>><i> "/usr/share/viewvc/lib/viewvc.py", line 4415, in main
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66]
</I>><i> view_error(server, cfg)
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66] File
</I>><i> "/usr/share/viewvc/lib/viewvc.py", line 4403, in view_error
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66]
</I>><i> debug.PrintException(server, exc_dict)
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66] File
</I>><i> "/usr/share/viewvc/lib/debug.py", line 81, in PrintException
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66]
</I>><i> server.write("<h3>An Exception Has Occurred</h3>\\n")
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66] File
</I>><i> "/usr/share/viewvc/lib/sapi.py", line 252, in write
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66]
</I>><i> self._wsgi_write(s)
</I>><i> [Tue Mar 01 07:31:43 2011] [error] [client 74.125.16.66] IOError:
</I>><i> client connection closed
</I>><i>
</I>><i> there is a problem.
</I>><i>
</I>><i> At first, I thought it was some users with crappy internet connexion,
</I>><i> but this example is googlebot.
</I>><i>
</I>><i> So far, we have a few options :
</I>><i> - going back on cgi would mask the errors, to the cost of speed
</I>><i> - asking to apache to recycle more often his childs would also mask the
</I>><i> error
</I>><i> - reallowing mod_wsgi to add a signal handler would likely do it too
</I>><i> - fixing the leak on viewvc side should IMHO better but maybe too
</I>><i> complex
</I>><i>
</I>><i> So I propose the following :
</I>><i>
</I>><i> - use python-flup to make viewvc run a external fastcgi process, so it
</I>><i> can register signal handler as he see fit, and do not interfer with
</I>><i> apache. This also bring consistancy ( since we use fastcgi for almost
</I>><i> everything, except bugzilla and django applications ), and would allow
</I>><i> a finer grained control ( as we can see each process taking memory and
</I>><i> plan ressources based on this ).
</I>><i>
</I>><i> WDYT ?
</I>
Since no one commented, I went ahead this morning. So far, it still work
fine, we will see if this help.
--
Michael Scherer
</PRE>
<!--endarticle-->
<HR>
<P><UL>
<!--threads-->
<LI>Previous message: <A HREF="003021.html">[Mageia-sysadm] Alamut memory problem
</A></li>
<LI>Next message: <A HREF="003022.html">[Mageia-sysadm] Proposal for maintainers database API
</A></li>
<LI> <B>Messages sorted by:</B>
<a href="date.html#3027">[ date ]</a>
<a href="thread.html#3027">[ thread ]</a>
<a href="subject.html#3027">[ subject ]</a>
<a href="author.html#3027">[ author ]</a>
</LI>
</UL>
<hr>
<a href="https://www.mageia.org/mailman/listinfo/mageia-sysadm">More information about the Mageia-sysadm
mailing list</a><br>
</body></html>
|
