IRC log of musicbrainz-devel on 2012-08-10

Timestamps are in UTC.

00:08:17 [demosdemon]
demosdemon has joined #musicbrainz-devel
00:11:02 [ruaok]
ruaok has joined #musicbrainz-devel
02:04:34 [nikki]
nikki has joined #musicbrainz-devel
04:48:31 [ruaok_]
ruaok_ has joined #musicbrainz-devel
04:52:09 [warp]
좋은 아침입니다!
04:52:58 [nikki]
おはようございます :P
04:53:57 [warp]
:D
04:57:56 [ianmcorvidae]
wow, this font has really ugly korean
04:59:31 [nikki]
it looks ok for me, but the font is too small for the first character
04:59:46 [warp]
the first character is a bit squashed in my fixed-width terminal font here as well.
04:59:57 [ianmcorvidae]
mine has ugly bitmap fonts I think
05:00:05 [nikki]
I mean, I can tell it starts with j and then there's a horizontal vowel and then either ng or h
05:01:44 [ianmcorvidae]
http://ianmcorvidae.net/images/uglykorean.png
05:03:06 [nikki]
eww yeah
05:03:16 [nikki]
the japanese is much nicer
05:03:36 [ianmcorvidae]
the japanese is basically how the rest of the font looks, stylistically
05:04:01 [ianmcorvidae]
I suspect other less-used things would get ugly, but
05:04:38 [ianmcorvidae]
georgian, for example, looks largely fine
05:05:07 [nikki]
the second character looks more like meun than eun to me :/
05:06:01 [nikki]
I suppose georgian has an advantage in that there aren't thousands of characters :P
05:06:13 [ianmcorvidae]
yeah, there is that :P
05:10:03 [warp]
seems very bold for no reason.
06:05:28 [warp]
wtf review board.
06:06:00 [warp]
(it doesn't let me type certain characters)
06:08:20 [ianmcorvidae]
warp: get off the diff page
06:08:36 [ianmcorvidae]
warp: it's keyboard shortcuts for that page specifically
06:08:39 [ianmcorvidae]
which is stupid, but yes :P
06:09:57 [warp]
this + pasting being semi broken makes it very frustrating to use review board.
06:10:13 [ianmcorvidae]
as long as you're not on the diff page it works exactly as before :P
06:10:34 [ianmcorvidae]
(comment popups on the diff page work fine, to my knowledge)
06:11:05 [warp]
our previously deployed version of review board didn't have broken pastes :)
06:11:17 [ianmcorvidae]
no, pronik updated it a few months ago
06:11:24 [warp]
yes, I know.
06:11:43 [ianmcorvidae]
and I've never had any trouble with paste -- except, of course, on the diff page, as mentioned
06:11:56 [ianmcorvidae]
I don't paste much though, I guess
06:12:11 [nikki]
I've never had any problems with pasting, but maybe I haven't used it since it was updated
06:12:41 [ianmcorvidae]
I think just your tx config thing was since the update, unless I'm forgetting something
06:12:51 [warp]
Obviously pasting in itself works, but if you do not touch the field after pasting, review board isn't aware there is something in the field, so clicking OK will nuke your paste.
06:13:06 [ianmcorvidae]
ah
06:13:15 [ianmcorvidae]
I have hit that bug, I hadn't realized that's what it was doing
06:13:55 [warp]
so generally I past the summary and description from the git commit. which now means paste summary, click in the field to focus on something harmless, hit a key (space), then click OK. and again for the description or anything else I need to paste.
06:14:21 [ianmcorvidae]
yeah
06:14:27 [ianmcorvidae]
I've only hit it with copy-pasting ticket numbers
06:14:43 [warp]
in itself not _that_ much of a burden, but it's very annoying when I've forgotton review board is broken
06:15:07 [ianmcorvidae]
yeah
06:15:13 [ianmcorvidae]
I should set up post-review
06:15:50 [warp]
I have a post-review python wrapper which I can use for typical cases. but these diffs aren't diffs against master, so I'm not sure my script will work.
06:16:13 [ianmcorvidae]
ah
06:16:56 [ianmcorvidae]
at present I still do all of mine by hand
06:17:17 [ianmcorvidae]
I've gotten very good at the git diff master...mbs-whatever<tab> > <whatever>.diff routine :P
06:17:17 [warp]
I'll push my script somewhere public after I'm done submitting this stuff to code review.
06:24:06 [warp]
unrelated, next weekend is hack weekend!
06:25:12 [ianmcorvidae]
true
06:25:15 [ianmcorvidae]
I should decide what I'm doing
06:25:34 [ianmcorvidae]
"fix a bunch of bugs" isn't quite glamorous enough, given that's what I do with the rest of my time ;)
06:30:00 [nikki]
coding?
06:38:13 [warp]
ianmcorvidae: https://gitorious.org/musicbrainz-post-review/musicbrainz-post-review
06:39:01 [warp]
ianmcorvidae: the only two ideas I had so far would be making picard more modular (but that sounds like a lot of work, and indeed not very glamorous)
06:39:15 [warp]
ianmcorvidae: and adding JSON-LD support to my /ws/2/ json work.
06:39:21 [ianmcorvidae]
more modular as in library vs. frontend stuff?
06:39:30 [ianmcorvidae]
(because if so that's a lot more work than a weekend, heh)
06:39:37 [ianmcorvidae]
JSON-LD might be neat
06:40:14 [ianmcorvidae]
if you can figure out how to encode everything into the relevant RDF
06:40:22 [warp]
ianmcorvidae: yes. the tagging files stuff shouldn't require Qt or any other GUI bits. so you can just import only those bits to write your own tagger.
06:40:39 [warp]
ianmcorvidae: I can just look at the existing RDFa for that.
06:40:45 [ianmcorvidae]
the issue is that at present it uses Qt stuff for all its internal XML and network stuff
06:40:57 [ianmcorvidae]
the current RDFa doesn't cover a lot of stuff, unfortunately, especially ARs
06:41:04 [ianmcorvidae]
but I guess you can just mirror those limitations
06:41:37 [warp]
hm, no RDF.rb on this machine.
06:41:42 [warp]
* warp installs.
06:42:29 [ianmcorvidae]
hm, copy-paste from gitorious doesn't work properly
06:42:30 [ianmcorvidae]
bah
06:43:17 [warp]
ianmcorvidae: I'll be happy if in a weekend a basic release (with artists, mediums, tracks) lookup is complete
06:43:33 [ianmcorvidae]
yeah
06:44:28 [ianmcorvidae]
oh, bah, your script probably says 'python' when I need 'python2'
06:44:47 [warp]
it usually says /usr/bin/env python
06:44:52 [ianmcorvidae]
yeah
06:44:56 [ianmcorvidae]
that's python3, I use Arch
06:45:09 [warp]
right. it shouldn't be difficult to make it compatible with both really.
06:45:39 [warp]
obviously you can run it with python2, but I'd accept patches with python3 compatibility fixes :)
06:45:45 [ianmcorvidae]
hehe
06:46:26 [ianmcorvidae]
heh, also needs 'virtualenv2' apparently
06:55:01 [bitmap]
when I try to run create_test_db on rika I just get failed: FATAL: Ident authentication failed for user "postgres"
06:57:28 [bitmap]
* bitmap wonders if it's possible to run tests on rika or if he should just set up his own server
07:11:52 [warp]
ianmcorvidae: curl 'http://rdf.greggkellogg.net/distiller?format=turtle&in_fmt=rdfa&uri=http://musicbrainz.org/release/aff4a693-5970-4e2e-bd46-e2ee49c22de7'
07:11:58 [warp]
ianmcorvidae: that looks fairly complete
07:12:22 [warp]
(though with some errors)
07:16:42 [ianmcorvidae]
bitmap: it should be possible to run tests, the DB should already be created
07:16:55 [ianmcorvidae]
you need to make a TEST entry in DBDefs of course, but
07:17:57 [bitmap]
* bitmap did so
07:18:11 [bitmap]
I was following the instructions here http://chatlogs.musicbrainz.org/musicbrainz-devel/2012/2012-05/2012-05-24.html
07:19:32 [ianmcorvidae]
yeah, you shouldn't need to run create_test_db.sh
07:20:32 [ianmcorvidae]
pick up after that I gues
07:21:16 [ianmcorvidae]
carton exec -Ilib -- prove t/
07:21:32 [ianmcorvidae]
or whatever filtering you want to do
07:21:35 [bitmap]
hm, the tests run now but it doesn't seem like there's anything in the database
07:21:37 [ianmcorvidae]
t/tests.t is usually all you need to run
07:21:40 [ianmcorvidae]
there won't be
07:21:49 [ianmcorvidae]
it creates an empty DB, and each test does a ROLLBACK after it's done
07:22:03 [bitmap]
ah, neat
07:22:08 [bitmap]
thanks :)
07:22:12 [ianmcorvidae]
basically each test runs in its own transaction which rolls back at the end
07:22:18 [ianmcorvidae]
no problem :)
07:31:28 [ianmcorvidae]
bitmap: see last comment on http://tickets.musicbrainz.org/browse/MBS-5119 -- the rel editor came up on an unrelated ticket, and jesus2099 has some comments that I don't know how to read :P
07:32:10 [nikki]
does anyone ever know? :P
07:32:24 [ianmcorvidae]
well, bitmap's got a better chance than the rest of us on this topic anyway :P
08:01:09 [ijabz]
ijabz has joined #musicbrainz-devel
08:03:41 [bitmap]
ianmcorvidae: added a comment to the whitespace ticket
08:03:47 [ianmcorvidae]
cool
08:04:02 [ianmcorvidae]
just doing ticket maintenance things as I work through scheduling slowly
08:04:05 [bitmap]
I guess the other ticket was http://tickets.musicbrainz.org/browse/MBS-5119 :)
08:16:58 [djce]
djce has joined #musicbrainz-devel
08:41:14 [ijabz]
ijabz has joined #musicbrainz-devel
08:49:56 [Mineo]
warp: there's actually a pep making /usr/bin/python a bit vague, so explicitly specifying a 2 or 3 is the way to go :)
08:54:11 [djce]
djce has joined #musicbrainz-devel
09:01:05 [bitmap]
I'm not sure how to work around https://github.com/SteveSanderson/knockout/issues/132
09:01:23 [bitmap]
whatever html parser the perl tests are using doesn't like that I have && in some attributes, but I can't escape them
09:01:49 [bitmap]
if I escape them, it isn't valid javascript
09:06:27 [bitmap]
oh, escaping them works in normal attributes, just not in <!-- comments -->
09:06:40 [bitmap]
disregard :)
09:22:48 [warp]
Mineo: I wasn't aware that python2 and python3 should be available on a default python install
09:22:59 [warp]
Mineo: good to know. do you know which PEP ?
09:24:20 [stefans_]
stefans_ has joined #musicbrainz-devel
09:40:30 [Mineo]
warp: it's http://www.python.org/dev/peps/pep-0394/
09:48:51 [ocharles]
morning
09:52:28 [Leftmost]
Leftmost has joined #musicbrainz-devel
09:54:01 [warp]
Mineo: thanks.
10:08:48 [djce]
djce has joined #musicbrainz-devel
10:15:40 [reosarevok]
reosarevok has joined #musicbrainz-devel
10:16:00 [reosarevok]
In case nobody has said so already (doubt it, but...) 502 and 500 in live like crazy
10:18:57 [ocharles]
i haven't seen anything reported
10:19:13 [ocharles]
because of course, we still don't have nagios giving me a single alert
10:19:44 [ocharles]
looks like the search server is non responsive?
10:20:03 [reosarevok]
Yeah, search was failing randomly too
10:20:08 [MBJenkins]
Project musicbrainz-server_beta build #69: FAILURE in 3 min 17 sec: http://ci.musicbrainz.org/job/musicbrainz-server_beta/69/
10:20:08 [MBJenkins]
* ollie: MBS-5029: When editing a relationship, refresh the old end points cover art
10:20:09 [MBJenkins]
* ollie: MBS-5029: Add comments to Edit/Relationship/Edit cover art refreshing logic
10:20:09 [MBJenkins]
* ollie: MBS-5078: Add the controlled for whitespace check on various tables
10:20:10 [MBJenkins]
* ollie: MBS-5071: Make sure spaces are collapsed in our custom Field::Text field
10:20:10 [MBJenkins]
* ollie: MBS-5079: Changing a work's language is only an auto-edit under certain conditions
10:20:11 [MBJenkins]
* ianmcorvidae: Disable showing the "Add Note" button when it does nothing.
10:20:11 [MBJenkins]
* nikki: More Amazon URL cleanup (MBS-4431, MBS-4698, MBS-4806)
10:20:12 [MBJenkins]
* nikki: Forgot to escape a dot
10:20:12 [MBJenkins]
* ollie: MBS-5079: Minor correction to test name
10:20:34 [reosarevok]
But that shouldn't cause 500 errors while editing URLs... we don't use the search server there (or do we?)
10:20:53 [ocharles]
no, i haven't seen those errors yet
10:20:56 [ocharles]
still hunting through logs
10:21:32 [ocharles]
i don't see a single timeout that's not /ws/2 in log
10:21:33 [ocharles]
logs*
10:21:41 [ocharles]
so i guess it's not a timeout?
10:22:13 [ocharles]
i don't see any errors in our postgresql log either
10:22:43 [ocharles]
so... need a bit more info
10:23:20 [ocharles]
http://stats.musicbrainz.org/webstats/nginx-rrd/drraw.cgi?Mode=view;Graph=1273836365.880 that certainly confirms what you're saying though
10:27:34 [reosarevok]
Dunno, editing URLs, loads of non-instant 502 coupled with some instant 500s
10:27:44 [reosarevok]
Can try to get some again by editing some more URLs :p
10:28:24 [reosarevok]
Oh, look
10:28:27 [reosarevok]
Didn't even need to edit
10:28:34 [reosarevok]
http://musicbrainz.org/url/01483c17-5364-4578-a323-167d11706862 gave me a 500 just by opening it
10:30:06 [reosarevok]
And 502 in http://musicbrainz.org/edit/relationship/delete?returnto=http%3A%2F%2Fmusicbrainz.org%2Furl%2Fe49b3b61-7e12-4762-bb1b-9aae2e399259&type1=url&type0=release_group&id=3976
10:30:12 [ocharles]
it is all feeling very slow
10:30:16 [reosarevok]
500 on refresh
10:43:33 [ianmcorvidae]
ocharles: noticing that djce's MBH-270 fix was committed about when this started happening
10:43:42 [ianmcorvidae]
ocharles: not sure if that's relevant, but it does correlate time-wise
10:44:02 [djce]
ianmcorvidae: ping me repeatedly if I need to fix something please
10:44:10 [djce]
in a meeting and reading CVs....
10:44:26 [ocharles]
ianmcorvidae: oh, that's interesting
10:44:42 [ocharles]
ianmcorvidae: what repo/commit?
10:44:50 [ianmcorvidae]
ocharles: frontend nginx 7846d1d
10:45:10 [ianmcorvidae]
ocharles: it's only SSL-related stuff and should only touch test, but
10:45:14 [ocharles]
mmm
10:45:35 [ocharles]
there's ee289 on lenny too
10:46:10 [ocharles]
djce: you only touched carl/lenny with this stuff, i take it?
10:46:18 [djce]
yes
10:46:27 [djce]
and only the hobbes vhost
10:46:29 [ianmcorvidae]
yeah, that's a bit later, though that does correlate with the brief drop in 5xx errors
10:46:30 [ocharles]
ok, I'm going to try temporarily reverting back a few revisions and see what happens
10:46:53 [ianmcorvidae]
(suggesting that the nginx restart there 'fixed' it temporarily)
10:53:55 [ocharles]
http://stats.musicbrainz.org/webstats/nginx-rrd/drraw.cgi?Mode=view;Template=1305563677.21559;Base=%2Fhome%2Fdjce%2Fpostgres-rrd%2Frrd%2F%2Fd%3Dmusicbrainz_db_20110516.rrd dafuq?
10:54:03 [ocharles]
I think it may well just be coincidental, this looks more worrying
10:54:49 [ianmcorvidae]
the huge climb in the past couple days is strange, yeah
10:55:05 [ianmcorvidae]
different timescale than the current 5xx stuff, of course, but
10:55:28 [ocharles]
true, but does match our release on 08-06
10:55:51 [ianmcorvidae]
at least some, yeah
10:56:03 [ianmcorvidae]
weird that it took a day after release to start climbing, I guess, but
10:56:27 [ocharles]
not necessarily a day, we could have just released at the end of the day
10:56:32 [ocharles]
in whatever timezone these charts are in
10:56:46 [ocharles]
hmm, they seem to only be an hour behind my system clock, so maybe that doesn't quite add up
10:57:05 [warp]
ocharles: that would be UTC
10:57:16 [warp]
your BST is one hour ahead of UTC.
10:57:29 [ianmcorvidae]
yeah, they're UTC
10:58:19 [warp]
ocharles: anything I can help with?
10:58:28 [ocharles]
warp: ideas would be good :)
10:58:33 [ocharles]
anything in nagios?
10:58:57 [djce]
djce has joined #musicbrainz-devel
10:59:06 [warp]
SEARCH is/was critical, don't see anything else.
10:59:43 [ocharles]
is or was?
11:00:03 [warp]
both :)
11:00:19 [warp]
zaphod/search-167-148 recovered, roobarb/search is still CRITICAL
11:00:25 [ocharles]
ah, k
11:00:48 [warp]
I only get the e-mails, I don't currently have access to the nagios dashboard
11:01:14 [ianmcorvidae]
5xx errors rrd graph is dropping slowly but definitely not decisively
11:01:55 [ocharles]
* ocharles nods
11:02:39 [warp]
there is no obviously load or memory problem on any of the hosts I looked at. bit you probably already checked that as well.
11:03:00 [ianmcorvidae]
it's sort of strange how roobarb appears to be doing nothing, given that it's also critical
11:03:08 [ianmcorvidae]
but I guess that could be load-balancing at work, eh
11:03:10 [ocharles]
that's the weird thing, the hosts all look fine, the logs don't contain errors
11:03:48 [warp]
ha, no roobarb is ok as well. I overlooked the recovery e-mail.
11:03:53 [ocharles]
:)
11:04:05 [ocharles]
doesn't nagios give you a dashboard page?
11:05:15 [ocharles]
the search servers are definitely not happy
11:05:23 [ocharles]
run 'tail -n0 -f /usr/local/mb_server-fastcgi/log/oddlog/current | grep -v 'warning' | tai64nlocal' in our frontends, and they are just scrolling errors
11:05:28 [warp]
ocharles: yes. but I do not have access to it.
11:05:34 [ocharles]
fan.tastic.
11:05:51 [warp]
ocharles: as in, I don't know the URL, and if I did I wouldn't know the passwords. (I think I recently dug up the url and tried).
11:06:34 [warp]
so the regular site should be working OK, as long as we don't do searches?
11:06:56 [ocharles]
you'd think so... but reosarevok is reporting timeouts everywhere
11:07:25 [warp]
I clicked on http://musicbrainz.org/release/aff4a693-5970-4e2e-bd46-e2ee49c22de7 and that is definitely taking its time.
11:07:39 [ocharles]
it's just weird that 'tail -f /etc/service/mb_server-fastcgi/log/oddlog/current | tai64nlocal | grep 'error' | grep -v 'ws/2' doesn't show errors for anything else
11:07:50 [warp]
(but it loads without a 5xx)
11:08:01 [warp]
(and a second time it is fast)
11:08:03 [ianmcorvidae]
hm, interestingly I just got that page really fast
11:08:04 [ianmcorvidae]
yeah
11:08:18 [ianmcorvidae]
suggests DB latency, second-and-later hit cache?
11:08:28 [ocharles]
* ocharles is definitely spending more time with graphite next week
11:08:57 [ianmcorvidae]
* ianmcorvidae is definitely getting myself logons for more of our servers so I can be more use than refreshing public graphs :)
11:09:03 [warp]
doing an artist search for "m-flo" also returns without trouble.
11:10:54 [warp]
man, that is a lot of complaining in that oddlog
11:13:17 [ocharles]
for some reason the cover art script seems to be doing queries almost every 5 seconds
11:13:18 [ocharles]
that seems odd
11:13:25 [ocharles]
(find_outdated_release)
11:14:30 [warp]
gah. a query from astro to a search server seemed to take very long. but subsequent queries (with different query= values) are instant.
11:14:35 [hrglgrmpf]
hrglgrmpf has joined #musicbrainz-devel
11:14:38 [warp]
this stuff is difficult to pin point.
11:15:43 [ocharles]
I've just stopped the cover art refresher
11:16:38 [ocharles]
I'm pretty sure that script was just starting up, querying, dieing, and then restarting. there is no logging for it at all
11:16:44 [ocharles]
(the log is discarded)
11:17:53 [ocharles]
no queries are taking over 2 seconds according to totoro, so i doubt that it's anything to do with that machine...
11:18:34 [warp]
the search server is either instant, or takes 30 seconds to respond. for the same query, with the same results.
11:21:19 [djce]
djce has joined #musicbrainz-devel
11:21:34 [ocharles]
warp: if you do tail -f /var/log/nginx/001-musicbrainz.access.log | grep 502 | grep -v '/ws' on all our machines, pingu is the only one struggling
11:21:37 [warp]
hrm, astro is connecting to a search server through lenny?
11:21:44 [ocharles]
so I'm going to take that out of rotation
11:22:17 [warp]
ok
11:22:50 [djce]
* djce is back, snacking on lunch. Ping me if I can help
11:23:05 [ocharles]
yea, that's just shifted it all to astro, as somewhat to be expected
11:24:09 [warp]
oh the search servers are load balanced. so presumably the 30 second responses are from the one which is b0rked, and the other responses are from the good server.
11:24:21 [ocharles]
warp: can we take the b0rked one out of rotation?
11:24:36 [warp]
as soon as I figure out which one, yes, I'd say so.
11:24:54 [ocharles]
:D
11:25:14 [warp]
yeah, .21 responds fast. .22 is slow.
11:25:19 [warp]
(or broken)
11:25:43 [ianmcorvidae]
roobarb is the problem one then, I guess
11:25:46 [warp]
djce: what is a good way to know which of lenny or carl is "active" ?
11:25:50 [ocharles]
warp: do you know what 10.1.1.247 is?
11:25:58 [djce]
warp: sudo ha-statall
11:26:00 [warp]
ocharles: yes, that's lenny.
11:26:04 [ocharles]
ah, ok
11:26:09 [djce]
ALL = active NONE = inactive
11:26:11 [warp]
ocharles: or lenny + carl.
11:26:14 [djce]
anything else = bad
11:26:57 [djce]
10.1.1.247 is the rate limiter service IP, provided by carl/lenny
11:26:59 [warp]
djce: awesome, thanks.
11:27:06 [djce]
(iir)
11:27:07 [djce]
(c)
11:28:05 [warp]
ocharles: ok, roobarb should be out of rotation.
11:28:47 [ocharles]
we're all quiet for 502s on frontends atm
11:28:56 [ocharles]
warp: hypothesis: the bad search server is constantly timing out, we have a backlog of request handlers that are all waiting for search responses, other requests thus timeout because they can't get serviced
11:29:02 [ocharles]
not entirely convinced, but it's plausable
11:30:25 [ocharles]
http://stats.musicbrainz.org/webstats/nginx-rrd/drraw.cgi?Mode=view;Graph=1273836365.880 i think we got it sorted
11:31:18 [warp]
right. I'm going to see if I can get this roobarb thing running again.
11:31:35 [ocharles]
still not a single 502 on any frontend
11:32:40 [warp]
restarted jetty on roobarb, it responds instantly again.
11:32:52 [ocharles]
so jetty isn't all that magic as we hoped :)
11:32:58 [warp]
nope
11:33:07 [warp]
I'll put it back in rotation.
11:33:45 [warp]
ok, roobarb should be in use again.
11:33:51 [ianmcorvidae]
I guess there's a nagios check we can at least get out of all this, perhaps something about response times for each seach server?
11:34:37 [ocharles]
we're certainly missing some reporting somewhere
11:35:27 [ianmcorvidae]
also: http://stats.musicbrainz.org/webstats/nginx-rrd/drraw.cgi?View=-1&Graph=1273836365.880&Start=end+-+4+hours&End=now&Mode=view&Shift=&Count= definitely fixed :)
11:35:34 [warp]
ocharles: I'll send an e-mail with what I did. roobarb being fussy was the only thing we found right? nothing else was considered troublesome and restarted, etc.. ?
11:36:01 [ianmcorvidae]
cover art script was restarted and the test https stuff was reverted, though I don't know that either of those was the actual problem
11:36:10 [ocharles]
we (me and ianmcorvidae) are a tad concerned by a large growth in tup_return from totoro
11:36:13 [ianmcorvidae]
(those reversions may have since been reverted, ocharles?)
11:36:19 [ocharles]
but totoro was deemed healthy
11:36:25 [warp]
ocharles: oh and you stopped some cover art thing.
11:36:26 [ocharles]
oh yea, I better bring lenny back up to HEAD
11:36:58 [ocharles]
back at ee289bf94b0e32c5e4a4b41d6ea8
11:37:45 [ocharles]
502's are coming back
11:37:53 [ocharles]
no wait
11:37:58 [ocharles]
i accidently scrolled my terminals :D
11:38:02 [ianmcorvidae]
hah
11:38:53 [warp]
haha
11:40:27 [ianmcorvidae]
hm, not seeing anything in the last release that should have even touched DB things
11:40:44 [ocharles]
nope, i did a diff too
11:41:17 [ianmcorvidae]
* ianmcorvidae sees what hosting things got touched in the last couple days
11:43:37 [ocharles]
rememeber it's not necessarily a change that we made, it could be something fell over and now different code paths are executing
11:43:49 [ianmcorvidae]
sure, but worth ruling it out
11:43:57 [ianmcorvidae]
I don't see anything in MBH
11:44:56 [ocharles]
* ocharles nods
11:50:10 [ianmcorvidae]
hm, could be the cover art script -- there's a big dip where it went down to more normal values while the script was off -- unless it hasn't been restarted?
11:50:52 [ianmcorvidae]
(it's back up to super-high, but it did drop just after it was turned off)
11:50:58 [warp]
ugh. I suck at reading nagios e-mails.
11:51:29 [ocharles]
it has been restarted, about 15 minutes ago
11:51:52 [ianmcorvidae]
http://stats.musicbrainz.org/webstats/nginx-rrd/drraw.cgi?View=-1&Template=1305563677.21559&Base=%2Fhome%2Fdjce%2Fpostgres-rrd%2Frrd%2F%2Fd%3Dmusicbrainz_db_20110516.rrd&Start=end+-+1+hours&End=now&Mode=view&Shift=&Count= so that drop in the middle is while it was off
11:52:03 [ocharles]
interesting
11:52:06 [warp]
so roobarb was CRITICAL, then recovered back to OK, then went CRITICAL again. so when I said "ha, no roobarb is ok as well. I overlooked the recovery e-mail.", I was WRONG.
11:52:18 [ocharles]
so now to find out why that script is not doing mucrh
11:52:19 [ocharles]
much*
11:55:13 [ocharles]
ianmcorvidae: apparently we have no out of date artwork
11:55:23 [ocharles]
so it's just running in a far too tight loop
14:23:19 [reoafk]
reoafk has joined #musicbrainz-devel
15:35:00 [stefans_]
stefans_ has joined #musicbrainz-devel
15:47:53 [plaintext]
plaintext has joined #musicbrainz-devel
15:48:52 [reoafk]
reoafk has joined #musicbrainz-devel
15:51:10 [ijabz]
ijabz has joined #musicbrainz-devel
16:08:34 [hawke_1]
hawke_1 has joined #musicbrainz-devel
16:12:05 [Leftmost]
Leftmost has joined #musicbrainz-devel
16:17:26 [hrglgrmp1]
hrglgrmp1 has joined #musicbrainz-devel
16:34:14 [reoafk]
reoafk has joined #musicbrainz-devel
16:41:43 [hrglgrmp1]
hrglgrmp1 has joined #musicbrainz-devel
16:55:39 [ruaok]
ruaok has joined #musicbrainz-devel
16:57:20 [hrglgrmpf]
hrglgrmpf has joined #musicbrainz-devel
17:06:30 [warp]
* warp got invited to an empty channel.
17:09:05 [Muz]
Maybe everyone left because you smell.
17:09:06 [reoafk]
Is that the "get lost" of IRC?
17:09:38 [ocharles]
warp: i was going to talk to you and ian there about something, but ian cleared it up in pm
17:09:48 [ocharles]
nice 6 hour lag though :P
17:10:10 [warp]
ocharles: invites are not very obvious in this client (with my config)
17:10:33 [ocharles]
evidently :)
17:10:55 [reoafk]
Maybe you should configure them so they mail you!
17:10:58 [reoafk]
* reoafk hides
17:11:09 [warp]
:P
17:13:57 [plaintext]
Hello
17:14:05 [plaintext]
I tried creating a manifest and moving a script to root/static/scripts from the tt file, but it didn't work
17:14:12 [plaintext]
then I tried to revert all the changes I made, and now the original won't work either :)
17:14:38 [plaintext]
I used statistics.js.manifest originally, and it used to work
17:15:00 [plaintext]
now I get this error: Uncaught TypeError: Object [object Object] has no method 'hashchange'
17:15:27 [plaintext]
here is the tt file, I removed all the scripts and added a single alert, but it doesn't get called
17:15:28 [plaintext]
https://github.com/balidani/musicbrainz-server/blob/master/root/logstatistics/category.tt
17:15:35 [plaintext]
does anyone know what's wrong?
17:24:57 [reoafk]
the code!
17:25:00 [reoafk]
*rimshot*
17:27:45 [ruaok]
plaintext: are you in the habit of committing your code at regular intervals?
17:27:59 [ruaok]
because if so, you can go back to the old version and see what you changed.
17:28:21 [ruaok]
you can examine each of the changes, undo each one, run it and narrow down where the trouble is.
17:29:42 [ocharles]
ruaok: guess you're aware of the downtimeyness today?
17:29:53 [ruaok]
I just sent a response in.
17:30:03 [ruaok]
nagios sent mail about what went down.
17:30:21 [ruaok]
it shouldn't have been hard to diagnose, unless the warning came too late.
17:30:39 [ruaok]
looks like we need to examine why nginx didn't fail over to dora.
17:33:19 [ocharles]
i don't get email from nagios
17:33:27 [ocharles]
i don't have any access to nagios, infact
17:33:42 [ruaok]
only dave and I do, since its hosted on non-mb hardware.
17:33:54 [ruaok]
open an MBH ticket for dave to ensure that nagios sends you mail too.
17:35:57 [ocharles]
For those who care: http://ocharles.org.uk/mb-arch-notes-1.pdf
17:35:59 [ocharles]
http://ocharles.org.uk/mb-arch-notes-2.pdf
17:36:08 [ocharles]
not too much there
17:36:47 [plaintext]
ruaok: nope, I didn't commit this yet before :(
17:37:11 [ruaok]
get in the habit of doing that. when you get something working, commit it.
17:37:11 [plaintext]
ruaok: but I don't even know what went wrong
17:37:18 [plaintext]
okay
17:37:23 [ruaok]
then you can always go back and find what went wrong
17:37:25 [plaintext]
lesson learned, I guess
17:37:47 [ruaok]
if you use vi, you can turn on eternal undo.
17:37:48 [ocharles]
you can always rebase to squash commits together
17:38:02 [ruaok]
even after you exit the editor, undo can persist.
17:38:15 [ruaok]
you can go back in time and forward in time. that way you could see what you changed.
17:38:29 [ruaok]
you could go back to an older version that worked, save it, then go forward again.
17:38:36 [ruaok]
then you can diff the old vs the new.
17:38:46 [plaintext]
well I did undo all of the changes up until it was working in category.tt
17:39:01 [plaintext]
then I even removed the js and added a single alert, but for some reason jquery gives a weird error
17:39:05 [ruaok]
for all of you vim users out there, check this out: http://oscon.tumblr.com/post/27706457604/oscon-instantly-better-vim
17:39:20 [ruaok]
paste the error
17:39:38 [plaintext]
Uncaught TypeError: Object [object Object] has no method 'hashchange'
17:39:44 [warp]
ruaok: I always got a response when testing the search server from astro (i.e. connecting to lenny). sometimes the response would take 30 seconds and sometimes it was instant. so, I assume lenny failed over to dora on each request after roobarb didn't respond within 30 seconds?
17:40:02 [plaintext]
and it was in jquery-1.5.1.js:869
17:40:15 [ruaok]
warp: ah, thats good to know.
17:40:36 [ruaok]
it sounds like jetty was partially failing and this not tripping nginx to fail roobarb out.
17:40:41 [warp]
ruaok: but it should probably take roobarb out of rotation after X amount of failed attempts?
17:41:00 [ruaok]
it should, but didn't. lets examine the config.
17:41:53 [ruaok]
max_fails=10 fail_timeout=10s
17:43:11 [ruaok]
10 fails in 10 seconds until its gets taken out.
17:43:33 [ruaok]
it sounds like it was getting at less than that.
17:43:42 [ruaok]
but how to know what the actual failures were?
17:44:30 [ruaok]
the docs example has 3f and 30s
17:44:40 [ruaok]
that seems more reasonable.
17:44:44 [ruaok]
thoughts?
17:45:10 [ocharles]
thoughts on?
17:45:13 [ocharles]
changing the settings to that?
17:45:17 [ruaok]
yep
17:45:27 [ocharles]
when does it come back into rotation?
17:45:36 [ocharles]
i think it was going out of rotation just fine, but coming back in very quickly
17:45:41 [ocharles]
but i'm not sure how it all works
17:47:37 [ruaok]
no idea. the docs dont say
17:48:42 [warp]
re
17:49:34 [warp]
ruaok: hrm. fail_timeout at 10 seconds seems odd, the request was given 30 seconds to finish, so that clause either isn't working or doesn't mean what I think.
17:49:52 [ruaok]
http://wiki.nginx.org/HttpUpstreamModule
17:50:46 [warp]
ah, that sounds aggressive
17:50:55 [ocharles]
time to get myself home
17:50:57 [warp]
so it needs to fail 10 times within 10 seconds.
17:51:13 [ruaok]
I doubt we ever triggered that.
17:51:15 [warp]
can it even register 1 fail in 10 seconds if a request is given 30 seconds? :)
17:51:42 [ruaok]
I'd be interested in trying 3 and 30s
17:51:44 [warp]
so, naively I would say lower max_fails and increase fail_timeout.
17:52:16 [warp]
i'd say 31s to avoid the edge case
17:52:26 [ruaok]
k
17:53:02 [ruaok]
ok, thats sounds sane enough to try.
17:53:16 [ruaok]
and we can artificially test this by taking one of the servers donw.
17:53:34 [warp]
yeah, probably.
17:53:58 [ruaok]
the question in my mind is where the 30 seconds comes from.
17:54:17 [ruaok]
is that the nginx timeout for a request to be forwarded?
17:54:36 [ruaok]
nothing from the search server should take more than a couple of seconds.
17:54:47 [ruaok]
the front ends only wait 5 seconds for a result.
17:55:03 [warp]
hm
17:55:45 [ocharles]
taking a server down won't give you a 30 second timeout
17:55:57 [ocharles]
replace the server with a 'sleep 40' server
17:56:13 [ocharles]
the search server regularly times out, see astro logs
17:56:30 [ruaok]
regularly when its "working" or "failing" ?
17:56:36 [ocharles]
hmm, actually less today
17:57:07 [ocharles]
i think the logs have just rotated though, which doesn't help ;)
17:57:08 [ocharles]
:)
17:57:52 [ocharles]
2012-08-10 17:57:33.610451500 LWP::UserAgent::get('LWP::UserAgent=HASH(0xc3893b8)', 'http://10.1.1.247:776/ws/2/recording/?max=15&type=recording&fmt=xml&offset=0&query=%28recording%3A%28Star~0.7%20AND%2069%20AND%20%5C%28Preziosso%20AND%20RMX%5C%29%29%20recording%3A%28Star~0.7%20AND%2069%29%20%29%20AND%20%28artist%3A%28Darek~0.7%29%20%29%20%20')
17:57:53 [ocharles]
there's one
17:58:10 [warp]
ruaok: is there anything on dora/roobarb themselves which would kill a request after 30 seconds?
17:58:12 [ocharles]
2012-08-10 17:57:35.485276500 LWP::UserAgent::get('LWP::UserAgent=HASH(0xdd062f8)', 'http://10.1.1.247:776/ws/2/recording/?max=25&type=recording&fmt=xml&offset=0&query=tnum%3A%281%29%20qdur%3A%28117%29%20artist%3A%283%20doors%20down%29%20track%3A%28kryptonite%29%20release%3A%28the%20better%20life%29'
17:58:15 [ocharles]
and another
17:58:16 [ruaok]
next time connect to the offending search server directly, without the load balancer.
17:58:23 [ruaok]
if the response is not immediate, it should fail out.
17:58:35 [ocharles]
also, any particular reason pingu has more weight? that difference of 1 weight seems to give a lot more traffic to pingu
17:59:29 [ruaok]
those weights are set so that the load on each server is about the same.
17:59:42 [ruaok]
they were all hovering around 4, so I would say thats perfect.
18:00:06 [ruaok]
why they have to be different, I have no idea.
18:00:28 [ocharles]
hmm, fair enough
18:00:32 [warp]
ruaok: oh wow, that is finetuning with some scary attention to detail :)
18:00:34 [ocharles]
anyway, reallly reallly off now
18:01:24 [ruaok]
:-)
18:02:01 [ruaok]
shall we go with 3 and 31s and see what happens?
18:02:20 [warp]
ocharles: you messed with lenny config at some point as well, right?
18:02:29 [warp]
ocharles: which commit did you checkout?
18:06:50 [warp]
(oh, he's off :)
18:07:11 [warp]
ruaok: are you testing 3 and 31s ?
18:07:19 [ruaok]
still pondering.
18:07:47 [hrglgrmp1]
hrglgrmp1 has joined #musicbrainz-devel
18:10:13 [ruaok]
I'm in #nginx asking questions there
18:10:55 [ruaok]
so far the nearly 400 people are all asleep. :)
18:12:32 [warp]
ruaok: somewhat common on freenode sadly. when it comes to responding to new people on an irc channel on freenode, I think musicbrainz is actually doing quite a good job :)
18:12:42 [ruaok]
srsly.
18:26:37 [hawke_1]
+1
18:26:57 [hawke_1]
without being a firehose like #ubuntu or #debian or any of the *really* populated channels
18:27:31 [ruaok]
even our customers who balked at getting support in IRC are quite happy.:)
18:46:05 [plaintext]
I tried to ask something in #spotify twice, and I didn't get a reply for 6-8 hours :)
18:56:23 [dukeleto]
dukeleto has joined #musicbrainz-devel
18:56:42 [dukeleto]
dukeleto has joined #musicbrainz-devel
19:01:28 [ocharles]
warp: the git reflog should function as a good audit log
19:01:31 [ocharles]
but we're back at master now
19:02:04 [djce]
djce has joined #musicbrainz-devel
19:38:17 [ruaok]
ruaok has joined #musicbrainz-devel
19:52:50 [ianmcorvidae]
plaintext: you need to load the jquery hashchange plugin -- but I don't know that you need that, that's for the fancy hash-urls for the timeline graph
19:53:01 [ianmcorvidae]
this was from a while ago, but )
19:53:47 [plaintext]
ianmcorvidae: I created a new manifest with just a single line for flot and it works now
19:54:04 [plaintext]
I'll try moving the script to a separate file later
19:54:55 [ianmcorvidae]
ah, yeah
20:00:16 [ocharles]
ruaok: one other thought, we might want to add a timeout on LWP for querying the search server
20:00:28 [ocharles]
so at least we don't start using up all the handlers
20:00:51 [ruaok]
ocharles: there is. its set to 5s. or at least we used to have that.
20:01:09 [ruaok]
if not, it needs to come back.
20:01:23 [ruaok]
djce: ping
20:02:26 [ocharles]
it is not timing out after 5 seconds
20:02:36 [ocharles]
because it's raising SIGALRM (which is our 30 second timeout)
20:03:00 [ocharles]
i think it might be worth adding $c->lwp, so we always use an lwp with sane settings
20:05:39 [ruaok]
and the LWP doesn't have an explicit timeout set?
20:05:42 [ruaok]
it should.
20:05:51 [ruaok]
want a ticket?
20:13:44 [ocharles]
if you want it fixed, yes :)
20:13:51 [ocharles]
i don't think lwp has a timeout set for search
20:13:56 [ocharles]
we set one for wikidocs when that fell over
20:14:06 [ruaok]
it did in classic. :) porting fail
20:14:34 [ocharles]
lack of tests to check we ported correctly fail
20:14:38 [ocharles]
and that was before me so.... blame shifted!
20:18:27 [ruaok]
you know? I swear I saw the search timeout after 5s.
20:18:32 [ruaok]
I've seen timeout errors.
20:19:17 [ocharles]
oh, there is one set to timeout, but one not...
20:19:19 [ocharles]
* ocharles investigates
20:19:54 [ruaok]
https://github.com/metabrainz/musicbrainz-server/blob/master/lib/MusicBrainz/Server/Data/Search.pm#L582
20:21:40 [ocharles]
odd, it looks like the web service should have the 5 second timeout, but that's specifically where I saw 30 second timeouts
20:21:40 [ocharles]
generally needs more investigating anyway, a ticket is still worth having
20:22:10 [ocharles]
yes, that's xml_serach
20:22:18 [ocharles]
notice that 'external_search' creates a $ua but doesn't set a timeout
20:22:22 [ocharles]
oh no
20:22:23 [ocharles]
i lie again
20:22:33 [ocharles]
sigh, friday evening, y u no good for programming?
20:22:38 [ruaok]
so, ticket or no ticket? :)
20:22:43 [ocharles]
ticket
20:22:47 [ruaok]
because its good for pub.
20:22:50 [ruaok]
k
20:23:01 [ocharles]
heh, no pub today, just jogging in this lovely hot indian summer evening
20:23:17 [ruaok]
yeah, I can't wait to go for a ride after work
20:26:25 [ruaok]
http://tickets.musicbrainz.org/browse/MBS-5124
20:48:11 [ruaok]
warp: you still up?
21:15:10 [djce]
ruaok: pong
21:26:29 [ruaok]
* ruaok returns
21:26:57 [ruaok]
djce: roobarb stopped responding to search requests earlier today.
21:27:18 [ruaok]
and everything tipped over because roobarb wasn't failed out by nginx.
21:28:02 [ruaok]
so, we're trying to figure out how we should change the proxy config to make it fail when a search server acts up
21:29:21 [ruaok]
we're considering updating the values for max_fails=10 fail_timeout=10s;
21:29:38 [ruaok]
and I'm also looking at proxy_read_timeout 60;
21:30:05 [ruaok]
but the docs are not very detailed, so its hard to figure out how it behaves
21:30:17 [ruaok]
any inights to the fail over process inside nginx?
21:31:39 [djce]
not really sorry, been thinking that we should have a proper test rig for it though
21:31:50 [ruaok]
* ruaok nods
21:32:14 [ruaok]
do you remember your reasoning for the 10, 10s setings?
21:33:04 [djce]
no, too long ago now
21:33:54 [ruaok]
I gotta see if nginx can give us logging info about this stuff
21:35:22 [djce]
I suspect it can if you whack the log level up high enough
21:35:27 [djce]
it can get pretty chatty
21:38:30 [ruaok]
* ruaok reads nginx dosc
21:57:33 [ruaok]
just taking roobarb down for a moment worked like a charm.
21:57:45 [ruaok]
I need to see if *any* errors were thrown by that. I dont think so.
21:58:23 [ruaok]
which is that is of little to no consequence, I'm inclined to restart the servers once a day.
21:58:29 [ruaok]
jetty that is.
21:58:38 [djce]
12 hours apart?
21:59:02 [djce]
or close together, both in a quiet time?
22:00:02 [ruaok]
I hadn't considered that.
22:00:09 [ruaok]
I'm certainly open for suggestions.
22:00:20 [ruaok]
an hour apart in quiet time sounds sane
22:01:22 [djce]
nod
22:02:23 [ruaok]
what was the unix cmd to remove a column from some data?
22:02:50 [djce]
cut
22:03:11 [djce]
e.g. if tabs then cut -f1-3,5- to remove column 4
22:03:24 [ruaok]
thk
22:03:37 [ruaok]
I want to nuke the url from the nginx logs
22:03:45 [ruaok]
makes them much easier to look for status codes.
22:04:01 [ruaok]
we do throw a lot of 400 errors because of bad queries.
22:04:45 [djce]
time for me to flake out. tomorrow, more house rewiring. fun fun fun
22:04:50 [djce]
nn all
22:04:58 [ruaok]
nn
22:05:02 [ruaok]
joy
22:07:33 [plaintext]
http://plaintext.mbsandbox.org/logstatistic/top_entities
22:08:16 [plaintext]
this works, but when I change jQuery's hide() and show() to hide("slow")/show("slow") the table loses the black line in the bottom after it's opened/closed once
22:08:33 [plaintext]
weird, but I guess the normal hide() and show() is ok
22:09:37 [ruaok]
hi plaintext
22:09:45 [plaintext]
hi ruaok
22:09:47 [ruaok]
I've got logs copying to the splunk machine now.
22:09:55 [ruaok]
next I need to sanitize them.
22:09:59 [plaintext]
great
22:10:07 [ruaok]
then we need to feed them to splunk.
22:10:24 [ruaok]
is there a command like "splunk add <file>" that I can call?
22:10:30 [plaintext]
I have no idea
22:10:40 [plaintext]
should I search for something like that?
22:10:47 [ruaok]
would you please?
22:10:51 [plaintext]
of course
22:11:53 [ruaok]
that page looks ok to me.
22:11:58 [ruaok]
looks really nice, actually. :)
22:12:08 [plaintext]
cool :)
22:12:26 [plaintext]
table column widths need some adjustment with js probably
22:12:42 [ruaok]
one thing that I see now is that we need to know the date ranges that this data applies to.
22:12:54 [plaintext]
hmm yes
22:13:03 [ruaok]
css is probably a better solution for table column widths.
22:13:34 [ruaok]
and for some reason the graph seems to sit a little lower than the table
22:14:03 [plaintext]
well I was thinking about js, because before we had graphs there were many tables underneath each other and different column widths looked ugly
22:14:12 [plaintext]
yeah, I'll look into that
22:14:37 [plaintext]
http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorfilesanddirectoriesusingtheCLI
22:14:45 [ruaok]
you can still manage that problem with css.
22:14:46 [plaintext]
the "add oneshot" command looks good here
22:14:49 [plaintext]
really?
22:14:52 [ruaok]
you should prefer to do things in css in most cases.
22:14:58 [ruaok]
leave JS to things that really need it
22:15:11 [plaintext]
hmm
22:15:14 [ruaok]
give the table cells a % width.
22:15:34 [ruaok]
and if each table is the same % wrt to its parent, they should all line up nicely.
22:16:28 [plaintext]
I overthought this
22:16:42 [ruaok]
yep, one shot is what I was looking for.
22:16:53 [ruaok]
plaintext: easy to do. always prefer the simplest solution possible.
22:17:13 [ruaok]
that will make life easier and you can have complexity for the really hard stuff.
22:17:18 [plaintext]
yeah
22:17:29 [ruaok]
my CS teacher once said: "Complexity is a scarce resource"
22:17:42 [plaintext]
that's good :)
22:17:42 [ruaok]
you can only have so much before things are just too complicated.
22:17:46 [ruaok]
:-)
22:20:24 [plaintext]
by the way what I currently do is a bit wasteful
22:20:35 [plaintext]
all the data is loaded through tempalte toolkit
22:20:50 [plaintext]
and just for graphs every report is loaded again via "/logstatistic/json/" + id
22:20:59 [plaintext]
just to get the extra details on how to display stuff
22:22:22 [ruaok]
you should populate the tables from the json data.
22:22:30 [plaintext]
from js?
22:23:08 [ruaok]
yeah.
22:23:24 [plaintext]
okay, I'll work on that tomorrow
22:23:27 [ruaok]
this somewhat contradicts what I just said, ironically enough.
22:23:41 [plaintext]
hehe
22:23:42 [ruaok]
but saving bandwidth vs a little bit of code complexity is the right tradeoff here.
22:24:17 [plaintext]
I haven't done much js/jQuery before, but jQuery feels really nice so far
22:25:35 [ruaok]
js without jquery is a real pain. no fun at all.
22:25:43 [ruaok]
but with it, it starts becoming fun.
22:25:59 [plaintext]
hehe
22:28:49 [plaintext]
this is my js signature: http://pastebin.com/raw.php?i=9nqEqzD5
22:30:30 [ruaok]
signature for what?
22:30:36 [plaintext]
my name
22:30:50 [plaintext]
I was really tired the other day :)
22:30:58 [ruaok]
lol
22:57:00 [the_metalgamer]
the_metalgamer has joined #musicbrainz-devel
23:32:51 [ruaok]
plaintext: still here?
23:33:08 [ruaok]
I think I am about to try feeding splunk a log via the oneshot interface
23:33:29 [plaintext]
ruaok: yeah
23:33:40 [plaintext]
hope it works
23:34:33 [ruaok]
I'm wondering how it will know what it import the data as.
23:35:23 [plaintext]
you can specify a sourcetype if that's what you mean
23:36:16 [plaintext]
and I saved an "nginx_log" source type when I was adding logs
23:36:31 [ruaok]
yes, that would be perfect.
23:43:07 [ruaok]
hmm. permissions problems.
23:43:20 [ruaok]
I need to leave now. I'll pick this up when I get home.
23:43:25 [plaintext]
okay
23:43:32 [ruaok]
when you wake up, look for new data. :)
23:43:44 [plaintext]
yay :)
23:43:54 [plaintext]
will you erase the old ones?
23:43:55 [ruaok]
<poof>
23:44:07 [plaintext]
I guess I'll see :P
23:45:23 [demosdemon]
heh, he's a mysterious one