IRC log of musicbrainz-devel on 2015-03-14

Timestamps are in UTC.

00:18:30 [ruaok]
ruaok has joined #musicbrainz-devel
00:27:49 [kahu]
kahu has joined #musicbrainz-devel
00:30:30 [ruaok]
aight. 2L of vino, half a bottle of tequila and a 4 hour bath. much better now.
00:31:05 [derwin]
hear hear
00:31:11 [derwin]
beer o clock over here.
00:38:19 [ruaok]
bed o clock for sane people here.
00:38:33 [ruaok]
so, we’re off to bed on the early side.
00:38:44 [ruaok]
thanks bitmap, I’ll have a look in the morning.
00:56:54 [Nyanko-sensei]
Nyanko-sensei has joined #musicbrainz-devel
01:48:58 [LordSputnik]
LordSputnik has left #musicbrainz-devel
01:49:40 [dufferzafar]
dufferzafar has joined #musicbrainz-devel
02:35:42 [djinni`]
djinni` has joined #musicbrainz-devel
04:05:09 [ianmcorvidae]
* ianmcorvidae notes for the concerned, it looks like modbot got stuck on something, so things have been held up for a few hours. I've killed it now, but replication won't have happened and edits might have been slow in closing
04:05:29 [ianmcorvidae]
most likely it'll fix itself in an hour
04:05:39 [ianmcorvidae]
(well, replication is happening now. but edits closing)
04:27:41 [KRS-Cuan]
KRS-Cuan has joined #musicbrainz-devel
04:36:23 [Leftmost]
Is Modbot drunk again?
04:36:29 [ianmcorvidae]
probably
04:38:08 [kahu]
kahu has joined #musicbrainz-devel
04:38:41 [ianmcorvidae]
* ianmcorvidae is running it once on beta
05:07:18 [ujjwal]
ujjwal has joined #musicbrainz-devel
05:32:28 [bitmap]
fyi vm.swappiness has been set to 0 on totoro to see if that can reduce the thrashing and stabilize things at all
05:35:46 [ujjwal]
ujjwal has joined #musicbrainz-devel
05:51:24 [JesseW]
JesseW has joined #musicbrainz-devel
06:03:38 [ujjwal]
ujjwal has joined #musicbrainz-devel
06:25:18 [JesseW]
* JesseW is going to do some test refactoring as a nice, relaxing way to avoid social drama elsewhere in their life...
06:25:58 [dufferzafar]
Building something like this using AcousticBrainz should be easy right? http://www.christianpeccei.com/musicmap/
06:26:33 [dufferzafar]
I mean, instead of computing features from files, we just query AB with an MBID of the song.
06:27:48 [JesseW]
dufferzafar: or better, work from a database dump
06:28:07 [JesseW]
it would be really neat to process a whole database dump that way... :-)
06:28:11 [dufferzafar]
oh yeah, didn't think about that
06:28:16 [JesseW]
presumably scrollable and zoomable...
06:28:40 [JesseW]
* JesseW doesn't want to make it myself, but would certainly enjoy the results!
06:30:51 [dufferzafar]
the dump is 34 GBs :/
06:41:13 [JesseW]
dufferzafar: eh, it's hard to find a less than 1TB drive lately (at least, where I hang around it is...)
06:41:29 [JesseW]
34 GB doesn't sound too bad. :-)
06:42:09 [dufferzafar]
my internet connection will take forever to download that
06:42:39 [JesseW]
dufferzafar: ah, that problem. Hm -- try to work on it via a cloud service? Cloudant or Heroku maybe?
06:42:52 [JesseW]
(both of those have free options)
06:43:41 [JesseW]
or request a few hundred mbids via queries, develop the code, then ask someone else to run it on the full database (i.e. me, mabe)
06:43:48 [JesseW]
s/mabe/maybe/
06:48:56 [bitmap]
hmm. I guess we haven't tweaked disk read-ahead at all on totoro
07:02:04 [bitmap]
set to 256 right now
07:02:07 [krisalpha]
krisalpha has joined #musicbrainz-devel
07:12:13 [JesseW]
hm, neat.
07:18:16 [bitmap]
perhaps...conservatively bump to 1024 and see if i/o improves? dunno
07:37:54 [ujjwal]
can we add calendar to select a date in musicbrainz?
07:40:51 [JesseW]
JesseW has joined #musicbrainz-devel
07:44:44 [JesseW]
ujjwal: a calendar would be very nice, yes. Maybe as a user-script, first?
07:45:15 [ianmcorvidae]
remember that we support a more complicated sort of date than most calendar-selection widgets, though
07:45:37 [JesseW]
There is a (very useful) user-script to parse dates and convert them to the year-month-day format expected... I can try and dig up a link
07:45:46 [ianmcorvidae]
all of year, month, and day are optional, and you can even e.g. have an empty year with a month and day
07:46:07 [ianmcorvidae]
where most premade widgets assume you only ever want a full date
07:46:12 [nikki]
https://bitbucket.org/96187/userscripts/src/HEAD/paste-a-date/paste-a-date.user.js
07:46:15 [ianmcorvidae]
so the UI would require some work
07:46:31 [ianmcorvidae]
and yeah, paste-a-date is good. we really need to get that into the server at some point
07:46:34 [JesseW]
what does (empty year, filled in month and day) supposed to be intrepreted as?
07:46:56 [ianmcorvidae]
the event happened on that month and day in an unknown year
07:47:07 [nikki]
e.g. someone's birthday is known, but not the year
07:47:11 [ianmcorvidae]
e.g. for the case where you know someone's birthday but not their age/the year
07:47:14 [ianmcorvidae]
yeah
07:47:23 [JesseW]
Ah, that makes sense. I hadn't thought of birth dates.
07:47:27 [ianmcorvidae]
technically you could even have something like known year, unknown month, known day
07:47:27 [nikki]
I'm not really sure a calendar would be that helpful, most of our dates are historical, often decades ago
07:47:41 [ianmcorvidae]
but that seems even less likely to be useful :P
07:48:49 [JesseW]
* JesseW wonders if you can have month-day only for both birth and death dates
07:48:55 [nikki]
yes
07:48:59 [nikki]
for any date fields
07:49:10 [nikki]
(maybe not release dates? not sure there)
07:49:10 [ianmcorvidae]
the only date field that has to be completely filled out is the one on the user profile page
07:49:16 [ianmcorvidae]
as far as I know, anyway
07:49:24 [ujjwal]
ujjwal has joined #musicbrainz-devel
07:49:26 [ianmcorvidae]
(and that one's obviously optional as well, but :P)
07:53:46 [JesseW]
interesting -- it will complain if it can determine that the end date is after the start date, though
07:54:05 [ianmcorvidae]
yes, but it can only determine that if there's a year at least
07:55:01 [JesseW]
and we don't support BC/BCE, I see.
07:55:10 [ianmcorvidae]
well, not properly
07:55:32 [ianmcorvidae]
chirlu can go into detail about the exact peculiarities of that
07:55:32 [JesseW]
it seems to object to negative values, period
07:55:52 [ianmcorvidae]
I believe we support them in some certain weird places... there's also always the issue of what to consider 0
07:56:08 [JesseW]
that also appears to be disallowed
07:56:12 [ianmcorvidae]
(invalid value, or is that BCE 1? if it's the latter, then -1 is BCE 2, etc., but people don't usually think that way, etc.)
07:56:25 [ianmcorvidae]
it's not properly implemented, there's some way you can trick it into allowing them
07:56:28 [ianmcorvidae]
don't remember what
07:56:30 [JesseW]
hm
07:56:31 [ianmcorvidae]
it's essentially unsupported though
07:56:40 [ianmcorvidae]
it might allow it with incomplete dates or something
07:58:00 [JesseW]
btw, do I need to do more to get my schema-change PR accepted into the upcoming schema-change, or is it sufficient? (you haven't had time to look at it is also a perfectly OK answer, of course)
07:58:58 [ianmcorvidae]
I haven't had a chance to look at it. it (after nikki's, since it was first) has been in my 'secondary goal' column for a bit, but getting the sitemaps stuff that was supposed to be done months ago done is first
08:00:29 [JesseW]
makes lots of sense.
08:01:24 [nikki]
* nikki will be very grumpy if later schema changes get merged first and cause conflicts with hers, so is pleased to hear that hers is higher up the list :P
08:01:38 [ianmcorvidae]
yup. so it's in the category of things I work on when I'm burned out for the day on the sitemaps work, or done what I set aside for that day already
08:02:03 [JesseW]
I just want to avoid missing the window when it can go in. I think I'll just do Work collections this schema change (unless it makes much more sense to toss in the other entities)
08:02:20 [ianmcorvidae]
smaller with that sort of work is often better
08:02:31 [ianmcorvidae]
if you can do something small really well, better than halfassing something bigger :)
08:02:46 [ianmcorvidae]
given there's only two chances a year to correct mistakes and all that
08:03:00 [ianmcorvidae]
(at least, if they're mistakes at a schema level that can't be fixed in code)
08:03:13 [JesseW]
well this is pretty small. Just one new table (with a few constraints) and a trivial change to an existing constraint
08:04:29 [nikki]
given that the schema changes should be pretty much identical for them all, it might be worth doing them anyway, even if you don't finish the code bits yet
08:04:53 [JesseW]
yep, that was the reason I thought it might be worth putting them in
08:05:09 [nikki]
(the deadline is for the actual schema bits, the interface parts can go in whenever as long as the schema supports it
08:05:11 [nikki]
)
08:05:39 [ianmcorvidae]
I will note that it's a lot *better* if the interface stuff makes it in the same time though :P
08:05:47 [JesseW]
* JesseW decides what the heck, I'll write them up. Shall I put them in the same PR, or a separate one, or multiple ones?
08:05:49 [nikki]
that much is true XD
08:05:53 [ianmcorvidae]
* ianmcorvidae remembers a few things we let sit for... rather a while >_<
08:06:03 [JesseW]
I'm pretty close to having the interface stuff done, too.
08:06:07 [ianmcorvidae]
same PR is fine, I'd say, as long as it still hasn't seen any review
08:06:16 [JesseW]
ok.
08:06:22 [ianmcorvidae]
if it has, still probably fine, just be especially sure to mention you've added a lot :)
08:06:32 [JesseW]
they will be very identical.
08:07:07 [JesseW]
the reason I had been waiting was to avoid having to duplicate fixes, once such review does happen
08:07:26 [JesseW]
nikki: can you link me to your PR, for comparison?
08:08:36 [nikki]
pfft :P it'll take me just as long to navigate to it :P
08:08:50 [ianmcorvidae]
it's the one about aliases, lol
08:08:50 [nikki]
https://bitbucket.org/metabrainz/musicbrainz-server/pull-request/1484/extend-alias-support-to-recordings/diff anyway
08:08:56 [ianmcorvidae]
yeah, that one :P
08:09:02 [JesseW]
thank you.
08:10:49 [JesseW]
ah, this combines the schema-change stuff and the rest of the code. Hm, I'll see if I can find a schema-change only PR...
08:11:18 [ujjwal]
ianmcorvidae, for solr gsoc project, I see mineo has added TODOs to his repo, I would like to know which are some additional features that are yet to be implemented
08:12:17 [JesseW]
ujjwal: for SOLR, I think "getting it functioning" is the main thing. :-)
08:12:35 [ianmcorvidae]
JesseW: s/functioning/finished/
08:12:40 [ianmcorvidae]
the parts that are done worked great :)
08:13:01 [JesseW]
yes, that's more accurate
08:13:31 [ianmcorvidae]
but yes, the goal is doing what needs doing to let it replace the existing search server. how that corresponds to whatever got added as a TODO I don't know
08:14:52 [nikki]
ooh I see bitmap added a comment on mine like a week ago
08:17:53 [JesseW]
any thoughts on whether to put each entity in two separate files, or just have two big(er) files with all the entities...
08:18:23 [nikki]
files?
08:18:32 [ianmcorvidae]
however it's more easily understood, I guess. I'd lean toward two bigger files
08:18:35 [ianmcorvidae]
update scripts I assume
08:18:39 [JesseW]
yep
08:18:43 [JesseW]
ok, two bigger files it is
08:18:44 [nikki]
oh, then yeah, I wouldn't split the upgrade scripts
08:19:17 [leyyin]
leyyin has joined #musicbrainz-devel
08:19:21 [ianmcorvidae]
as far as it actually being run, the scripts are compiled together to run in one transaction (or, well, two, one for the "run everywhere" and one for the "only on standalone and master instances", but)
08:19:37 [ianmcorvidae]
so it's organizational, and I think they're more similar than different. so I'd put them together :)
08:20:48 [JesseW]
nods
08:22:55 [JesseW]
and I shouldn't put BEGIN and COMMIT lines in them, should I?
08:23:24 [ianmcorvidae]
no, those should be there but will be stripped by the compilation process
08:23:31 [ianmcorvidae]
likewise the \set ON_ERROR_STOP line
08:23:48 [JesseW]
ok, good
08:24:04 [ianmcorvidae]
they should work correctly and within a transaction when run by themselves, so they should have the begin/commit, but the compilation will remove those bits too
08:24:20 [JesseW]
just one transaction per file, though...
08:24:23 [JesseW]
?
08:24:36 [ianmcorvidae]
yeah. no real reason to split it up further than that
08:25:14 [yeeeargh]
yeeeargh has joined #musicbrainz-devel
08:25:24 [JesseW]
nods
08:28:13 [JesseW]
the primary key constraint goes on the slaves (unlike the other constraints), right?
08:28:22 [ianmcorvidae]
yes
08:33:37 [JesseW]
I'm not going to make URL collections. :-)
08:33:47 [JesseW]
but I'll do all 9 other ones...
08:34:18 [JesseW]
area, artist, instrument, label, place, recording, release_group, series, and work
08:35:16 [ianmcorvidae]
* ianmcorvidae wonders about area and instrument since they're limited in editorship other ways
08:35:27 [ianmcorvidae]
but maybe that doesn't matter :)
08:36:05 [nikki]
there was the idea of moving subscriptions to be collections
08:36:10 [JesseW]
Instrument collections could be a nice way to signify instruments you know how to play, or are interested in music containing them
08:37:16 [JesseW]
Area collections could be used for geographical areas whose music you want to collect, or whose artists you might be able to go see
08:37:43 [nikki]
which now has its own ticket - http://tickets.musicbrainz.org/browse/MBS-8282
08:37:46 [ianmcorvidae]
yeah. both of those are recursing a lot more than we do (and probably, are reasonably able) as far as something like subscriptions
08:38:17 [ianmcorvidae]
so they won't very likely be *directly* useful, within the site, as those things
08:38:31 [ianmcorvidae]
but there's probably not any real reason to exclude them, anyway, they'll just be less useful :P
08:38:51 [nikki]
the recursion stuff is a separate problem :P
08:39:28 [ianmcorvidae]
sure. doesn't mean it isn't relevant :P
08:39:29 [nikki]
I mean, I can see it being useful to be able to subscribe to an area or instrument that people keep misselecting
08:39:46 [JesseW]
and could presumably be solved with further schema changes (i.e. the addition of more indexes, etc.)
08:41:03 [JesseW]
I'm not sure what the use of release group collections might be...
08:41:45 [diana_olhovik]
diana_olhovik has joined #musicbrainz-devel
08:41:52 [ianmcorvidae]
release collections where you don't know which exact edition you care about, most likely
08:41:52 [nikki]
heh. https://beta.musicbrainz.org/area/2f555dfa-7864-47ef-a048-a077f5ed5fb0/artists those probably aren't from japan :P
08:41:54 [JesseW]
I'm just making them all toplevel collection types
08:42:09 [JesseW]
heh
08:42:24 [ianmcorvidae]
though that also skirts the issue of the much more invasive change that is multi-entity-type collections
08:42:32 [ianmcorvidae]
which should not happen just yet, probably :P
08:42:38 [JesseW]
not yet, no
08:43:09 [nikki]
yeah, being able to add entities to *a* collection is good enough to start with XD
08:43:14 [nikki]
we can worry about combining them later
08:43:42 [ianmcorvidae]
yeah XD
08:43:42 [nikki]
if people have to make "foo bar (artists)" and "foo bar (releases)", they'll live
08:45:05 [JesseW]
yep
08:45:37 [JesseW]
can I make multiple ADD CONSTRAINTs in the same ALTER TABLE?
08:46:45 [JesseW]
looks like it
08:48:15 [diana_olhovik]
diana_olhovik has joined #musicbrainz-devel
09:02:04 [Nyanko-sensei]
nikki: U.S.A., Japan >_>
09:02:36 [JesseW]
* JesseW has updated their PR: https://bitbucket.org/metabrainz/musicbrainz-server/pull-request/1522/schema-changes-to-add-collections-for-all/commits
09:02:38 [nikki]
* nikki isn't sure what you mean
09:02:56 [JesseW]
* JesseW will now try sleeping again
09:03:38 [nikki]
(if you're explaining it, then yes, I was aware of it, I went to check if anyone had misselected it after mentioning people misselecting things :P)
09:04:26 [Nyanko-sensei]
* Nyanko-sensei wanted to joke on people misselecting things even if it says japan
09:04:34 [nikki]
oh
09:04:37 [nikki]
well, ok :P
09:06:57 [Freso]
"JesseW | Instrument collections could be a nice way to signify instruments you know how to play, or are interested in music containing them" -- or, y'know, for the instrument collectors out there, to note the kinds of instruments they have.
09:07:52 [ianmcorvidae]
haha. that'd be useful if we also added a thing for an entry-specific note per item in collection, which is another thing that's historically been suggested
09:08:27 [ianmcorvidae]
re: instrument collectors' collections being stored as instrument collections
09:09:01 [Freso]
Well, it'd be more useful. Doesn't mean it couldn't be useful at all.
09:09:07 [nikki]
http://tickets.musicbrainz.org/browse/MBS-1658 I guess
09:10:15 [Freso]
Maybe people like alastairp/the MTG crowd could find instrument collections useful if they're doing some research on a group of instruments.
09:10:40 [Freso]
Well, I just wanted to affirm that instruments collections can theoretically be useful. :)
09:11:27 [nikki]
maybe someone just wants to make a collection of instruments they think look or sound funny :P
09:12:21 [ianmcorvidae]
or perhaps place names? :P
09:12:24 [nikki]
yes!
09:12:25 [ianmcorvidae]
(i.e. areas, I mean)
09:12:30 [ianmcorvidae]
grobbendonk
09:12:33 [nikki]
* nikki is totally gonna do that now
09:12:34 [ianmcorvidae]
and... there was another one
09:12:49 [nikki]
buffelspruit
09:12:50 [ianmcorvidae]
well, there's a bunch
09:12:52 [ianmcorvidae]
yes!
09:16:58 [chirlu`]
chirlu` has joined #musicbrainz-devel
09:18:04 [alastairp]
it could be interesting to say “here’s a group of instruments that are common in culture x"
09:18:14 [ianmcorvidae]
yeah, definitely
09:18:30 [alastairp]
though, we have things like “spanish acoustic guitar” in some cds which are definitley not from our spanish collection
09:18:47 [ianmcorvidae]
it's something tags would make sense for, but presumably normal users can't tag instruments and areas
09:18:52 [nikki]
to some extent we can do that by linking to areas, but cultures aren't areas so it doesn't work perfectly
09:18:58 [ianmcorvidae]
(I haven't confirmed that's true. or if we even have tags on those. but anyway)
09:18:59 [nikki]
I thought they can?
09:19:04 [nikki]
we do have tags now
09:19:08 [ianmcorvidae]
you probably know better than I do XD
09:19:11 [alastairp]
our experience with tags are that they’re terrible
09:19:16 [ianmcorvidae]
but tags are pretty imperfect
09:19:20 [alastairp]
mostly because people can’t spell
09:19:20 [ianmcorvidae]
to say the least
09:19:40 [ianmcorvidae]
collections can be as curated as the user who makes them want them to be, which you can't enforce with tags anyone can work with
09:20:12 [ianmcorvidae]
(well, I mean, you can have it be "instruments tagged X by Y user" but then that's basically a collection, so :P)
09:20:59 [ianmcorvidae]
culturally-centered collections are one of the things I think multientity collections could be really cool for too
09:21:08 [ianmcorvidae]
(with the same caveats about "not this schema change" of course)
09:21:42 [ianmcorvidae]
being able to have releases, works, labels, artists, etc. that all are good exemplars of whatever your theme is would be cool :)
09:21:46 [nikki]
hm, culturally centred stuff sounds like something I was experimenting with ages ago
09:22:20 [ianmcorvidae]
ideally we or someone third-party would also write a cool interface that lets you look at them more smoothly than just a bunch of entity-specific lists, but
09:23:17 [chirlu`]
I looked somewhat into the notify-collection-subscribers-about-deleted-entities thing and found no place where we delete from editor_subscribe_artist_deleted and similar tables.
09:23:27 [nikki]
I was experimenting with being able to pick a country and have it display some page with a variety of things from that country, like recent releases, artists with birthdays, etc (I mean, not that we even have that on a non-country-specific level but)
09:23:31 [chirlu`]
Which makes me believe it is never done.
09:23:32 [ianmcorvidae]
chirlu`: we don't delete from them, would be why :)
09:23:43 [ianmcorvidae]
there's a whole bunch of old entities in those on prod
09:23:48 [nikki]
mbidssss :D
09:24:36 [chirlu`]
I guess it is fine to have the artist_deletion info forever.
09:24:40 [ianmcorvidae]
as nikki alludes, we've considered using that data among other sources to recover MBIDs that have gone away, should we ever set something up to e.g. look at deleted entity edits
09:25:00 [chirlu`]
But storing who was subscribed to it at the time of deletion? :)
09:25:08 [ianmcorvidae]
oh, right
09:25:08 [ianmcorvidae]
hm
09:25:22 [ianmcorvidae]
yeah, deleting from the subscription tables would probably make some sense, huh :)
09:25:29 [chirlu`]
Doesn’t happen that often, I guess.
09:25:30 [ianmcorvidae]
even if not the entity ones
09:25:30 [nikki]
I also wanted to be able to return different error codes for "we had that but deleted it" versus "uh, don't recognise that"
09:25:39 [ianmcorvidae]
well, it also applies to merges, I think
09:25:42 [ianmcorvidae]
which might be more commen
09:25:45 [ianmcorvidae]
common*
09:25:57 [chirlu`]
There is Gone (410 I believe).
09:26:10 [ianmcorvidae]
yup
09:26:12 [ianmcorvidae]
was gonna suggest that XD
09:26:22 [nikki]
yeah, but right now we don't know whether we had it and deleted it or just don't recognise it XD
09:26:44 [nikki]
gah, I'm not motivated to work on that schema change thing right now :(
09:26:47 [ianmcorvidae]
we should clearly find somewhere in MBS to use 418, but :P
09:26:55 [chirlu`]
We know, we just don’t check it.
09:27:10 [chirlu`]
(For some relatively-recently-deleted entities.)
09:27:11 [nikki]
we do?
09:27:14 [ianmcorvidae]
we only kinda know
09:27:29 [ianmcorvidae]
yeah, for stuff that's new enough to be in the _deletion tables and which has subscriptions, we could look at it
09:27:44 [ianmcorvidae]
but that's a lot of conditions :P
09:28:26 [chirlu`]
I think it’s always put into _deletion, whether someone was subscribed to it or not.
09:28:35 [chirlu`]
(Of course, limited to artist/label/series.)
09:28:43 [ianmcorvidae]
yeah, I meant the type restriction
09:28:56 [ianmcorvidae]
has the possibility of subscriptions, I guess I mean, not 'has subscriptions' :)
09:29:39 [chirlu`]
I was wondering if we really need separate artist_deletion, label_deletion, etc. tables.
09:29:49 [chirlu`]
They don’t have foreign keys after all.
09:30:38 [ianmcorvidae]
I believe it's done that way so as to be 'cleaner'. whether we agree with that assessment I don't know
09:30:58 [nikki]
ianmcorvidae: btw, http://tickets.musicbrainz.org/browse/MBS-8294, iirc you had some idea about the data column or something that you might want to add as a comment
09:31:19 [ianmcorvidae]
ah, right
09:31:44 [ianmcorvidae]
my thought was if we wanted to we could put the data column in a separate table (or at least, somehow separate it for the dump), since that'll be a lot of the data in the edit table
09:32:25 [ianmcorvidae]
and that way we could still have stuff like edit status, type, editor, etc. etc.
09:32:35 [ianmcorvidae]
"we" as in whoever downloads this to use, I mean
09:33:33 [ianmcorvidae]
obviously that'll produce edits you can't yourself display, but if you just want to do something like look at distributions of edits by users or types or statuses or whatever
09:34:39 [ianmcorvidae]
I'll also make my own comment as far as the option of actually outright partitioning the edit table
09:34:48 [ianmcorvidae]
which is more technical but possibly cleaner :)
09:34:53 [nikki]
heh
09:35:43 [nikki]
I remember seeing something about postgres and being able to have multiple tables where it knows which table to store the data in and my first thought was split the edit table by year XD
09:35:47 [nikki]
(or something)
09:35:51 [ianmcorvidae]
yup
09:35:54 [ianmcorvidae]
that's exactly what I'm thinking of XD
09:36:15 [nikki]
yeah, thought it was probably was
09:36:34 [nikki]
just amused that when I came across it, the edit table was the exactly what I thought of
09:40:19 [chirlu`]
Is there a legitimate use case for adding standalone recordings inline (from a search)? Wondering if I should ticket for its removal.
09:40:44 [chirlu`]
Because it’s often misused. :-/
09:40:49 [ianmcorvidae]
perhaps adding a work and adding videos of it being performed, on youtube, at the same time?
09:41:13 [ianmcorvidae]
standalones are always a bit funky, prior distribution as far as "using a standalone is the right choice" is pretty bad
09:42:04 [chirlu`]
(Example misuse: https://beta.musicbrainz.org/edit/31637273)
09:42:41 [ianmcorvidae]
that's not standalones at all, though?
09:42:53 [ianmcorvidae]
I mean, obviously it's not a valid release
09:43:33 [chirlu`]
Most of the recordings there were added as standalones by the editor.
09:43:47 [chirlu`]
Which is why they are still there even though the medium is gone.
09:44:05 [ianmcorvidae]
right, and I guess that's a problem because they're obnoxious to merge afterwards?
09:44:18 [ianmcorvidae]
(I mean, the recordings exist, I don't think that's under dispute)
09:44:55 [chirlu`]
I’m not sure yet were the medium went.
09:45:09 [nikki]
I've seen people use it to add videos from the release editor (since you can't set the video flag otherwise), also when adding a track when editing a release so that they can add relationships without having to wait a week (those are both arguably just problems with our interface, but still a useful use of them right now)
09:46:05 [chirlu`]
The main problem seems to be that people don’t understand choosing “Add a new recording” in the RE will add a new recording for them.
09:46:51 [nikki]
heh. and then there are complaining that users are using "add a new recording" too much :)
09:46:56 [nikki]
+people
09:47:32 [chirlu`]
Well, both is wrong.
09:47:54 [chirlu`]
But adding a standalone recording from the search when “add a new recording” would be enough …
09:48:23 [chirlu`]
(“Most” was perhaps a bit exaggerated, BTW. ;-) )
09:49:15 [chirlu`]
https://beta.musicbrainz.org/edit/31636539 is one example from that 500-track medium.
09:49:31 [chirlu`]
“song length not found in database”
09:50:39 [nikki]
well, I'm not sure removing the option to add a standalone recording would help, they'd probably just select a random other recording instead if they're so confused they don't realise they can just use the "add a new recording" option
09:50:51 [nikki]
which is far worse than some orphaned recordings
09:54:44 [nikki]
and the medium presumably went away with https://beta.musicbrainz.org/edit/31827578
09:56:03 [chirlu`]
Ah, thanks.
09:56:14 [chirlu`]
How did you find it?
09:56:59 [nikki]
I checked the editor's edits and noticed the add edit passed, so figured it must've been removed and searched for remove release edits made since it was added
09:57:09 [nikki]
then just searched in the page
09:57:33 [chirlu`]
Ah, I had searched for remove release and remove medium edits.
09:57:58 [chirlu`]
Which meant I ran into my own bug and never continued. :-p
09:58:12 [nikki]
which bug is that?
09:58:17 [ruaok]
ruaok has joined #musicbrainz-devel
09:58:28 [ianmcorvidae]
some remove medium edits don't display
09:58:32 [nikki]
ahh
09:58:34 [ianmcorvidae]
because of a change chirlu made that's in beta :)
09:58:36 [chirlu`]
MBS-8293 probably?
09:58:37 [mb-chat-logger]
http://tickets.musicbrainz.org/browse/MBS-8293
09:58:41 [ruaok]
moin
09:58:44 [nikki]
moin
09:58:49 [chirlu`]
Moinmoin.
10:00:19 [chirlu`]
Does it make sense that “Remove release” edits aren’t in the edit history of their track artists?
10:00:41 [chirlu`]
(Hence not in notification emails either.)
10:01:35 [ianmcorvidae]
I'm not sure. we don't really have a clear way of deciding what belongs there
10:01:47 [ianmcorvidae]
that does seem like it wouldn't be the expected behavior, maybe
10:01:50 [nikki]
probably not. destructive edits should be more prominent
10:02:05 [ianmcorvidae]
yeah, that's better reasoning than mine
10:02:46 [chirlu`]
The addition is in all of them …
10:04:05 [chirlu`]
The sysadmin ninjas haven’t by any chance arrived yet?
10:04:12 [nikki]
heh
10:04:17 [ruaok]
none here. :(
10:04:41 [ruaok]
maybe reboot totoro?
10:05:10 [ruaok]
it helped carl.
10:05:11 [ianmcorvidae]
if we do, we should probably ensure persistence of the vm.swappiness thing bitmap changed
10:05:25 [ruaok]
oh, what was that?
10:05:28 [ianmcorvidae]
and possibly set the vm.overcommit_memory setting that broke things earlier
10:05:31 [ianmcorvidae]
lemme find the pastebin
10:05:49 [ianmcorvidae]
http://pastebin.com/raw.php?i=CUk9haLK
10:05:50 [chirlu`]
He set it to 0, I think.
10:05:59 [ianmcorvidae]
yeah
10:06:18 [ianmcorvidae]
setting the vm.overcommit_memory one made the server very upset for a bit, throwing lots of out of memory errors, until we reset it to 0
10:06:25 [ianmcorvidae]
the swappiness one is still there though
10:06:29 [ianmcorvidae]
I think helped, if only mildly
10:06:43 [ianmcorvidae]
dunno if he did anything with readahead, which he was mentioning earlier
10:06:54 [ianmcorvidae]
... probably he's not still awake. but anyway
10:07:15 [ruaok]
did the vm.swappiness help at all?
10:07:30 [chirlu`]
Hard to tell.
10:07:57 [ianmcorvidae]
I think a little, but certainly less than we'd like
10:08:05 [ianmcorvidae]
it may be more meaningful if it's set at boot, though
10:08:16 [ruaok]
ok, lets set it at boot and then reboot.
10:08:19 [ianmcorvidae]
since obviously this instance set it after a bunch was *already* in swap
10:08:38 [ruaok]
there is literally nothing that can explain what is going on.
10:08:49 [ruaok]
I changed buckets to reduce traffic.
10:08:54 [ruaok]
turned off caa.
10:08:59 [ruaok]
turned off indexing.
10:09:05 [ruaok]
*nothing* really changed.
10:09:29 [ruaok]
I think the tuning chirlu` and I did helped keep the site somewhat usable at times, but nothing amazing.
10:09:41 [ianmcorvidae]
yeah, a lot of what we've done has been tuning, not really fixing
10:09:48 [ianmcorvidae]
which is useful, but not fixing the main issues :(
10:09:50 [ruaok]
I’m so solidly out of ideas.
10:10:02 [ianmcorvidae]
did we switch beta to the master branch?
10:10:05 [ruaok]
right, if we knew what the issue was we could fix it. maybe.
10:10:11 [ianmcorvidae]
it's very slim, of course, but
10:10:13 [ruaok]
I did not.
10:10:22 [ianmcorvidae]
lemme do that. at least rule one more thing out
10:10:29 [ruaok]
k.
10:10:41 [ianmcorvidae]
bitmap: ^ don't deploy stuff to beta if this is still on :)
10:11:21 [chirlu`]
It’s also problematic that it takes some hours after each change to tell if it helped.
10:11:52 [Junior]
Junior has joined #musicbrainz-devel
10:14:30 [ianmcorvidae]
beta now running master
10:14:43 [ruaok]
lets give it a few
10:17:08 [chirlu`]
Hm, it’s the Add medium that is associated with all track artists, not Add release.
10:17:24 [chirlu`]
So there is some symmetry in that Remove release isn’t.
10:27:59 [ruaok]
sigh, make a change, get rewarded with a load spike. :(
10:39:12 [chirlu`]
* chirlu` reads in the news about “ein Miss-Match”. *g*
10:46:22 [ruaok]
ok, time to reboot totoro?
10:46:32 [ruaok]
ianmcorvidae: are the changes totoro properly committed?
10:47:53 [chirlu`]
Perhaps reduce shared_buffers some more, as we’re restarting anyway?
10:47:54 [ruaok]
notice put up.
10:48:05 [ruaok]
ok, sounds good.
10:48:17 [chirlu`]
No banner on beta.
10:49:01 [ruaok]
fixed
10:49:35 [ruaok]
:work_mem => "80MB"
10:49:43 [ruaok]
:shared_buffers => "12GB",
10:49:45 [ruaok]
ya?
10:49:51 [chirlu`]
+1
10:49:53 [ruaok]
k
10:50:32 [ruaok]
ianmcorvidae: you about?
10:52:28 [ruaok]
ianmcorvidae: prod.
10:52:40 [ruaok]
* ruaok needs to know if kernel changes are comitted on totoro.
10:54:45 [ruaok]
root@totoro:/etc# tail -10 /etc/sysctl.conf
10:54:45 [ruaok]
# Do not accept IP source route packets (we are not a router)
10:54:47 [ruaok]
#net.ipv4.conf.all.accept_source_route = 0
10:54:48 [ruaok]
#net.ipv6.conf.all.accept_source_route = 0
10:54:50 [ruaok]
#
10:54:51 [ruaok]
# Log Martian Packets
10:54:53 [ruaok]
#net.ipv4.conf.all.log_martians = 1
10:54:54 [ruaok]
#
10:54:56 [ruaok]
# Added by ruaok
10:54:57 [ruaok]
vm.swappiness=0
10:54:57 [ruaok]
that??
10:55:15 [ruaok]
root@totoro:/etc# sysctl -a | grep swap
10:55:21 [ruaok]
vm.swappiness = 0
10:55:30 [ruaok]
which is what we have in current config.
10:56:08 [ruaok]
anyone see any issues with this?
10:58:24 [Mineo]
Mineo has joined #musicbrainz-devel
11:00:11 [ruaok]
ok, taking the site down.
11:02:29 [ruaok]
reboot started.
11:06:19 [ruaok]
this is taking uncomfortably long. :(
11:06:44 [Mineo]
fscking?
11:06:50 [chirlu`]
Well, it needs to swap everything back in, write everything to disc …
11:06:53 [ruaok]
I can’t see the console.
11:07:14 [ruaok]
both plausible.
11:07:15 [chirlu`]
I’ve had Firefox take several minutes to quit.
11:09:52 [ruaok]
ok, this is taking too long.
11:10:02 [ruaok]
better ask digital west to have a look at the console.
11:11:41 [bitstorm]
bitstorm has joined #musicbrainz-devel
11:13:29 [ruaok]
ok, they are going to have a look.
11:23:25 [chirlu`]
Surprisingly low activity on IRC and in the forums. I’d expected more questions. :)
11:23:41 [ruaok]
everyone has given up already. :)
11:29:59 [ruaok]
groan.
11:30:06 [ruaok]
he didn’t have his rack key.
11:30:12 [ruaok]
finally found the right key.
11:30:13 [chirlu`]
:)
11:30:21 [ruaok]
then hooked up the crash cart to pingu.
11:30:24 [ruaok]
not totoro.
11:30:32 [ruaok]
he’s going back now to look at the right server.
11:33:59 [randydwni]
randydwni has joined #musicbrainz-devel
11:36:23 [ruaok]
apparently it up, siitting at the login screen.
11:36:36 [chirlu`]
Hm.
11:36:40 [chirlu`]
But no ssh?
11:36:50 [ruaok]
no pings, nada.
11:36:56 [ruaok]
I’m having him power cycle it.
11:37:21 [ruaok]
it sometimes had issues getting its nic running.
11:38:53 [ruaok]
randydwni: you’re the man! :)
11:39:46 [ruaok]
fun, postgres didn
11:39:48 [ruaok]
t start.
11:41:11 [ruaok]
gah!
11:41:22 [chirlu`]
Config typo?
11:41:26 [ruaok]
no.
11:41:38 [ruaok]
randydwni just got around to rebooting it. again.
11:41:40 [ruaok]
not sure why.
11:41:51 [ruaok]
postgres had just started.
11:41:57 [ruaok]
I was about to get things moving again.
11:52:17 [ruaok]
this hardware flakiness is not making me happy.
11:52:23 [ruaok]
but it may point to the root of the problem.
12:19:21 [suyash]
suyash has joined #musicbrainz-devel
12:32:22 [ruaok]
ok, we have a newly configured network interface.
12:32:29 [ruaok]
now we need to get a new cable installed.
12:44:04 [chirlu`]
Beta still has the old banner. :-)
12:52:53 [bitstorm]
it's alive
12:53:30 [ruaok]
for now. :)
13:01:24 [Nyanko-sensei]
Nyanko-sensei has joined #musicbrainz-devel
13:08:13 [ianmcorvidae]
ruaok: sysctl stuff is in chef too, see the stuff about shmmax/shmall
13:08:18 [ianmcorvidae]
ruaok: so it can probably go in there too? not sure
13:08:45 [ianmcorvidae]
* ianmcorvidae was out and about at the time, anyway
13:08:50 [ruaok]
it should. lets see how it does.
13:09:03 [ruaok]
totoro was totally swap free,but is now using swap again.
13:09:32 [chirlu`]
With swappiness 0, that means it ran out of physical memory.
13:09:51 [ruaok]
joy.
13:10:09 [ruaok]
so, I’m stumped. utterly stumped.
13:10:15 [v_s]
v_s has joined #musicbrainz-devel
13:10:30 [ruaok]
so, short of logging queries and poking at them, I have no idea what else to try.
13:10:52 [ruaok]
interstingly enoguh, what does help….
13:11:07 [ruaok]
selectively asking PG to shut down back-ends.
13:11:35 [ruaok]
I just pick the ones that have the highest ram usage and ask PG to terminate those.
13:11:45 [ruaok]
never fails to bring the load down.
13:11:47 [diana_olhovik_]
diana_olhovik_ has joined #musicbrainz-devel
13:11:49 [ruaok]
what does that tell us?
13:12:03 [ruaok]
I suppose I can write a script that does that. :)
13:12:16 [ianmcorvidae]
sounds like a great way to get corrupted replication packets :P
13:13:00 [chirlu`]
Aren’t they made in transactions?
13:13:09 [ruaok]
speaking of which, I have no idea how those are doing.
13:13:17 [ruaok]
yep, here we go again.
13:13:21 [ianmcorvidae]
as far as I know they've been fine
13:13:24 [ruaok]
2GB into swap.
13:13:40 [ianmcorvidae]
and yeah, I think that'd be fine. killing stuff on astro does break replication packets, though
13:14:18 [ruaok]
the thing I don’t understand is why that works.
13:14:30 [ruaok]
that ought to give us a clue about possible problems, no?
13:15:23 [ianmcorvidae]
I dunno
13:15:23 [ruaok]
139 backends and most of them are idle.
13:15:36 [ianmcorvidae]
I guess it says backends are holding onto too much memory after transactions end
13:15:40 [ianmcorvidae]
or something?
13:15:47 [ruaok]
yes, something like that.
13:15:59 [ruaok]
as if they are leaky.
13:15:59 [ianmcorvidae]
I don't think I know enough about how postgresql works as far as that to contribute much
13:16:21 [ianmcorvidae]
it suggests it's not shared_buffers that's the issue, I think
13:16:37 [ianmcorvidae]
but perhaps that's misunderstanding somewhat
13:17:10 [ruaok]
we’re on 9.1.9
13:17:12 [ruaok]
there is a 9.1.15
13:17:23 [ianmcorvidae]
look at the changelog, maybe?
13:17:32 [ruaok]
let’s
13:18:07 [ruaok]
http://www.postgresql.org/docs/9.1/static/release-9-1-14.html
13:18:09 [ruaok]
interesting.
13:19:21 [ruaok]
9.1.10 has several memory leak issues fixed.
13:22:23 [ianmcorvidae]
yes, most of which don't seem to apply, but still
13:22:51 [chirlu`]
“Fix memory overcommit bug when work_mem is using more than 24GB of memory” maybe.
13:23:11 [ruaok]
I doubt that we use that much work_mem.
13:23:20 [chirlu`]
Though the question is still why this all started a few days ago.
13:24:08 [ruaok]
some table, some index grew too big and the query planner too a different turn.
13:24:12 [ruaok]
one that uses much more ram.
13:24:41 [ruaok]
ianmcorvidae: I don’t remember all the things needed to upgrade PG
13:24:51 [ianmcorvidae]
I haven't participated in past PG upgrades :)
13:25:02 [ruaok]
compile code with cube extensions.
13:25:07 [ianmcorvidae]
I'm not sure; we may be on the provided version at this point
13:25:18 [ruaok]
provided?
13:25:19 [ianmcorvidae]
as I recall that was part of the point of upgrading the OS
13:25:25 [ianmcorvidae]
the ubuntu version
13:26:09 [ruaok]
9.1.11 is installed.
13:26:11 [ruaok]
I see.
13:27:09 [ianmcorvidae]
and yes, .11 is what we're running
13:27:11 [ianmcorvidae]
not .9
13:27:29 [ianmcorvidae]
(select version())
13:27:59 [ruaok]
yeah, I didn’t realize we were running from packages now.
13:28:35 [ianmcorvidae]
package most recent version is .15 though
13:28:47 [ianmcorvidae]
so I guess it should be possible to just upgrade it with apt
13:29:05 [ianmcorvidae]
this is within a 9.1 version so we won't need to do the pg_upgrade annoyance
13:29:23 [ianmcorvidae]
hopefully next time we do that it's jumping to 9.4 or so :)
13:29:24 [ruaok]
yea, but there are a number of things to consider.
13:29:46 [ruaok]
we have two extensions.
13:29:51 [ruaok]
and pending.so
13:29:52 [bitstorm_]
bitstorm_ has joined #musicbrainz-devel
13:30:12 [ruaok]
and pgq needs to be installed as well, but I suspect that comes from packages.
13:30:16 [ianmcorvidae]
extensions at least are packaged as well; I don't know how/if they'll work with new versions
13:30:40 [ruaok]
worst case we can compile them by hand, no?
13:30:42 [ianmcorvidae]
re pgq, there's always getting https://bitbucket.org/metabrainz/musicbrainz-server/pull-request/1502/switch-to-rabbitmq-from-pgq done :)
13:30:59 [ianmcorvidae]
pg_amqp is also installed, re: that, of course
13:31:32 [ruaok]
this will require both of us doing this.
13:31:49 [ruaok]
I’m not comfortable doing this by myself.
13:32:13 [ruaok]
do you have some time tomrrow after you’ve slept?
13:32:51 [ianmcorvidae]
I should, though I'm not sure what my sleep is doing right now
13:34:08 [ruaok]
well, I’m awake and have time.
13:34:13 [ruaok]
if you fancy tackling it now...
13:34:51 [diana_olhovik_]
diana_olhovik_ has joined #musicbrainz-devel
13:34:54 [ianmcorvidae]
I'd fizzle quickly right now if any shit hit the fan, so probably for safety no
13:35:15 [ruaok]
smrt
13:35:40 [ruaok]
can you predict the next window of opportunity?
13:37:09 [ianmcorvidae]
relative to an assumption of you sleeping at normal times for you, I guess it'd be after you've slept?
13:37:33 [ruaok]
or, our normal time then. :)
13:37:49 [ianmcorvidae]
heh yeah, possibly
13:38:16 [ianmcorvidae]
I'm failing to successfully add numbers to determine what time it is there, but I guess midafternoon
13:38:28 [ianmcorvidae]
so if we don't do that it'd be you staying up pretty late, I guess
13:38:41 [ruaok]
define pretty late.
13:39:39 [ianmcorvidae]
almost certainly later than you'd like, I think? six to eight hours from now
13:39:52 [ianmcorvidae]
I mean, this is assuming I go have a night's sleep starting nowish
13:40:02 [ruaok]
8 hours is fine.
13:40:11 [ruaok]
not the best thing to do on a saturday night, but doable.
13:40:36 [ianmcorvidae]
what time is that for you then? since my sleepy brain is failing to calculate it
13:41:23 [ianmcorvidae]
(I guess I can go to nikki's page, heh. I guess that'd be something like 23:00?
13:41:32 [ruaok]
yes.
13:41:59 [ianmcorvidae]
I guess the question is if that then falls into *you* being dead to the world quickly if there's a shit-hits-fan situation
13:42:16 [ruaok]
I’ll be fine until 1am.
13:42:23 [ruaok]
probably 2am.
13:42:58 [ianmcorvidae]
k. then I guess I'll set an alarm for 3pm or thereabouts, which is 8:15 from now?
13:43:16 [ruaok]
yep.
13:43:34 [ruaok]
ok, I’m going to try and not pay attention to MB for the next 8 hours.
13:43:39 [ruaok]
I’m soo burnt out.
13:43:53 [ruaok]
I’ll put up a blog post telling people what is up.
13:44:04 [chirlu`]
* chirlu` proposes to check out beta on beta again and perhaps restarting CAA.
13:44:08 [ianmcorvidae]
yup. I'm running an analyze on the DB which will almost certainly not help but I decided may as well happen
13:44:20 [ianmcorvidae]
chirlu`: I'll put beta on beta, sure
13:44:25 [ruaok]
I’ll turn on caa.
13:44:34 [ruaok]
might as well DOS ourselves. :)
13:44:53 [chirlu`]
From the graphs, it doesn’t seem to have much of a difference …
13:44:57 [ianmcorvidae]
you say that like it's different :P
13:44:59 [ianmcorvidae]
heh
13:45:35 [ianmcorvidae]
beta's back, anyway
13:46:14 [ruaok]
caa is back.
13:50:03 [diana_olhovik_]
diana_olhovik_ has joined #musicbrainz-devel
13:57:23 [ujjwal]
ujjwal has joined #musicbrainz-devel
14:06:24 [diana_olhovik_]
diana_olhovik_ has joined #musicbrainz-devel
14:37:23 [AnjaliUjjainia]
AnjaliUjjainia has joined #musicbrainz-devel
14:40:23 [AnjaliU]
AnjaliU has joined #musicbrainz-devel
14:52:29 [AnjaliUjjainia]
AnjaliUjjainia has joined #musicbrainz-devel
14:54:37 [chirlu`]
chirlu` has left #musicbrainz-devel
15:01:03 [mb-chat-logger]
New post: blog: Hosting issues & downtime tonight <http://blog.musicbrainz.org/2015/03/14/hosting-issues-downtime-tonight/>
15:17:05 [CatQuest]
CatQuest has joined #musicbrainz-devel
15:34:58 [xps2_]
xps2_ has joined #musicbrainz-devel
15:36:18 [arx8]
arx8 has joined #musicbrainz-devel
15:43:41 [JesseW]
JesseW has joined #musicbrainz-devel
15:59:11 [CatQuest]
CatQuest has joined #musicbrainz-devel
16:07:03 [ujjwal]
ujjwal has joined #musicbrainz-devel
16:44:09 [lokeshw24]
lokeshw24 has joined #musicbrainz-devel
16:58:01 [johtso]
johtso has joined #musicbrainz-devel
17:26:32 [ujjwal]
ujjwal has joined #musicbrainz-devel
17:31:06 [lokeshw24]
Hi guys! I had sent a small proposal regarding gsoc project - Onsite notifications
17:31:14 [lokeshw24]
on the devel mailing list
17:31:26 [lokeshw24]
I'm waiting for the reply.
17:32:07 [derwin]
hi
17:37:12 [adhawkins]
adhawkins has joined #musicbrainz-devel
17:43:02 [JesseW]
JesseW has joined #musicbrainz-devel
17:48:29 [reosarevok]
reosarevok has joined #musicbrainz-devel
17:48:30 [reosarevok]
reosarevok has joined #musicbrainz-devel
18:04:07 [anjali2906]
anjali2906 has joined #musicbrainz-devel
18:42:11 [MBJenkins]
Project musicbrainz-server_beta build #470: SUCCESS in 16 min: http://ci.musicbrainz.org/job/musicbrainz-server_beta/470/
18:42:11 [MBJenkins]
Michael Wiencek: Check if attributes have changed since _.defer before validating (MBS-8295)
18:47:05 [anjali2906]
anjali2906 has joined #musicbrainz-devel
19:02:23 [JesseW]
JesseW has joined #musicbrainz-devel
19:09:18 [anjali2906]
anjali2906 has joined #musicbrainz-devel
19:16:46 [lokeshw24]
lokeshw24 has joined #musicbrainz-devel
19:26:49 [MBJenkins]
Project musicbrainz-server_beta build #471: SUCCESS in 15 min: http://ci.musicbrainz.org/job/musicbrainz-server_beta/471/
19:26:50 [MBJenkins]
Michael Wiencek: Fix silly binding context issue (MBS-8296)
19:32:19 [anjali2906]
anjali2906 has joined #musicbrainz-devel
19:39:20 [chirlu`]
chirlu` has joined #musicbrainz-devel
19:45:24 [arx8]
arx8 has joined #musicbrainz-devel
19:46:51 [arx8]
arx8 has joined #musicbrainz-devel
19:58:58 [lokeshw24]
lokeshw24 has joined #musicbrainz-devel
20:51:08 [diana_olhovik_]
diana_olhovik_ has joined #musicbrainz-devel
21:11:19 [CatCat]
CatCat has joined #musicbrainz-devel
21:42:06 [anjali2906]
anjali2906 has joined #musicbrainz-devel
21:44:00 [ruaok]
wow, 15 minute load over 100. sigh.
21:44:24 [Leftmost]
That's... bad.
21:44:43 [ruaok]
yep.
21:45:02 [ruaok]
we’re going to try a postgres upgrade in our next futile attempt to fix it
21:45:15 [chirlu`]
Swap is relatively low.
21:46:21 [ruaok]
I haven’t been logged in in the last 7 hours.
21:46:27 [ruaok]
so, I have no clue.
21:46:54 [ruaok]
interesting.
21:47:24 [chirlu`]
I’m looking at stats.mb.org.
21:47:34 [ruaok]
* ruaok nods
21:47:36 [ruaok]
Im loading that now.
21:47:38 [chirlu`]
Apparently the load 100 was a short spike.
21:47:41 [ruaok]
I want to see the progression
21:48:35 [ruaok]
interesting. at the worst time swap was 10-11GB
21:48:38 [chirlu`]
The swapspace usage and the I/O (particularly write) have improved, may be caused by the shared_buffers/work_mem changes.
21:49:00 [ruaok]
if it is stable at 3GB-4GB now, then I think we may actually be on the right track.
21:49:16 [ruaok]
I’m starting to think that.
21:49:25 [ruaok]
should we drop work_mem to 64MB?
21:49:29 [ruaok]
thats an easy change.
21:52:04 [ruaok]
we even have 6GB of ram free atm.
21:52:13 [ruaok]
* ruaok makes the work_mem drop to 64MB happen
21:54:29 [ruaok]
even if we keep it from swapping, the load is still too high for everything to be normal.
22:03:12 [ianmcorvidae]
* ianmcorvidae appears
22:03:19 [ruaok]
hey.
22:03:23 [ianmcorvidae]
* ianmcorvidae also notes that I changed two settings in pgbouncer before I went to bed
22:03:31 [ianmcorvidae]
since I see there's talk of things maybe being betteR
22:03:32 [ruaok]
see the link above, I’m making a plan.
22:03:34 [ianmcorvidae]
r?
22:03:40 [ruaok]
not really.
22:03:43 [ianmcorvidae]
ah, okay
22:04:02 [ruaok]
we’re using less swap, which is a good sign, but the site has been mostly unusable.
22:05:11 [ruaok]
I keep getting disconnected form etherpad losing my changed.
22:05:13 [ruaok]
s
22:05:24 [ruaok]
ianmcorvidae: is google docs an option for you yet?
22:07:01 [Leo_Verto]
I blame that particular etherpad host
22:07:02 [ianmcorvidae]
if you want I can make it work. if it's editable by anyone it's fine anyway, it'll just be a different account of mine than you're used to
22:07:26 [ruaok]
yeah, its a throw away.
22:07:35 [ruaok]
hmm. ok, let me try that.
22:07:44 [chirlu`]
Why explicitly uninstall Postgres first?
22:08:12 [ZaphodBeeblebrox]
ZaphodBeeblebrox has joined #musicbrainz-devel
22:08:58 [ruaok]
chirlu`: what is the right way to do it? upgrade?
22:09:27 [chirlu`]
Normally I’d just let apt handle it.
22:09:47 [ianmcorvidae]
I don't see why it'd be more than updating apt's package lists and then just apt-get install (which will reinstall/upgrade)
22:09:50 [ianmcorvidae]
yeah
22:10:05 [Leo_Verto]
apt is quite good at doing that
22:10:23 [Powlow]
Powlow has joined #musicbrainz-devel
22:10:46 [JesseW]
I've generally been happy with Wikimedia's etherpad...
22:10:48 [ianmcorvidae]
re: pgq I don't actually know what the process is there. with the other extensions, I'd honestly guess that there's nothing needed
22:10:49 [ruaok]
ok, step removed.
22:11:01 [ianmcorvidae]
this isn't a major upgrade, so
22:11:30 [ruaok]
does that mean that pendingchange.so doesn’t need to be changed either?
22:11:46 [ianmcorvidae]
recordchange I assume you mean, but yes, I think that's true
22:11:51 [ruaok]
yeah
22:12:17 [Leo_Verto]
JesseW, wikipedia has it's own instance?
22:14:00 [JesseW]
Leo_Verto: http://etherpad.wikimedia.org/
22:14:37 [ruaok]
ianmcorvidae: ok, my thoughts are in the doc.
22:15:50 [ianmcorvidae]
okay. I *think* that the steps for installing extensions with the question marks will be no-ops but it's fine if we have them there
22:16:14 [ruaok]
let’s hope so.
22:16:36 [ianmcorvidae]
with 'take site down', remember to do the CAA as well
22:16:40 [ianmcorvidae]
and beta
22:16:46 [ruaok]
yep.
22:16:48 [ruaok]
I’ll not that.
22:16:50 [ianmcorvidae]
but that's probably in your brain, just making sure it is if it wasn't :)
22:17:05 [ianmcorvidae]
we need to add a step to stop the caa indexer stuff
22:17:13 [ianmcorvidae]
in fact we may have already fucked that up
22:17:15 [ianmcorvidae]
* ianmcorvidae goes to check
22:19:30 [ianmcorvidae]
heh yup, now it's catching up
22:19:42 [ijabz2]
ijabz2 has joined #musicbrainz-devel
22:19:50 [reosarevok]
Oh
22:19:54 [ianmcorvidae]
anyway, we should stop pgq-proxy and caa-indexer on pinoprime and then once it's done we should restart them in reverse order
22:20:04 [reosarevok]
Is that why some art that was uploaded wasn't getting thumbnails and such?
22:20:17 [reosarevok]
(not in MB, I mean, but not in the IA either)
22:20:19 [ianmcorvidae]
yes
22:20:22 [reosarevok]
Ok :)
22:20:37 [ianmcorvidae]
caa-indexer produces index.json which the IA uses to produce thumbnails
22:20:40 [ianmcorvidae]
so yeah :)
22:21:13 [ruaok]
indexer and pgq-proxy has been noted.
22:21:16 [ianmcorvidae]
k
22:21:24 [ianmcorvidae]
not sure if pgq itself will need restarting
22:21:37 [ianmcorvidae]
if so it can be done separately though, I'm fairly sure
22:22:35 [ruaok]
banners set on both mb and beta.
22:23:15 [ianmcorvidae]
I can stop cron now if we're not going to wait 40 minutes, so I'll go do that
22:23:23 [ruaok]
go
22:24:32 [ianmcorvidae]
ready to trigger producing a replication packet once site's down
22:25:25 [ruaok]
I’m disconected from the damn pad.
22:25:30 [ianmcorvidae]
heh
22:25:53 [ruaok]
ok, google doc it is. what email should I invite?
22:26:16 [ruaok]
I’ll just post a link for everyone
22:28:18 [ruaok]
sanity check that ian?
22:28:26 [ruaok]
once you’re happy, I can take the site down.
22:29:17 [ianmcorvidae]
yup.
22:29:24 [ianmcorvidae]
stopped pgq-proxy and caa-indexer too
22:31:13 [ruaok]
ready?
22:31:18 [ianmcorvidae]
one sec
22:31:21 [ianmcorvidae]
checking apt on totoro
22:32:08 [ianmcorvidae]
to make sure it knows about the new package :)
22:32:54 [ruaok]
and there is the astro confirmation email. :)
22:32:58 [ruaok]
* ruaok is standing by
22:33:13 [ianmcorvidae]
er. apt sources show lucid, that seems questionable
22:34:12 [ianmcorvidae]
ah, I guess those are commented out at least
22:37:13 [ianmcorvidae]
I guess we're using some pgdg thing from apt.postgresql.org, but I'm having trouble determining why that doesn't have .15
22:37:37 [ruaok]
lovely.
22:38:26 [ianmcorvidae]
I mean, it *is* in their repo
22:38:31 [ianmcorvidae]
but it seems to not be properly finding that
22:44:33 [ianmcorvidae]
bah. okay, it does know it exists
22:44:43 [ianmcorvidae]
* ianmcorvidae hates apt sometimes :P
22:44:48 [ianmcorvidae]
ruaok: ready now, I think.
22:44:51 [ruaok]
ok.
22:45:12 [ianmcorvidae]
everything but the CAA looks to be down
22:45:51 [ianmcorvidae]
but I'll wait for confirmation, anyway
22:46:00 [ruaok]
found it.
22:46:08 [ianmcorvidae]
k. replication packet
22:46:10 [ruaok]
confirmed.
22:46:11 [ianmcorvidae]
* ianmcorvidae waits for that
22:46:17 [ianmcorvidae]
k
22:48:14 [ianmcorvidae]
collate is fine
22:48:22 [ruaok]
nice.
22:48:25 [ianmcorvidae]
unaccent is fine
22:49:11 [ianmcorvidae]
not sure how to check recordchange and pgq. hm
22:49:26 [ruaok]
for record change...
22:49:33 [ianmcorvidae]
* ianmcorvidae changes a language frequency then changes it back
22:49:46 [ruaok]
check that “pending” and “pendingdata” or whatever they are called are empty.
22:49:51 [ruaok]
then a change and undo.
22:50:04 [ruaok]
check for errors and or records in rep tables.
22:50:16 [ianmcorvidae]
heh
22:50:23 [ianmcorvidae]
we didn't stop the script updating amazon coverart
22:50:26 [ianmcorvidae]
which updated some stuff.
22:50:32 [ianmcorvidae]
so anyway there's stuff in there
22:50:35 [ianmcorvidae]
* ianmcorvidae checks that adding more works
22:51:27 [ianmcorvidae]
yup
22:51:38 [ruaok]
PGQ?
22:52:08 [ruaok]
and I guess postgres is already started by this poin
22:52:11 [CatQuest]
CatQuest has joined #musicbrainz-devel
22:53:27 [ianmcorvidae]
yeah
22:53:46 [ruaok]
I trust basic queries were fine, right?
22:53:50 [ianmcorvidae]
yes
22:53:59 [ianmcorvidae]
restarted caa indexer and pgq-proxy asw ell
22:54:03 [ianmcorvidae]
I think we're ready to come back
22:54:09 [ianmcorvidae]
* ianmcorvidae gets cron going again
22:54:13 [ruaok]
coming up
22:55:21 [CatQuest]
sorry to come in like a dog and all over the place, but you guys know anything more about whats wrong?
22:55:23 [ianmcorvidae]
* ianmcorvidae will get the beta banner
22:55:38 [ruaok]
not really, no. that is a the problem.
22:55:45 [CatQuest]
* CatQuest sees you're still working hard (and on a saturday too) to fix
22:55:47 [ianmcorvidae]
well, both banners actually
22:55:56 [ruaok]
main should be gone now.
22:56:02 [ruaok]
it keeps 502ing on me. :(
22:56:08 [ianmcorvidae]
banners are gone
22:56:10 [ianmcorvidae]
I got both of them, heh
22:56:12 [CatQuest]
that really sucks. it's the worst when you can't figure not how to fix it but WTF the problem is inthe first place
22:56:15 [ruaok]
k
22:56:23 [ianmcorvidae]
cron, caa-indexer, pgq-proxy back
22:56:27 [ianmcorvidae]
I think tweeting is what remains
22:56:31 [ianmcorvidae]
here's hoping it helps, heh
22:56:34 [ruaok]
CatQuest: http://blog.musicbrainz.org/2015/03/14/hosting-issues-downtime-tonight/
22:56:45 [CatQuest]
:thumbsup:
22:56:55 [ruaok]
twatted
22:57:04 [ruaok]
woo load of 100.
22:57:05 [ruaok]
:)
22:57:20 [ianmcorvidae]
haha
22:57:38 [ianmcorvidae]
* ianmcorvidae gets atop up
22:57:44 [ianmcorvidae]
should have done that *before* we brought it all back, but oops :)
22:58:20 [ruaok]
eek, not very responsive at all, is it?
22:59:32 [ianmcorvidae]
yes, but I think it's just the startup one right now
22:59:40 [ruaok]
* ruaok nods
23:00:00 [ruaok]
and the site is starting to respond nicely now.
23:00:03 [CatQuest]
hmm beta actually loads and the search worked snappy atleast
23:00:28 [ruaok]
yeah, everything is fine for a while after a restart
23:00:32 [CatQuest]
hmmm
23:01:10 [CatQuest]
for how long?
23:01:24 [ruaok]
5 to 20 minutes usually.
23:01:28 [CatQuest]
:(
23:01:52 [CatQuest]
having hourly restarts in not an option :B that's worse than windows ME
23:03:21 [ianmcorvidae]
* ianmcorvidae watches memory usage as it starts climbing
23:03:52 [ruaok]
a perfectly tuned postgres should eat all the ram. but no more.
23:04:13 [CatQuest]
it couldn't be something stupid like hardware malfunction?
23:04:22 [ruaok]
and often times it behaves nicely, sitting right at the edge.
23:04:35 [ruaok]
and then BAM something drives it over the line and its never the same again
23:04:49 [ruaok]
the cyclic nature of the problem bugs me too.
23:05:01 [CatQuest]
* CatQuest should shut up and go eat since he knows nothing anyway
23:05:03 [CatQuest]
but I sincerely hope you guys figure it out.!
23:05:13 [ruaok]
thx
23:09:00 [ruaok]
now totoro is bouncing right where it should.
23:09:56 [ruaok]
we also have quite a spread of process sizes. From 10% to 3%. That’s qiute a bit of ram difference.
23:09:58 [ijabz2]
ijabz2 has joined #musicbrainz-devel
23:11:04 [Leo_Verto]
time to pull up the stats!
23:11:45 [ianmcorvidae]
non-cache memory usage is sitting pretty high-up
23:11:51 [ianmcorvidae]
which is the thing that eventually makes it swap
23:12:26 [ianmcorvidae]
35600MB-ish of 48295MB
23:13:35 [ruaok]
where can I see non-cache memory usage stats?
23:13:47 [ianmcorvidae]
htop has the cache in yellow and non-cache in green
23:14:04 [ianmcorvidae]
or run e.g. free -m
23:14:09 [ianmcorvidae]
where it's the second line
23:14:21 [ianmcorvidae]
(the -/+ buffers/cache"
23:14:24 [ianmcorvidae]
line)
23:14:57 [ruaok]
thx
23:15:45 [ruaok]
yep, and off into swap we go.
23:16:09 [ruaok]
I may need to go and beg josh berkus again.
23:19:23 [ruaok]
yep, as soon as the procs grow to about 10% we go into swap. without fail.
23:26:05 [Leo_Verto]
what's the current swappiness?
23:26:08 [Leo_Verto]
cat /proc/sys/vm/swappiness
23:26:13 [ruaok]
6GB deep.
23:26:26 [ruaok]
and 0
23:27:54 [chirlu`]
It got changed from the default 60 yesterday.
23:28:08 [chirlu`]
Not much of an improvement.
23:29:25 [ruaok]
I can’t decide if postgres is somehow broken or if we’re somehow missing some traffic that somehow sneaks in and overloads it.
23:29:34 [ruaok]
but the overloading seems to be going wrong, no>
23:29:47 [ruaok]
I would expect more non-swap io if everything was ok.
23:30:03 [ruaok]
it should never swap, if it were properly tuned, no?
23:30:48 [Leo_Verto]
hmm, found this https://stackoverflow.com/questions/7731371/postgres-causing-swapping-on-centos
23:31:07 [derwin]
what version of kernel
23:31:16 [derwin]
it matters wrt swappiness behavior at 0
23:31:28 [ruaok]
> Linux totoro 3.2.0-57-generic #87-Ubuntu SMP Tue Nov 12 21:35:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
23:32:08 [derwin]
http://www.percona.com/blog/2014/04/28/oom-relation-vm-swappiness0-new-kernel/ is what I mean, seems to be 3.5.x
23:32:27 [ruaok]
do we need to lower SHMMAX since we’re not using as much shared_buffers??
23:33:26 [chirlu`]
I think it’s only a maximum limit, doesn’t influence behaviour otherwise.
23:33:38 [ruaok]
that was my impression, just checking.
23:34:19 [ruaok]
so, the big question on my mind: are we simply overloading out DB server or is there something fundamentall wrong that needs fixing?
23:34:43 [ruaok]
it is just too much traffic, then we’ve been trying to fix it the wrong way.
23:34:55 [ruaok]
and by too much traffic… I should restate that.
23:35:19 [ruaok]
some queries could’ve changed and suddenly got more expensive.
23:35:42 [ruaok]
I did a check for proper indexes. everything looked ok.
23:35:44 [Leo_Verto]
how many connections to the DB exist? 3?
23:35:57 [ruaok]
derwin: how do you check for possible missing indexes?
23:36:09 [ruaok]
Leo_Verto: different machines that can connect?
23:36:16 [ruaok]
5-6
23:36:46 [ruaok]
but our traffic hasn’t picked up.
23:36:51 [ruaok]
* ruaok is so confused and tired
23:37:08 [derwin]
I use percona toolkit analysis of the slow query log
23:37:14 [derwin]
but that's mysql...
23:37:42 [Leo_Verto]
what's shared_buffer set to?
23:37:48 [Leo_Verto]
*buffers
23:37:58 [ruaok]
12GB
23:38:07 [ruaok]
48GB in the box.
23:38:24 [ruaok]
was 20GB and things were worse then.
23:38:39 [derwin]
are there system level graphs
23:38:46 [derwin]
of things like i/o to varius partiotions
23:38:58 [derwin]
and load and running processes and stuff?
23:39:08 [ruaok]
yes.
23:39:09 [ruaok]
hang on.
23:39:16 [chirlu`]
http://stats.musicbrainz.org/
23:39:27 [ruaok]
http://stats.musicbrainz.org/webstats/nginx-rrd/drraw.cgi?Mode=view;Dashboard=1305570746.28532
23:39:47 [ruaok]
http://stats.musicbrainz.org/webstats/nginx-rrd/drraw.cgi?Browse=AllTemplates
23:40:16 [ruaok]
http://stats.musicbrainz.org/mrtg/drraw/drraw.cgi?Mode=view;Template=1196204920.6439;Base=%2Fvar%2Fwww%2Fmrtg%2F%2Ftotoro_disk-swapspace.rrd;View=3
23:41:02 [Gentlecat]
Gentlecat has joined #musicbrainz-devel
23:41:11 [ruaok]
I used this to find indexes we might need.
23:41:21 [ruaok]
I created indexes listed there, but nothing helped.
23:41:29 [derwin]
I am about to be socially engaged IRL
23:41:32 [derwin]
but I will check those ASAP
23:41:34 [ruaok]
thx
23:42:16 [ianmcorvidae]
* ianmcorvidae went for a drink run... things are looking fine right this second, did we do something to make that happen?
23:42:29 [Nyanko-sensei]
Nyanko-sensei has joined #musicbrainz-devel
23:42:30 [Leo_Verto]
a stacked graph of ram and swap would be so useful
23:42:37 [ruaok]
no, no real change.
23:42:43 [ruaok]
still in swap.
23:42:50 [ruaok]
ok atm, but load was at 20 a while ago
23:42:53 [ianmcorvidae]
uh
23:42:59 [ianmcorvidae]
htop doesn't agree
23:43:12 [ianmcorvidae]
we're using 100MB of swap, nothing more
23:43:20 [ianmcorvidae]
I guess it's just a lull
23:43:24 [ruaok]
ding.
23:43:24 [ianmcorvidae]
but it's a better lull than I've seen in ages
23:43:40 [ruaok]
see load 12 in 15 minute load avg
23:44:00 [ruaok]
its also past bed time in the EU, so out peak is starting to wane
23:44:17 [ianmcorvidae]
hm, yeah
23:44:25 [ianmcorvidae]
well, for now nothing to do but keep monitoring it
23:44:41 [ruaok]
I’m going to monitor my eye lids soon.
23:44:42 [chirlu`]
There were still enough 502s after upgrade: http://stats.musicbrainz.org/webstats/nginx-rrd/drraw.cgi?Mode=view;Graph=1324917157.2341
23:44:44 [ianmcorvidae]
I'm guessing that's also a "time for ruaok to get to bed"
23:45:05 [chirlu`]
It’s “normal” for them to come and go.
23:45:21 [ianmcorvidae]
yes, but they've not been going away to this degree before :)
23:45:27 [Leo_Verto]
oh nice, found the stacked graph http://stats.musicbrainz.org/mrtg/drraw/drraw.cgi?View=-1&Template=1196204920.6439&Base=%2Fvar%2Fwww%2Fmrtg%2F%2Ftotoro_disk-swapspace.rrd&Start=end+-+4+hours&End=now&Mode=view
23:45:33 [ianmcorvidae]
it's not fixed, but I do think we're improving things over time
23:45:43 [ruaok]
ianmcorvidae: agreed.
23:46:01 [ruaok]
our DB server is nicely tuned, but probably simply overloaded with…. something.
23:46:16 [ruaok]
here is the result of the stackoverflow query that I just linked:
23:46:16 [ruaok]
https://gist.github.com/mayhem/423b084043235fb78642
23:48:04 [ianmcorvidae]
ruaok: you didn't actually link the stackoverflow, but yeah, that looks believable
23:48:13 [ianmcorvidae]
nothing much new there though
23:48:31 [ianmcorvidae]
not sure what's up with instrument, that one's interesting
23:48:37 [ruaok]
http://stackoverflow.com/questions/3318727/postgresql-index-usage-analysis
23:48:56 [ruaok]
its tiny, so its cached. at least that is my guess.
23:49:05 [ianmcorvidae]
edit_url is possibly concerning
23:49:06 [ianmcorvidae]
hm, possibly
23:49:16 [ruaok]
it has a composite index.
23:49:21 [ruaok]
and I made an index just on edit.
23:49:25 [ruaok]
no difference.
23:49:41 [ruaok]
right then.
23:49:48 [ruaok]
http://www.someecards.com/usercards/viewcard/MjAxMi1mMjFiMWQ5ZmM3MGY2Yjc5
23:50:01 [ruaok]
nn all!
23:50:05 [ianmcorvidae]
yup. nn
23:50:09 [ianmcorvidae]
* ianmcorvidae will monitor as best I can
23:50:11 [ianmcorvidae]
but you're gone :P
23:51:06 [ianmcorvidae]
* ianmcorvidae looks at index bloat again just for shits 'n giggles
23:51:39 [chirlu`]
What has certainly improved is the disk I/O, particularly writes: http://stats.musicbrainz.org/mrtg/drraw/drraw.cgi?Mode=view&Template=1196205342.7170&Base=%2Fvar%2Fwww%2Fmrtg%2F%2Ftotoro_diskstats-sda-count.rrd
23:52:08 [chirlu`]
Though still far higher than before the RAM upgrade even.
23:52:10 [kahu]
kahu has joined #musicbrainz-devel
23:52:11 [ianmcorvidae]
yeah
23:52:23 [ianmcorvidae]
most of that has been swap read and write as far as I could tell
23:52:30 [ianmcorvidae]
but maybe there's more to it
23:53:02 [Leo_Verto]
uhh, the ram upgrade http://stats.musicbrainz.org/mrtg/drraw/drraw.cgi?Mode=view;Template=1196204920.6439;Base=%2Fvar%2Fwww%2Fmrtg%2F%2Ftotoro_disk-virtualmemory.rrd;View=3
23:53:31 [Leo_Verto]
that's virtual memory, swap and physical
23:53:48 [chirlu`]
Yes.
23:55:21 [Leo_Verto]
so if the ram upgrade triggered it, networking overload is kinda unlikely, isn't it?
23:55:53 [chirlu`]
It didn’t trigger it, or we would have had the problems months ago.
23:56:54 [chirlu`]
It actually improved query performance in many cases (as expected, because more data is cached).
23:57:24 [ianmcorvidae]
it improved it until the schema change, when it basically all went back to the same crap
23:57:27 [ianmcorvidae]
IIRC
23:58:22 [Leo_Verto]
any idea what causes this long-term rising and suddenly dropping of swap usage? http://stats.musicbrainz.org/mrtg/drraw/drraw.cgi?Mode=view;Template=1196204920.6439;Base=%2Fvar%2Fwww%2Fmrtg%2F%2Ftotoro_disk-swapspace.rrd;View=3
23:59:13 [chirlu`]
Server restarts, probably.
23:59:31 [Leo_Verto]
so just mem leaks?
23:59:51 [chirlu`]
For the rising part, as swappiness was higher before yesterday, long-unused pages were swapped out.