log☇︎
76000+ entries in 0.569s
mircea_popescu: oh i see.
davout: http://btcbase.org/log/2016-12-30#1593458 <<< vaguely rings a bell, does anyone have some pointers at hand so i can go read/re-read? ☝︎
phf: jurov: well, it's not clear where "disregard all locks" comes from in the original request. if the actual operations are as asciilifeform describes, i.e. sporadic inserts, and sporadic selects, then there will be no locks. my point is that there's no "disregard all locks" in postgresql, you solve it by knowing what lock you're hitting, and then designing your query to sidestep the lock
mircea_popescu: it really should be up to operator wtf, if i want to read dirty let me read dirty what sort of decision is this for designer to make. ☟︎
asciilifeform: trinque: that looks potentially useful, i'ma look at it in detail when i come back from meat .
asciilifeform: yeah when i put on my shit diving suit and went down into the docs, i found none.
mircea_popescu: i wasn't proposing mysql in any sense.
asciilifeform: for all of the cruel things i have said about postgres : it crashed 0 times.
asciilifeform: mysql is a shitsandwich, and i will not touch it (it fucking CRASHES)
asciilifeform: mircea_popescu: it is currently a cached image, i implemented it. the cached snapshots however last for a limited time (iirc i have it set to half hr per url)
mircea_popescu: mysql doesn't lock reads on write locks ; i expect any rmdbs should be capable via config.
mircea_popescu: i dunno if it supports dirty reads.
asciilifeform: if there is some other way of doing it, i'm all ears
mircea_popescu: m of art in whatever, am i not impressed!)
mircea_popescu: if this is the path you must walk to go from solipsist-alf to socially-integrated-alf i can see it, but hurry it up already it's irritating.
asciilifeform: mircea_popescu: suggest algo, i'm all ears.
asciilifeform: (and bernsteinization requires access to ~all~ moduli, as i think is obvious, and not simply 'most recent ones')
asciilifeform: i dun give half a shit about 'image'. laying out the fact of why the thing is as it is.
asciilifeform: mircea_popescu: actually i do not. because it will require 100x more complex mechanism.
mircea_popescu: i gather you already do b. is it index-sorted ?
asciilifeform: mats: i especially loved the 1 single av signature offered
phf: asciilifeform: right, i was going to get to that :}
asciilifeform: when i profiled it, 99% of the time is spent in 'do we have this key hash? no? insert; do we have these fp's? no? insert...'
phf: i'm not even arguing with you, i'm saying that the ~full extent~ of what "move it to psql" is going to do is ~eliminate cross-boundary issue~ that is all. so it'll shave some significant overhead, but it's not a silver bullet.
asciilifeform: believe or not, i actually put some work into this thing
phf: what i'm saying is that a significant fraction of "1000s of queries AND ..." is the cross-boundary. you compile queries on c side, you send them to psql, it then parses, prepares results, serializes, sends it to c side, c side has to now parse all over again
asciilifeform: i think so?
phf: asciilifeform: did you understand what i said?
asciilifeform: i for one do not expect to live long enough to make a serious attempt at such a thing.
asciilifeform: http://btcbase.org/log/2016-12-30#1593516 << recall, i wrote to bernstein himself. ☝︎
phf: asciilifeform: i'm just trying to establish the dataflow here, for my own curiosity
trinque: but I am not arguing for something here; you'd know what you want
asciilifeform: i need ~less~ access to db, not moar.
mircea_popescu: though i am unaware anyone ever implemented this ; because, of coruse, i am unaware anyone used the guy's algo for any other purpose than gawking.
asciilifeform: where i have O(1) access to them.
asciilifeform: trinque: i need random-access in O(1) to them for bernsteining
trinque: I am sadly, quite good at SQL if you want the thing translated
asciilifeform: also did i mention that the entire db get shat out every time we bernstein ?
asciilifeform: (and, painfully, i had to find the offending garbage by hand!)
asciilifeform: mircea_popescu: i'd first like to know what this'll do to integrity-on-mains-failure
asciilifeform: but i do not expect miracle.
asciilifeform: mircea_popescu: probably. i'ma run with new knob settings as soon as it is safe to reset the db.
mircea_popescu: incidentally asciilifeform since we're now doing open source db optimization shared_buffers is probably a larger concern. what is it ? defaults to 125mb but i'd readily see it 1-4gb in your case.
a111: Logged on 2016-12-30 14:32 Framedragger: (do note, 'work_mem' is per user / per request. so may be easier to DoS. thought i should mention this for completeness)
a111: Logged on 2016-12-30 14:30 asciilifeform: i'ma try it soon.
a111: Logged on 2016-12-30 14:20 asciilifeform: i even spoke with career dbists, answer was 'your application is monstrous abuse and you need a cluster'
mircea_popescu: http://btcbase.org/log/2016-12-30#1593170 << he's not even kidding, i'm going to start banning. ☝︎
asciilifeform: jurov: lulzily, searching for 'c++17' i get... airplanes
asciilifeform has a relative who, until recently retiring, programmed in PL/I ! i shit thee not
asciilifeform: but yes, cpp standardization is beginning to resemble that of PL/I
jurov: i was just paraphrasing, don't remember the exact word
diana_coman: jurov, make cpp haskell again I gather?
mod6: when I get a free moment, i'll throw the latest eulora on there. can be my mining box. :]
mod6: i had obsd on it like for nearly all of '16... but wasn't doing anything with it. so i threw linux on there.
mod6: oh, hey, actually. so I've got a box.
mod6: nice! i haven't done any sledding yet. gotta do that one of these times.
diana_coman: fwiw I can confirm that current code compiles perfectly fine on gcc 4.4 in any case
asciilifeform: jurov: i am guilty of referring to anything that wasn't in my childhood borland3.1 as cpp11
asciilifeform: ah hm. i have nfi what is 'planeshift'. 3dengine?
asciilifeform: mircea_popescu: presently i have nfi what part is published, or where
asciilifeform: i'ma have to gather the courage and read this thing with own eyes, at some point.
mircea_popescu: sure. i'm not saying it must be standardized. just, there.
asciilifeform: well here's one typical scenario, i find a pgp-signed historic patch (e.g., linus) and want to see what vintage key etc
mircea_popescu: asciilifeform hey, i'm not surfe i want it to work on heathens what.
mircea_popescu: incidentally, i would say deedbot also counts as tmsr keyserver now.
asciilifeform: most of the time i paste in a key from somewhere, there it is, from april.
asciilifeform: the one obvious optimization i was considering was to avoid all dupe checks on key submit and simply deduplicate prior to each bernsteining. but this has serious cost in ui consistency, no more could submitters expect to see a result that is guaranteed to make sense after they submit.
mircea_popescu: asciilifeform ah, i didn't realise you were happy with linux readcache.
mircea_popescu: i take it you were considering this for next rewrite
Framedragger: i suspect then that the inserts/sec slowness is due to postgres currently making really damn sure that *all* layers of cache are forced. this "full forcing of cache for every row" is what makes things slower; but it's also the only really-super-reliable approach for the case at hand (remote box).
Framedragger: (i'm not saying 100 is not barf-magical.)
asciilifeform: i'm not convinced that they are separable.
mircea_popescu: well not exactly like that, but i guess that may work for a heuristic early on.
mircea_popescu: because it is not computer-possible to have what you describe without what i describe.
mircea_popescu: Framedragger see here's what graybeard means : i see that statement, and I KNOW there's a footnote somewhere you don't know about / bother to mention which says "except when abendstar in conjunction with fuckyoustar when it's 105th to 1095th column".
asciilifeform: gnupatch, as i warned long ago, MUST die.
mircea_popescu: his problems i mean
asciilifeform: mircea_popescu: tbh i had a 'what the hell is all this' reaction to reading ben_vulpes and phf vtron problemz
mircea_popescu: asciilifeform hey, i only said they exist, i didn't say their brains work.
a111: Logged on 2016-12-30 05:22 phf: i think it treats one of the names as canonical
Framedragger: here's what i'm thinking: disable synchronous_commit , but set 'checkpoints' so that results are flushed to db every $n inserts/updates. i can see however how you may barf from such an idea, "it's either reliable, or isn't".
asciilifeform: but could again if i gave it a multi-GB writecache
asciilifeform: mircea_popescu -- switched hosts, i -- slowly, painfully, rewrote the thing...
Framedragger: "lose weeks of work" is insane :( i'm sorry to hear that. *this* would not expose you to that scenario. but one would have to pin down still-possible data loss scenarios, if any.
asciilifeform: the machine isn't at my house and i have no control over the mains supply.
Framedragger: ah right, yes i see
asciilifeform: that means i won't be trying it for couplea weeks. i don't restart db until a current parcel is through submitting.
Framedragger: (do note, 'work_mem' is per user / per request. so may be easier to DoS. thought i should mention this for completeness) ☟︎
asciilifeform: i'ma try it soon. ☟︎
Framedragger: ah hm. tbh i'd still change work_mem because it's ridoinculously low by default, but i hear ya.
Framedragger: ahh right, i assume those include in-memory sorts
Framedragger: asciilifeform: i take it you are certain that main bottleneck and 'hogger' is the numerous inserts?
a111: Logged on 2016-07-18 18:08 asciilifeform: i know of no file system that would not choke.
asciilifeform: now if only i had a pill against http://btcbase.org/log/2016-07-18#1505027 ☝︎
Framedragger: to be 100% certain, i'd have to check. i see your concerns.
Framedragger: right. i just thought about checkpoint_completion_target (set to say 0.9) which may help with inserts, but ultimately you're right, physical reality
asciilifeform: i absolutely can't have a db req returnig before disk is written.
Framedragger: busy for a bit, i don't want to cite you sth without thinking about it
asciilifeform: soo which knob should i max, Framedragger ?
asciilifeform: i even spoke with career dbists, answer was 'your application is monstrous abuse and you need a cluster' ☟︎