asciilifeform: prolly forever

asciilifeform: i pissed on 'db' concept as a student, and i piss today: custom data structure for each job! the year ~is~ 1972.

asciilifeform: hence the slow methodical spray of petrol

asciilifeform: all of it.

asciilifeform: it's malignantly retarded, and i'ma burn it down.

asciilifeform: or wait, 'spittoon is in one strand' ??

asciilifeform: http://btcbase.org/log/2016-12-30#1593712 << why should i ever get garbage when reading ~unrelated~ datum?!! ☝︎

asciilifeform: 'keep monkeys from injuring self and others'

asciilifeform: liquishit.

asciilifeform: all of'em

asciilifeform: *tonnes

asciilifeform: it is precisely chewing gum, 1000 tonned of it

asciilifeform: http://btcbase.org/log/2016-12-30#1593675 << mircea_popescu nails it: postgres is crippled 'for yer own good' ☝︎

asciilifeform: hence eatblock on airgapped box.

asciilifeform: ditch it, and ditch randos and their shitblocks, and 0--current sync takes 6 or so hrs.

asciilifeform: davout: mempool operation slows sync 100x

asciilifeform: which, in turn, ate mircea_popescu 's vintage block set

asciilifeform: all of my nodes, fwiw, descend from the eatblock experiment

asciilifeform: eatblock -- eats it

asciilifeform: whole block

asciilifeform: aha

asciilifeform: http://btcbase.org/log/2016-12-30#1593682 << prb aint got dumpblock ☝︎

asciilifeform bbl, meat

asciilifeform: trinque: that looks potentially useful, i'ma look at it in detail when i come back from meat .

asciilifeform: (doesn't prove that it is absent)

asciilifeform: yeah when i put on my shit diving suit and went down into the docs, i found none.

asciilifeform: if someone knows , from memory, the relevant knob: please write in.

asciilifeform: aah ok

asciilifeform: for all of the cruel things i have said about postgres : it crashed 0 times.

asciilifeform: mysql is a shitsandwich, and i will not touch it (it fucking CRASHES)

asciilifeform: mircea_popescu: it is currently a cached image, i implemented it. the cached snapshots however last for a limited time (iirc i have it set to half hr per url)

asciilifeform: phf, mircea_popescu , et al : one thing that would immediately make a very palpable difference in speed is if there were a permanent way to order postgres to perform all reads immediately, disregarding all locks.

asciilifeform: if there is some other way of doing it, i'm all ears

asciilifeform: mircea_popescu: point of 'nursery' was to do the 'do we have this fp? how about this? ...' a few thou. at a time, is all.

asciilifeform: (and yes, it is the obviously correct way to process thous. of keyz, no question)

asciilifeform: how to make the www piece respond at all while this runs ?

asciilifeform: mircea_popescu might be social-integrated-genius but often recommends algo that adds up to escherian skyscraper. and so results in headache thread.

asciilifeform: mircea_popescu: suggest algo, i'm all ears.

asciilifeform: http://usagl.com << from above, lulzy, old-school voice telecom co.

asciilifeform: (won't import in gpg, but ~will~ in js www-based shitpgptrons)

asciilifeform: looks like flipolade

asciilifeform: holy shit was that a ... user submission?!

asciilifeform: aaaaah lel

asciilifeform: mircea_popescu: the other obvious thing would be to dispense with 'real time submission' entirely, and when someone dumps in a key, it goes into next batch. but we discussed this earlier in this thread, it would mean that the thing cannot be used as sks-like tool.

asciilifeform: this has to be done programmatically ?

asciilifeform: phf: what means 'rebuild it on a crash'

asciilifeform: (and bernsteinization requires access to ~all~ moduli, as i think is obvious, and not simply 'most recent ones')

asciilifeform: mircea_popescu: but yes, for next version (presently only exists in my notebook) there is a nursery and it gets merged into main table at night. but this makes for considerably more complex system, where there are two very distinct types of submission, 'realtime' and 'scripted' , and they get treated quite differently.

asciilifeform: phf: there is

asciilifeform: and no, you can't query the nursery every time somebody loads a url, or you get SAME performance as now, omfg

asciilifeform: well 1 db, 2 sets of key/fp/factor tables. ☟︎

asciilifeform: otherwise we get same speed as now.

asciilifeform: only the 'adult' db

asciilifeform: but what this adds up to is to have ~two~ quite separate phuctors. we wouldn't query the nursery, for instance, when someone keys in a url with a hash

asciilifeform: it is the most obvious unmassaged piece , aha. the correct algo is , imho, to have separate 'nursery' (gcism term of art) table for the batch submits.

asciilifeform: i dun give half a shit about 'image'. laying out the fact of why the thing is as it is.

asciilifeform: that way there is exactly one procedural path for key submission, and no duplicate logic.

asciilifeform: they get thrown into same hole as if human submits.

asciilifeform: which is to say, rewrite of WHOLE thing. we had a thread.

asciilifeform: mircea_popescu: actually i do not. because it will require 100x more complex mechanism.

asciilifeform: (which was for some 'script kid' php kit)

asciilifeform: mats: i especially loved the 1 single av signature offered

asciilifeform: (in fact, dumping out the entire db, and properly bignumizing, takes about 3min total for the current db.)

asciilifeform: on entire set.

asciilifeform: these end up parsed into operable bignums every shot. but, surprisingly, this never takes > 3 minutes !

asciilifeform: phf: understand also, postgres can't store bignums as such, it stores strings

asciilifeform: when i profiled it, 99% of the time is spent in 'do we have this key hash? no? insert; do we have these fp's? no? insert...'

asciilifeform: (or second, depending how to count)

asciilifeform: the current iteration is, iirc, the third from-the-ground rewrite.

asciilifeform: believe or not, i actually put some work into this thing

asciilifeform: that that's probably not it.

asciilifeform: phf: actually the wwwtronic piece of phuctor is in python and does the precompiled queries thing

asciilifeform: i think so?

asciilifeform: phf: what part of 'this isn't the bottleneck' was unclear

asciilifeform: this does not always help.

asciilifeform: mircea_popescu is seeing it through the naive vertically integrating rockefeller eyes, 'power plant expensive? let's put it right in my mansion'

asciilifeform: if it somehow had to happen inside postgres, it would not bypass the lock.

asciilifeform: understand, the only reason why the thing works at all, is that this one small part of it, the bernsteinization, can be made ~entirely~ independent from the db locking idiocy

asciilifeform: and not the bernsteining.

asciilifeform: because the actual bottleneck is '1000s of queries AND inserts / second AND guaranteed realtime consistent'

asciilifeform: a sql or similar db system with built-in bignumatron could be useful and interesting. but no such thing exists. nor would it solve the actual bottleneck in phuctor if it were to be discovered tonight.

asciilifeform: i for one do not expect to live long enough to make a serious attempt at such a thing.

asciilifeform: mircea_popescu: if you think trb is a hard nut to crack, picture reading, grasping postgres.

asciilifeform: so that leaves 1 plausible explanation.

asciilifeform: and at this point it is imho unlikely that he has not heard of phuctor.

asciilifeform: 0 answer.

asciilifeform: http://btcbase.org/log/2016-12-30#1593516 << recall, i wrote to bernstein himself. ☝︎

asciilifeform: so it has not been a priority, because batching will tremendously complicate the moving parts.

asciilifeform: the querying of 'do-we-have-this-factor' is maybe 1% of the load.

asciilifeform: it is, by lightyears, the best known algo for batch gcd, also.

asciilifeform: by definition, that's what it does.

asciilifeform: there is no way around this.

asciilifeform: phf: bernstein's algo operates on ~all known moduli simultaneously~

asciilifeform: much less an optimized one

asciilifeform: mircea_popescu: sql doesn't have a bignumatron

asciilifeform: i need ~less~ access to db, not moar.

asciilifeform: so holy shit is this not screamingly obvious

asciilifeform: the individual bignums.

asciilifeform: mircea_popescu: algo ~demands~ O(1) random access to the bignums.

asciilifeform: mircea_popescu: it's a screaming nope