asciilifeform: Framedragger: it is doubtless possible to optimize .
asciilifeform: whenever ANYTHING is added , it first has to be searched for in each of these classes of item, in the case that it may already exist there.
asciilifeform: and moduli, stored separately, linked bidirectionally to/from keys, ditto factors for same.
asciilifeform: it is a db of ~5 mil KEYS, each of which is from 1 to 20kB, and links, etc.
asciilifeform: it is not a wordpress blog with ~static 5 mil words.
asciilifeform: try to apprehend the scale of the thing.
asciilifeform: because, for instance, the key-eater is a separate process. and has nfi that someone else wants the db.
asciilifeform: if you want anything from the db, you wait. for , possibly, HALF A MINUTE
asciilifeform: we are at the farthest possibly limit of what can be done on one box, at anything like reasonable budget (whether paying the cost of a small european flat for a server for public service is 'reasonable' is separate question)☟︎
asciilifeform: and there is ~0 way around it , other than by doing the static thing.
asciilifeform: Framedragger: db being hammered 24/7 with 'do we have this hash' 'do we have this fp' 'add this and this' 1000/sec is the bottle.☟︎☟︎☟︎
asciilifeform: and, to continue to rain on the parade, if every www site has to be run like mpex, it will cost. and the range of things that can be provided 'for the public', 'for phreeee', will correspondingly shrink.
asciilifeform: which seems to be beyond the current technological state of the art.
asciilifeform: ben_vulpes: the main thing wanted by 'people doing most of the analysis' is a BOX THAT STAYS UP
asciilifeform: db is under ~constant ~100% load as-is.
asciilifeform: i cannot offer anything REMOTELY resembling real-time replication, either.
asciilifeform: (backups are a monthly affair, any more often would -- again -- grind whole thing to a screeching halt)
asciilifeform: the most i can offer to anyone is a static copy of the db. and that is supposing that the box comes back up, and stays up.
asciilifeform: it needs a key eater that doesn't saturate the db capacity 24/7 -- yes. a server that doesn't fall down when washington farts on it -- also yes. 'js visualizations' ??
asciilifeform: i utterly do not see , i confess, why phuctor needs 'visualizations' etc
asciilifeform: but it is neither here nor there, the server - falls down whenever enemy wants it to fall down.
asciilifeform: mats: familiar with idea of 'risk homeostasis' ?
asciilifeform: ('but,' , said my internal mircea_popescutron, 'nobody had the slightest inclination to examine anything!' -- but this is besides the point)
asciilifeform: i wrote, in the qntra piece, 'examine debianized boxes for nsaware'. now 'owner' will have a chance to clean up before any mass 'examination' takes place.☟︎
asciilifeform: Framedragger: it is unreachable WHEN IT COUNTS
asciilifeform: the 24/7 ssh pipe i had to the box on dedicated display - also dead.