asciilifeform: at any rate imho the flat-file variant oughta be tried 1st.

asciilifeform: but you'll need to dedicate an entire disk

asciilifeform: Framedragger: there is a udev trick, iirc, for giving particular users access to a given raw block dev

asciilifeform: Framedragger: you asked at same time as i .. lol

asciilifeform: ( i wonder if there are any traditional fs that will actually give you 16G of contiguous blocks, if asked nicely ) ☟︎

asciilifeform: !~later tell Framedragger the table system can be easily prototyped , without writing any kernel code, by using 16G file on ordinary disk. (at the obvious speed penalty.)

asciilifeform: at any rate this is a 'theoretical' problem, imho 'not burning'.

asciilifeform: ( and ~yes~ right now it is very easy to see 'who wired to whom' by looking at ssh, but in the future we will presumably route multi-hop, via a gossiptron, and it will be less obvious )

asciilifeform: by looking at who never plaintcp's to whom on 8333 !

asciilifeform: BUT this would make it very easy for the enemy to determine who is 'wired' to whom

asciilifeform: (can, say, ban them from talking on 8333 to each other in iptables, but this is stupid because then they will not share the perfectly-good ip with OTHER peers)

asciilifeform: i'd like to think of an elegant solution to this ☟︎

asciilifeform: (right now, e.g., zoolag and dulap happily also find each other over ordinary peer table and connect plaintext)

asciilifeform: trinque: fwiw i still know of at least 1 unsolved tickle re 'wires' -- how to avoid multiple redundant connections

asciilifeform: 'if one synchronized swimmer drowns, must the others?' (tm) (unknown)

asciilifeform: (would like to test 2way 'wiring' first on a non-infrastructural set of boxen )

asciilifeform: congrats trinque !!

asciilifeform: rather than 'let's put cock into gas tank because it fits'

asciilifeform: i meant 'cheats' that use the machine as-prescribed and save multiple GB.

asciilifeform: you don't need any moar kludges, the scheme described, will work.

asciilifeform: why would you ever not have the nx bit set for a motherfucking table

asciilifeform: ( however it will be intuitively obvious, to anyone familiar with x86/x64 and similar archs, that this is physically possible. i.e. you never actually need to store a contiguous empty GB in memory, if you can manipulate the page table )

asciilifeform: but THIS i will leave as an exercise for the reader.

asciilifeform: you can use x64's page table to 'cheat' and store a sparse form ! ☟︎

asciilifeform: the ~other~ thing i'll point out, is that you do not in fact need 16GB of ram, to implement that table , if you can do it in the kernel

asciilifeform: i like 32 (recall, 31, we need 1 bit for collision marker) because it translates cleanly to our physical machine.

asciilifeform: depending on the collision rate.

asciilifeform: i'll round this out by observing that potentially you might be able to get away with less than 32bits of table (i.e. less than 16GB of space)

asciilifeform: ought to help those folx for whom kindergarten happened long, long ago, to picture the subj of today's thread.

asciilifeform: http://preshing.com/20160314/leapfrog-probing << from same www, also notbad.jpg illustration of schoolbook hash tables.

asciilifeform: Framedragger: http://preshing.com/20110504/hash-collision-probabilities << re earlier. quick likbez on collision probability calculation using 'birthday theorem'

asciilifeform: whereas to someone without (e.g. the folx here) it is ~infinite.

asciilifeform: because the cost, to someone who ALREADY has fab capability , is miniscule

asciilifeform: Framedragger: the other thing is that it is quite impossible to come up with meaningful cost figure

asciilifeform: Framedragger: there was old thread with mircea_popescu , where he stated that usg and china attempted it at same time, and perma-deadlocked ☟︎

asciilifeform: (enemy is welcome to bash his skull to a pancake against the cement wall)

asciilifeform: so this scenario, imho, is out.

asciilifeform: much cheaper to reindex a trb node, what, 1 night, than for enemy to bake new asics.

asciilifeform: at any rate, working around this idiocy would be very cheap -- say we index by top32 of keccak(txid) instead of plain txid. in the event of. ☟︎

asciilifeform: (you could in fact use a repurposed miner, for this, as i understand)

asciilifeform: but this'd cost'em moar than simply to mine.

asciilifeform: so that they collide

asciilifeform: enemy might start to 'mine' txids

asciilifeform: do the arithmetic, it isn't as if anyone can cancel the 'block per 10min' thing. ☟︎

asciilifeform: (i would hope that l1 will last until trbi..)

asciilifeform: if you start seeing them ~regularly~ (say, trb makes it to 50 yrs from nao) you put in a l2 table.

asciilifeform: if you get moar collisions, you simply end up in the slowtable moar often.

asciilifeform: and you have here ~whole thing.

asciilifeform: disklookup is the 'go to nth cylinder and before you get to n+1st cylinder, THERE's yer tx' from earlier still.

asciilifeform: 'linearlookup' is the slow-table from earlier

asciilifeform: (lookup table consisting of, literally, rows of full txid followed by where-on-disk, that you'd chug through whenever collisionbit is set)

asciilifeform: Framedragger, ben_vulpes : i predict that there will be few enough collisions that you can shunt ~all~ collision cases off to an O(N) lookup table, for next decade, with ~0 measurable loss of avg.case performance.

asciilifeform: nobody else does this!11

asciilifeform: i am nearly convinced that he was, at one time, a microshit employee.

asciilifeform: idiot put them ~everywhere

asciilifeform: (some time grep for the sleeps, it is instructive) ☟︎

asciilifeform: 0 reason to let it

asciilifeform: trinque: i've found that 'kill', which syncs the db, followed by kill -9, which nukes shitoshi's pointless wait idiocy, worx 100%. ☟︎

asciilifeform: and reduce all of the bdbisms to calls to table ops.

asciilifeform: the gnarly part is to dissect all of shitoshi's idiot special cases (e.g. 'ReadOwnerTxes')

asciilifeform: imho the 'hard part' is not even to implement this table, it is freshman homework, but to unravel the liquishit in trb and learn where to even put the lookup/write ! ☟︎

asciilifeform: the table can index individual tx.

asciilifeform: now store each tx on a 4096bt 'cylinder' boundary, and you're golden

asciilifeform: btw mircea_popescu's suggestion from last night, to dispense with 'blocks' as separate class of object, and simply store sequence of tx, with 'this was in block B' field added -- would work quite well with this scheme.

asciilifeform: i was recently considering implementing a similar scheme for phuctor, and realized today that it would work entirely well in trb.

asciilifeform: if you want to consider 'what would give best performance on the iron we have, with minimal moving parts, with the trb we have' this is what comes out.

asciilifeform: so in practice, you can have O(1) seeks ~while~ storing the blocks back-to-back in a classical trb.

asciilifeform: that is, you don't need 8 bytes to say which 4kB slice has the beginning ☟︎

asciilifeform: except that you don't actually need byte-addressing !

asciilifeform: however block index is not large -- say we have 500,000 blocks, and you want to know where on disk lives its beginning, and your disk takes 64bit lba addressor. so you need 500,000 * 8bytes == 4,000,000 bytes ! 3 floppy's worth