log☇︎
71000+ entries in 0.459s
mircea_popescu: the noion that hdd is usable or useful is a cute pipe-dream of the web generation, unsupported in cold reality. ☟︎
mircea_popescu: http://btcbase.org/log/2017-03-10#1624307 << it is the case, one collision a year sort of deal./ ☝︎
Framedragger: mircea_popescu: but if you read further down, we were saying that it may be possible to just access a raw block device without kernel module :)
a111: Logged on 2017-03-10 17:16 asciilifeform: the Right Thing would probably be to have a very simple kernel driver that takes a specially-marked disk partition and gives userland trb linear use of it, as plain array
mircea_popescu: http://btcbase.org/log/2017-03-10#1624297 << this sounds like a quite elegant solution, yes. ☝︎
mircea_popescu: http://btcbase.org/log/2017-03-10#1624278 << this blowing seems altogether a foregone conclusion. ☝︎
asciilifeform: http://btcbase.org/log/2017-03-10#1624149 << this is a hosting co in hk, and domain peddler. betcha ~all~ of their shithosting offerings, are debianized. ☝︎
asciilifeform: ( speaking of golden toilet, factory sent me a gold-plated ruler! complete with drill guide. -- yes, made of finest pcb . for lulz. )
asciilifeform: this, if it was a mystery in 2014, is plainly visible today imho
asciilifeform: mircea_popescu: i am not a psychiatrist, do not specialize in 'patient has arms, legs, nerves are fine, but he won't move'em'
mircea_popescu: asciilifeform usg really can't make anything. you're basically saying "siliconed botoxed cali chick could cook a decent meal." of course she could. of course she won't.
asciilifeform: kinda disgraceful, not like there isn't a usg 15nm fab or two still standing in upstate new york
mircea_popescu: not unlike tv advertising to my chained girl, "a better life". really ? i think if she wanted that "better" she'd have found her way by now.
Framedragger: well, *in principle* deeper traversal should really not be a problem at all. in practice - guess we shall see
a111: Logged on 2017-03-10 16:55 asciilifeform: ... so that a 256-bit turd, e.g., 3ec455a2a84e978687a3990cec73f36b324fbd28e03603c6d9fc52018b001558, can be taken and matched to a block # where said tx lives.
mircea_popescu: the people who actually have to work for their accomplishments a la beerbohm would no doubt be quite upset.
Framedragger: apparently calling fclose() while syscall is running on that descriptor in another thread is 'not a good idea' (outcome maybe platform/implementation specific), but that's an avoidable corner case.
Framedragger: (anyway, will share tool i wrote if only because couldn't find anything fitting the task out there, but need to polish a bit first, etc.)
Framedragger: some (very initial) symlink stats - more stuff will have to wait - given a "here are 1mn 'transactions' which symlink to files, resolve links and read from linked files in random order for 100mn times" setup, with one-folder-deep structure, like so: "simple_f1/e5/e5edc34c57d5ea2ea99cfe16d04655aa000c3d7f268022d2b21f95928fa34674 -> /files_f1/99997.txt", most basic stats are: ☟︎
asciilifeform: i contemplate it as more of a 'demolition charge', when trbi exists.
trinque: there's what, a million or so in the stash? plenty to build asciilifeform's fab.
asciilifeform: maybe, who the fuck knows, collect enough data re winblows rng defects, via a future 'uci', to break satoshi's keys.
asciilifeform: it is a general-purpose shitrng telescope
trinque: we'll make a CL lib for p ☟︎
asciilifeform: and a new everything-else
asciilifeform: trinque: then i'll need a cl pgp parser ☟︎
trinque: I would, in fact, at this point drop postgresql for a persistence layer for CLOS that did not use the thing under the hood.
asciilifeform: Framedragger: i did a good measure of tweaking, and exhausted current time budget for that chore
Framedragger: yeah, i see what you mean, and everything. i'm not convinced that phuctor db is using the most it can from postgres, but neither you nor me have time to investigate this presently, i guess (and i may not be able to do a great job there anyway)
a111: Logged on 2017-02-19 03:54 asciilifeform: (iirc we had a thread where i described how corporate ameritards, if given a problem like phuctor, would happily soak up a few $mil and megawatt of iron)
asciilifeform: using general-purpose db where performance is a critical concern, inevitably gets you http://btcbase.org/log/2017-02-19#1615434 . ☝︎
Framedragger: tree rebalance is a feature of a balance binary tree which is sometimes the right tool for the job, cmon :)
asciilifeform: Framedragger: all you need is to check a read candidate against the current list of active writes. ☟︎
Framedragger: yes, but how to cheaply ensure the latter. you won't have a shitload of locks now will you
asciilifeform: (i can make a 400GB file on my disk, right now, and it will cost me a ~constant factor vs 'raw disk' i/o)
asciilifeform: trb will not even need to know whether it was given a real disk, or flat file.
asciilifeform: in fact it can be simply a file descriptor given on command line.
Framedragger: well, 'disk' could just be partition which isn't a bad thing anyway, right?
asciilifeform: Framedragger: there is a udev trick, iirc, for giving particular users access to a given raw block dev
Framedragger: also, wonder if there could be a relevant linux cap'ability for allowing raw access to given /dev/block. but maybe not.
Framedragger: asciilifeform: yeah! i may have time / want to do this in ~april. just to be clear, there's really no way to have underlying userland fs allocate a contiguous block, right?
asciilifeform: at any rate this is a 'theoretical' problem, imho 'not burning'.
asciilifeform: ( and ~yes~ right now it is very easy to see 'who wired to whom' by looking at ssh, but in the future we will presumably route multi-hop, via a gossiptron, and it will be less obvious )
asciilifeform: and yes i could put in a '--noplaintextpeer=x,y,z...' flag, BUT
asciilifeform: (would like to test 2way 'wiring' first on a non-infrastructural set of boxen )
trinque: if you want a reciprocal wire lemme know
Framedragger: that's a horrible idea for sure, but hey, free space to cheat in
asciilifeform: why would you ever not have the nx bit set for a motherfucking table
asciilifeform: ( however it will be intuitively obvious, to anyone familiar with x86/x64 and similar archs, that this is physically possible. i.e. you never actually need to store a contiguous empty GB in memory, if you can manipulate the page table )
asciilifeform: you can use x64's page table to 'cheat' and store a sparse form ! ☟︎
asciilifeform: Framedragger: just as you do not need 366 people in a room for a birthday collision.
Framedragger: aha, so there's high probability that there will *be* a collision across the entire space.
Framedragger: and yeah, this 'generate collisions' scenario seems like a harder version of 'just mine'
asciilifeform: (enemy is welcome to bash his skull to a pancake against the cement wall)
asciilifeform: much cheaper to reindex a trb node, what, 1 night, than for enemy to bake new asics.
asciilifeform: (you could in fact use a repurposed miner, for this, as i understand)
asciilifeform: if you start seeing them ~regularly~ (say, trb makes it to 50 yrs from nao) you put in a l2 table.
Framedragger: there's an assumption as max num of collisions here, of course, but obvs in practical terms it's a very safe assumption...
asciilifeform: i am nearly convinced that he was, at one time, a microshit employee.
trinque: that's right, it's doing a sleep or two of several minutes
trinque: god forbid the disk always be in a coherent state, eh? ☟︎
Framedragger: i guess one can imagine a single sequence of tx then, simply.
asciilifeform: now store each tx on a 4096bt 'cylinder' boundary, and you're golden
Framedragger: it's a really Good Thing that the hashing function which spits out transaction hashes gives *uniform distribution*. no congestion / too many collisions expected, and this scheme leverages that.
asciilifeform: i was recently considering implementing a similar scheme for phuctor, and realized today that it would work entirely well in trb.
Framedragger: (i see how good it is to be aware of how actual disks read data here. some theoretician would propose a pointer-exact-location scheme instead...)
asciilifeform: so in practice, you can have O(1) seeks ~while~ storing the blocks back-to-back in a classical trb.
asciilifeform: Framedragger: the 1024bt thing works in all cases, it simply means that you don't have to point to beginnings of blocks, but to a place very near the tx
Framedragger: well *that sounds like a very decent idea*. :)
Framedragger: right! ahh that's nice. (so just to clarify, the 1024 byte block trick wouldn't work if there's a collision (unless additional budget / w/e))
asciilifeform: and if set, you gotta look at the next machine word (each entry has a certain amount of space for collisions)
asciilifeform: if you're 'rich', it can be a table entirely like l1
Framedragger: bear with my slowness, can you clarify how it looks like if there's a collision in the initial lookup?
asciilifeform: point being, to point at a tx sitting on a multi-TB disk, you don't need to be able to address it ~bytewise~
asciilifeform: then a 31bit (remember, highest bit means 'go to l2 table') offset gives you a 2^^41 bit space in which your table entry can point to the beginning of a block.
ben_vulpes: asciilifeform: not a c proggy, no
asciilifeform: (and thereby a 16GB l1 table would give you wholly in-memory index for ~all tx lookups)
asciilifeform: ok, now question for ben_vulpes , trinque , mircea_popescu , et al : anybody got a quick c proggy that will eat blkcut's blocks and produce a linear list of tx ? i'd like to actually calculate the current 32bit tabular collision rate.
asciilifeform: the Right Thing would probably be to have a very simple kernel driver that takes a specially-marked disk partition and gives userland trb linear use of it, as plain array ☟︎☟︎
asciilifeform: it has obvious minus, in that you need a kernel driver
asciilifeform: yes, first time, i kept waiting for someone to open a schoolbook and describe this very elementary algo
Framedragger: this is quite nice, and as you say, seek operation already gives a small chunk which should cover most/all tx for current state of affairs (total number of transactions)...
Framedragger: oh i finally understood, literally all there is when one seeks to location 3ec455a2 is a list of block numbers. (or single block number.)
asciilifeform: one seek, on a standard disk, gives you at least 4kB (blocks are 512b, but no disk made in past 20 years actually ~stops at~ 512)
asciilifeform: also on a reasonable box you can keep the l1 table in ram
Framedragger: trinque: it's just a kindergarten way of wrapping up some syscalls. will obviously benchmark outside it later. i wasn't completely certain that my tool wouldn't trash the host fs. :)
Framedragger: aha, right! so it's basically a (small) hashtable.
asciilifeform: picture a 16GB contiguous block of ssd. you cut it up so that it corresponds to 2^32 x 32bits.
asciilifeform: btw i will also put down in the log, one very simple possible algorithm for a 'txidx-fs' : ☟︎☟︎☟︎
asciilifeform: and likewise when a new block comes, all of the tx in it, get indexed this way.
asciilifeform: ... so that a 256-bit turd, e.g., 3ec455a2a84e978687a3990cec73f36b324fbd28e03603c6d9fc52018b001558, can be taken and matched to a block # where said tx lives. ☟︎
Framedragger: will get a way to test real disk soon, didn't want to run on personal trashy PC, hence shitty server
Framedragger spent longer than wants to admit sorting out his heap and valgrind'ing. too much python is bad for a person
Framedragger: getting ~4000-7000ns for symlink resolution to real path for a 1mn symlink dir structure, e.g.
mircea_popescu: yeah, it's not lost on me that i'm the loudest preacher of continence in a cloister of monks muchly more restrained and disciplined than myself.
asciilifeform: http://btcbase.org/log/2017-03-10#1624188 << i built myself a 'games box' , not long ago, 'to unwind', but turned out that the part of my head that was able to enjoy this, had atrophied, found myself constantly 'if i'm this awake, oughta be running $experiment, finishing up $unfinisheds', etc. ended up giving the (princely) fuckofftron to brother to play with. ☝︎
Framedragger: i like my rc airplanes. "the will of history necessitates you to X" has a marx'ified hegelian vibe :p ☟︎☟︎☟︎
mircea_popescu: well, anyway, i suppose it is a good thing the intelligent people involved could you know, pursue their own self-determined will and etcetera. who knows how many glorious flights of amateur rc airplanes this obviously reasonable arrangement gave the world!
asciilifeform: one perverse fact, is that microshit had a winblowz ecosystem for it, ready to go, it could have been he 'designated winner' in intel's place. except -- no, because gotta target the konsoomer chump junk ...
asciilifeform had a prof in uni who taught systems architecture (cpu design) with itanic as 'what not to do'