log☇︎
70600+ entries in 0.556s
Framedragger: (and yah i'm sure there's lots of crud in the former worth dropping..)
mircea_popescu: im not against the idea in the slightest. i'm just very unpersuaded by the theory hard drives work, to any spec, in any sanemanner. ☟︎
mircea_popescu: i don't think anyone correctly represents the tower of accidental good luck that stands between a disk seek and your desired pron.
Framedragger: mircea_popescu: (i thought with "sounds great on paper" you were responding to possible kernel module workarounds. but if you're against the *whole* idea, fair enough.)
a111: Logged on 2017-03-10 17:14 Framedragger: i'm not sure if you do need driver
mircea_popescu: i said years back the race is won. but meanwhile the usg utterly lost it, ot little concern of anyone, itself included.,
asciilifeform: mircea_popescu: i am not a psychiatrist, do not specialize in 'patient has arms, legs, nerves are fine, but he won't move'em'
mircea_popescu: asciilifeform i have to protest re the scheme. i'm merely needling you in my spare time.
asciilifeform: i guess some bean counter decided that this gambit has higher ev ~in btc~ than , e.g., nsa mining
mircea_popescu: not unlike tv advertising to my chained girl, "a better life". really ? i think if she wanted that "better" she'd have found her way by now.
mircea_popescu: ah, i don;t particularly care. anyone googling me who ends up on etoro deserves his punishment.
asciilifeform: denominated in, i can only guess, ethertardium?
Framedragger: kk as long as we're clear - i've seen those "but what would happen with 1000s of symlinks in single folder!1" comments so thought that should be tested, etoo.
mircea_popescu: Framedragger i didn't intend any distinctiuon betrween "symlink resolution" "directory traversal" et al.
mircea_popescu: http://btcbase.org/log/2017-03-10#1624235 << i thought this was entirely obvious, but yes. ☝︎
asciilifeform: aha, i recall
mircea_popescu: asciilifeform i did foreshadow.
mircea_popescu: it ~should~ work, i ask because man who burned tongue on soup
Framedragger: that said, i'm pessimistic about this whole ext4 thing just like asciilifeform. no reason not to direct time into his latest proposal
Framedragger: oh wait, i was wrong re thread-safe functions. kernel takes care of that. has internal locking mechanism when dealing with fread/fwrite etc.
mircea_popescu: can i run 64 in parallel ?
Framedragger: (anyway, will share tool i wrote if only because couldn't find anything fitting the task out there, but need to polish a bit first, etc.)
Framedragger: given time, i was also meaning to run lots of strace to see what is in actuality happening with multiple symlinks etc, but maybe sisyphus
asciilifeform: i originally favoured it on account of simplicity
Framedragger: no disagreement tbh, i probably know your reasoning
asciilifeform: at this point ftr i'm quite firmly convinced that baking in massive open sores shitbucket, e.g. ext4, is The Wrong Thing
Framedragger: asciilifeform: you're right, i was meaning to reset caches (`echo 3 >/proc/sys/vm/drop_caches` should be enough i think), but then forgot
Framedragger: (5ms read on an ssd ain't too pretty, but would have to check distribution of these kinds of tails, i guess.)
Framedragger: Processed 1000000 lines (min/max target numbers seen: 1, 100000) - beginning benchmarks now... [i.e., 'target block' numbers are in this range. 1m symlinks point to 100k files randomly.]
asciilifeform: i contemplate it as more of a 'demolition charge', when trbi exists.
asciilifeform: i did say, 'martian example', not immediate plan to action.
asciilifeform: at some point, yes, i'll rewrite it again.
trinque: nah nah I have no prescription for phuctor
asciilifeform: which i currently don't have.
asciilifeform: trinque: then i'll need a cl pgp parser ☟︎
trinque: I would, in fact, at this point drop postgresql for a persistence layer for CLOS that did not use the thing under the hood.
asciilifeform: Framedragger: i did a good measure of tweaking, and exhausted current time budget for that chore
Framedragger: yeah, i see what you mean, and everything. i'm not convinced that phuctor db is using the most it can from postgres, but neither you nor me have time to investigate this presently, i guess (and i may not be able to do a great job there anyway)
a111: Logged on 2017-02-19 03:54 asciilifeform: (iirc we had a thread where i described how corporate ameritards, if given a problem like phuctor, would happily soak up a few $mil and megawatt of iron)
asciilifeform: Framedragger: i am increasingly finding that 'general purpose db' is, like the infamous 'vise-grip', The Wrong Tool For Every Job ☟︎☟︎
Framedragger: of course, i agree, i'm just saying it's not "inherent problem of all db omg"
Framedragger: i guess i can see that
Framedragger: (i guess depends on nature of parallel work. e.g. if it's some kind of sequential re-confirm, launch threads with different segments, etc.)
asciilifeform: (i can make a 400GB file on my disk, right now, and it will cost me a ~constant factor vs 'raw disk' i/o)
asciilifeform: for completeness, i will note that you can model any 'whole disk' variant with an enormous flat file, in userland, at some speed penalty. ☟︎
asciilifeform: Framedragger: you asked at same time as i .. lol
Framedragger: ah, that's the q i guess
asciilifeform: ( i wonder if there are any traditional fs that will actually give you 16G of contiguous blocks, if asked nicely ) ☟︎
Framedragger: asciilifeform: yeah! i may have time / want to do this in ~april. just to be clear, there's really no way to have underlying userland fs allocate a contiguous block, right?
asciilifeform: and yes i could put in a '--noplaintextpeer=x,y,z...' flag, BUT
asciilifeform: i'd like to think of an elegant solution to this ☟︎
asciilifeform: trinque: fwiw i still know of at least 1 unsolved tickle re 'wires' -- how to avoid multiple redundant connections
asciilifeform: i very well might, shortly.
asciilifeform: i meant 'cheats' that use the machine as-prescribed and save multiple GB.
Framedragger: heh you can even use things like nx bit if not wanting to reorganize table, i would guess!
asciilifeform: ( however it will be intuitively obvious, to anyone familiar with x86/x64 and similar archs, that this is physically possible. i.e. you never actually need to store a contiguous empty GB in memory, if you can manipulate the page table )
asciilifeform: but THIS i will leave as an exercise for the reader.
asciilifeform: the ~other~ thing i'll point out, is that you do not in fact need 16GB of ram, to implement that table , if you can do it in the kernel
asciilifeform: i like 32 (recall, 31, we need 1 bit for collision marker) because it translates cleanly to our physical machine.
asciilifeform: i'll round this out by observing that potentially you might be able to get away with less than 32bits of table (i.e. less than 16GB of space)
Framedragger: asciilifeform: yeah, i see what you mean
a111: Logged on 2017-03-10 18:25 Framedragger: hard to believe, but then, i should read logs at least
Framedragger: hard to believe, but then, i should read logs at least ☟︎
Framedragger: i'm guessing that an interested state could totally do that
Framedragger: while at it, and i'm guessing it's in the logs.., i wonder when was the last time some tried to come up with an approximation what the costs to get to >50% network hashing power would be ☟︎
asciilifeform: (you could in fact use a repurposed miner, for this, as i understand)
Framedragger: yeah. hmh - i guess that's it. literally. nice scheme
asciilifeform: (i would hope that l1 will last until trbi..)
asciilifeform: ( i can already picture mircea_popescu spitting out his breakfast, 'modern hdd dun have cylinder, you nut' -- except, it in effect DOES, fetches massive blox , whether mechanical or ssd, by design, for ages now ) ☟︎
asciilifeform: Framedragger, ben_vulpes : i predict that there will be few enough collisions that you can shunt ~all~ collision cases off to an O(N) lookup table, for next decade, with ~0 measurable loss of avg.case performance.
asciilifeform: i am nearly convinced that he was, at one time, a microshit employee.
asciilifeform: trinque: i've found that 'kill', which syncs the db, followed by kill -9, which nukes shitoshi's pointless wait idiocy, worx 100%. ☟︎
Framedragger: hmh, at least functions make up not the *worst* interface seen, but still lotsa work and weird mutable shit sprayed all around, i imagine
asciilifeform: (as far as i can see, it is unused code)
Framedragger: yeah that's one reason i'm not too attracted to trb, tbh, the amount of sewage gruntwork required to decouple shit from the monolith.
Framedragger: decent ideas, what can i say
Framedragger: i guess one can imagine a single sequence of tx then, simply.
asciilifeform: i was recently considering implementing a similar scheme for phuctor, and realized today that it would work entirely well in trb.
Framedragger: (i see how good it is to be aware of how actual disks read data here. some theoretician would propose a pointer-exact-location scheme instead...)
Framedragger: yeah i forget sometimes. fixed block length is nice for this...
asciilifeform: ok, now question for ben_vulpes , trinque , mircea_popescu , et al : anybody got a quick c proggy that will eat blkcut's blocks and produce a linear list of tx ? i'd like to actually calculate the current 32bit tabular collision rate.
Framedragger: have separate service taking care of that? i mean, kernel driver is this kind of 'externality', too (and also ring0)
Framedragger: here i have an ssd seek profiler which just needs root
Framedragger: i'm not sure if you do need driver ☟︎
asciilifeform: i dare say the tabular algo is simple enough to actually be implementable.
Framedragger: at least i have the excuse of not having looked at the bdb problem / staying away from trb for the time being :p
asciilifeform: yes, first time, i kept waiting for someone to open a schoolbook and describe this very elementary algo
Framedragger: is this the first time you articulated this approach here? i think that's the best on can have for fs-tx-db
Framedragger: oh i finally understood, literally all there is when one seeks to location 3ec455a2 is a list of block numbers. (or single block number.)
asciilifeform: Framedragger: it is literally the smallest number of moving parts i can think of , that'd do the job.
Framedragger: asciilifeform: hmm, very nice. i suppose it's as close to fixed-length as is possible given current bitcoin
Framedragger: trinque: it's just a kindergarten way of wrapping up some syscalls. will obviously benchmark outside it later. i wasn't completely certain that my tool wouldn't trash the host fs. :)
asciilifeform: how to ~write~ to this data structure is, i think, obvious to the reader, i do not need to describe.
asciilifeform: if there are no collisions (i.e. there is no other tx on the disk whose id begins with 3ec455a2) then that entry in the array IS the block index where the tx lives.
asciilifeform: btw i will also put down in the log, one very simple possible algorithm for a 'txidx-fs' : ☟︎☟︎☟︎
Framedragger: what i want to do later when i find time is, actually read file, too, of course.
Framedragger: aha, right. i haven't even looked at bdb.
a111: Logged on 2017-03-10 03:46 mircea_popescu: Framedragger i don't get it, you graphed some functions ? or ?
asciilifeform: trinque: i.e. your login is 'tunnel', not 'deedbot'
trinque: asciilifeform: getting rejected by dulap with that ssh key I gave ya.