2900+ entries in 0.019s
Framedragger: well i did, but you disagreed with the definition
Framedragger: (re. x windows, obviously one can have different ways and capabilities per particular system/process, etc)
Framedragger: i only see folks denying the concept of isolation, i maintain that it's not altogether a useless concept. disagreement in premises
Framedragger: but rather more to do with deployment, reproducability, ops, etc
Framedragger: that said, docker's 'main aim' isn't even isolation for the sake of security, so my weak point is even weaker.
Framedragger: "used docker" therefor "defined by tshirt" q.e.d.
Framedragger: a process by which one restricts the system-provided resources that a particular entity (just "process", for now, say) is exposed to; the effect of which is that the entity functions as if in a stand-alone system dedicated to its purpose
Framedragger: *sigh* i'll grant it that concepts are leaky and can't apply hammer to all problems etc etc; but you can't pretend that the core concept of isolation is not useful. just isolating fs, process, and network is already useful.
Framedragger: any suggestions for isolation from folks here? just cgroups/chroot?
☟︎ Framedragger: i expect folks regard the "crash and go away" as a feature - just restart it bro, etc.
Framedragger just had some cold borscht. << amazing and recommend for those who haven't tried
Framedragger: (i'll just mention that for persistent data, you are supposed to use bind mounts, not internal docker storage.)
Framedragger: ah i may have read it, did it mention DBs, too, as in "why the fuck would you put db in there"
Framedragger: i have some good memories from using bsd jails some years ago. the core concept of isolation (fs, process, network, etc) is not bad. providing integrated interface not a bad thing, either. problem with docker is it doesn't do it in a consistent way, is too bloated, is ~proprietary +/-, and does the abstraction in a way that invites lazy people to be even more lazy and reckless.
Framedragger: currently stealing booze from that inventory. some good old shit in there i tell you
Framedragger: ah trinque up'd himself just to remind me that docker is shit lol
Framedragger: apparently offloads export/computation to neo4j, a hipsta graph db
Framedragger: yeah the sarcasm is probably not black enough tbh
Framedragger: i'm sure it's justified by the other amazing parts of the fs, and when we look at the other parts we'll see glorious code that works
Framedragger: but good to have concrete reproducible data to back it up, sure.
Framedragger: mircea_popescu: you sure you want benchmarks? /me thinks it's a lost cause
Framedragger: you're right, the growth *won't* be linear. but...
Framedragger: 59M for 1000tx. *before* symlinks are stored. (granted, they'll be small)
Framedragger: well. if we do linear extrapolation, bout 14400 terabytes for transaction index given 250mil transactions....
Framedragger: mircea_popescu: which means that i don't see how it could work even given mystery-amazing fs performance (...), *space-wise*.
Framedragger: *59M* for 1000 'transactions', *before* symlinks are even saved. just the folders (7-level-deep folders.)
Framedragger: problem is multiple homework/class/job-domains, and the context-switching :) but yeah
Framedragger: kk. so, ok. only thing is i'm swamped in march, so will have to wait. (if anyone wants c code i wrote so far, ping me)
☟︎ Framedragger: yes that's what i meant. which, i dunno, maybe bad assumption of 'only one chain', or sth.
Framedragger: yeah but i forgot how to bitcoin. i guess blockheight bad idea?
Framedragger: but if it's for 'defensive' benchmarking, sure. just pointing out.
Framedragger: mircea_popescu: so in your proposed-to-be-tested scheme, there are two separate eight-deep trees? may i ask, why do blocks need their own tree - after all, it's just an int. do you expect block number to overflow an unsigned 32 bit int? because you *really* don't need 8-deep structure for dispersing 2**32 nodes (again:
http://fd.mkj.lt/stuff/fsgraph1.png /
http://fd.mkj.lt/stuff/fsgraph2.png )
Framedragger: (you'd need to have a *lot* of blocks to have average num of files per second-to-deepest dir be >= 1; i dont think one needs 8 levels, but i see the point in trying this)
Framedragger: also, as asciilifeform said, cache can really confuse the hell out of any metrics. e.g., disk cache. so i'd need to probably restart whole box to be sure (yes lol, but i think i should)
Framedragger: mircea_popescu: to be clear, the way this would work is, there'd still be symlinks at the bottom ends of the dir structure, pointing to blocks (which are stored in a single dir, say)?
Framedragger: yeah i especially liked the amazing speed of directory deletion
Framedragger: hey, you wanted some fs test, i'm just reporting on levels of shittiness found
Framedragger: asciilifeform: mircea_popescu: for completeness, i should state that it may be "workable" (in the sense of slightly less horrible) to just keep a flat dir tree structure, one or two levels deep - if you don't ask fs to list files in dir and just want to access filenames you already know, it's ~okay-ish. but i think i agree that the whole fs idea needs to be dumped, in general
Framedragger will check tomorrow if the insane size was from his shitty c. but actually, probably not - in ext3/ext4, a folder is an inode and an inode points to unique data block - minimum size of which is 4k. given an expansive recursive tree, you get what you get.
☟︎ Framedragger: (btw the 'creation' is not bottlenecked by python or w/e; straight syscalls and simple c, and the random hex generator is a small footprint)
Framedragger: (~60k to store 'transaction' (excluding symlink itself)!! this is the price of deeper fs trees)
Framedragger: (would advise against deep folder structure assuming no concrete reason to prefer it. just-symlinks (+/-) seems better. but then not sure if to think much of fs anyway.)
Framedragger: oh lord. *creation* of seven-levels-deep directories (in the format of "6d56/a6f4/f1d5/67a3/505c/a7d0/9c72/6fff/e75e/a482/e36b/6b5b/7421/f9cf/e36a/", with last 8th level being symlink) takes a long time, and is also space-wasteful on ext4. to 'store' 1k transactions it takes ~0.41s and takes up 59M of space. this without actual symlinks (should be fast but should check later). *removal* is ~0.45s
Framedragger: they must have really low self esteem to require this kind of labyrinthine way of pretending to themselves to be real important nao
Framedragger: yeah but that's what i meant by same, gave closed copy the source version of which was available (and known to him, presumably?)
Framedragger: lol, what can i tell you, sad state of affairs
Framedragger: i don't think he meant himself, but rather other, external wots.
Framedragger: ~maybe up for it but *that*'d have to wait till later
Framedragger: ...yeah, i think that the proper way to do it then is to take entire disk, populate it with things to a high degree and (somehow) with enough spatial dispersion, and benchmark, and restart entire thing between benchmarks.
Framedragger: right, yeah, but the (really) random seeks.. i hoped to tumble it a bit at least
Framedragger: maybe another caching layer is still caching, hrgh.
Framedragger: i must write a short comedy screenplay for how discussions develop in #trilema. "i want to take a picture" ... ... ... [intense calculations of incident gamma rays to reclaim lost pixels in images] [furious rehash of late renaissance representationalism]
Framedragger: asciilifeform: something about needing to discard 'invalid values' with rdtsc (even when set up correctly - apparently best to 'pin' particular cpu, etc etc..) oh god.
Framedragger running on barebones xeon (not docker) nao, fwiw
Framedragger: asciilifeform: i thought that's why folks use CLOCK_MONOTONIC? i guess the irony is that it's not..monotonic in the end, still. :(
Framedragger: ..but if one wants to measure fs performance, he's stuck with syscalls.