209500+ entries in 0.124s
mircea_popescu: it's probably
the cause for
the whole spend
thing, as a sort of unexamined insurance.
mircea_popescu: i'm not even proposing we abuse ext, i just wish
to know if it could work. so far, unencouraging.
mircea_popescu: well so
then. it never happened because you never published. dun dun dun.
a111: Logged on 2017-03-10 16:57 asciilifeform: btw i will also put down in
the log, one very simple possible algorithm for a 'txidx-fs' :
mircea_popescu: currently
the dependency is bdb + ext? ; if it becomes ext? it is less not same.
mircea_popescu: so
there's no "replacing". excising part of
the
tumour.
mircea_popescu: this is a misrepresentation :
the
turd is in
there already.
mircea_popescu: (and if
the foregoing didn't happen where you went
to school... you didn't.)
Framedragger: problem is multiple homework/class/job-domains, and
the context-switching :) but yeah
mircea_popescu: you ever go
to school ? what usually happens
there's a chick
there
that's really good pre-puberty.
then she starts bleeding, and she skips some classes / homeworks / attentionpaying. and
then... she can never catch back up again. because interlocking.
☟︎ a111: Logged on 2017-03-11 14:33 Framedragger: kk. so, ok. only
thing is i'm swamped in march, so will have
to wait. (if anyone wants c code i wrote so far, ping me)
mircea_popescu: what i want
to hear is, (preferably proof) as
to why journaling filesystem can't store files in directories!
phf: i
think
the deep value in an exercise like "replace db with a filesystem" is
the reduction of moving parts. ext2 is a straightforward inode based
tree, with a separate relocation phase, etc. journaling adds
the whole overhead (for it's primarily cognitive) of secondary redundancy
that you now have
to factor into all your considerations
Framedragger: kk. so, ok. only
thing is i'm swamped in march, so will have
to wait. (if anyone wants c code i wrote so far, ping me)
☟︎ mircea_popescu: (pretending for a second
the design is sane, which it isn't -- who
the fuck counts by int a set of hashed items omfg)
Framedragger: yes
that's what i meant. which, i dunno, maybe bad assumption of 'only one chain', or sth.
mircea_popescu: can definitely also store by shifted blockheight, 0000/0000 etc. it will still be a
thing as large as
the other one
Framedragger: yeah but i forgot how
to bitcoin. i guess blockheight bad idea?
Framedragger: mircea_popescu: so in your proposed-to-be-tested scheme,
there are
two separate eight-deep
trees? may i ask, why do blocks need
their own
tree - after all, it's just an int. do you expect block number
to overflow an unsigned 32 bit int? because you *really* don't need 8-deep structure for dispersing 2**32 nodes (again:
http://fd.mkj.lt/stuff/fsgraph1.png /
http://fd.mkj.lt/stuff/fsgraph2.png )
mircea_popescu: there's an item in
the specification of journaling
that it must not work which i missed or something ?
mircea_popescu: Framedragger no seriously, nothing wrong with it. now we have numbers.
they're good
to have.
phf: i believe alf even pointed out
the obvious "journaling file systems are going
to journal"
a111: Logged on 2017-03-09 17:41 mircea_popescu: Framedragger
the most pressing matter
to my eyes right now is getting ext2/ext4 benchmarked for our specified purpose.
mircea_popescu: phf well he's considering what he's considering, seeing how he's doing
the measuring. i was kinda biasing
towards ext2 in
the previous discussions (which i guess nobody reads or something ?) , but hey, can't impede man's independent manhood!
a111: Logged on 2017-03-11 13:51 mircea_popescu: contrary
to what ANYONE may pretend, ext4 IS NOT A FS!!!! it's a ridiculous
toy at best.
mircea_popescu: Framedragger in any case i don't expect
to optimize BEFORE DESIGNING holy shit.
talk about early optimizations.
this is
the measuring stage. you optimize nothing.
mircea_popescu: (and if
this is untenable,
THEN
THE DESIGN GETS MODIFIED!!! no fucking "solutions" of shoving shit under carpet and letting mp discover it in 2017 whiole spending however many years eating food we didn't pay for and pretensions
to "engineering" and "intellectual lifge" we don't deserve.)
mircea_popescu: if
the
total number of blocks your machine can produce is 2**4096,
then your design will also store 2**4096 blocks.
mircea_popescu: Framedragger
the point is
that we don't want
to make any more provedly breaking systems.
Framedragger: (you'd need
to have a *lot* of blocks
to have average num of files per second-to-deepest dir be >= 1; i dont
think one needs 8 levels, but i see
the point in
trying
this)
mircea_popescu: so your storage looks like /(blocks,
txn, addies, whatever)/abcd/etc/(abcd.dat or abcd.symlink etc)
mircea_popescu: 2. index
to
those blocks (say, eg,
to find
txn, or anything else) is stored in a SEPARATE dir structure, and at
the bottom
there's simlinks
to
the block files.
mircea_popescu: 1. actual blocks (1mb files) are stored in a directory structure, based on
their hash say.
this is 8 deep because hey, max filecount in a dir, we want
to make a proper system.
Framedragger: also, as asciilifeform said, cache can really confuse
the hell out of any metrics. e.g., disk cache. so i'd need
to probably restart whole box
to be sure (yes lol, but i
think i should)
Framedragger: mircea_popescu:
to be clear,
the way
this would work is,
there'd still be symlinks at
the bottom ends of
the dir structure, pointing
to blocks (which are stored in a single dir, say)?
mircea_popescu: Framedragger suppose you store
the blocks whole. what is
the seek
time of one dozen 1mb files dispersed randomly in a 8 deep directory structure, defined as (time when all are in ram) - (time when call was made) ?
mircea_popescu: "oh i had a job i went
to work
through
traffic every morning"
mircea_popescu: NONE OF
THEM DID ANYTHING USEFUL. AT ALL. BUNCH OF PRETENTIOUS POINTLESS USELESS IMBECILES NOT WORTH PISSING AWAY FROM!
Framedragger: yeah i especially liked
the amazing speed of directory deletion
a111: Logged on 2017-03-11 01:11 mircea_popescu: i can't bring myself
to move my piss away from dks, or anyone in his generation's face.
Framedragger: hey, you wanted some fs
test, i'm just reporting on levels of shittiness found
mircea_popescu: im not using
this
thing. how is it better
than windows ?
Framedragger: asciilifeform: mircea_popescu: for completeness, i should state
that it may be "workable" (in
the sense of slightly less horrible)
to just keep a flat dir
tree structure, one or
two levels deep - if you don't ask fs
to list files in dir and just want
to access filenames you already know, it's ~okay-ish. but i
think i agree
that
the whole fs idea needs
to be dumped, in general
mircea_popescu: ie,
the reason usgtard is all "oh, random is not really broken" when phuctor came out exactly reduces
to "well, it's fucking broken, but you should see
the filesystem!"
mircea_popescu: contrary
to what ANYONE may pretend, ext4 IS NOT A FS!!!! it's a ridiculous
toy at best.
☟︎ a111: Logged on 2017-03-11 04:05 asciilifeform: Framedragger: imho
the 'use existing fs'
thing is a dead end.
a111: Logged on 2017-02-27 12:10 mircea_popescu: dude,
they fucking gutted
them. olympus agreed
to pay
the usg ~70 billion yen in fines, and install obama's children as an "independent outside monitor". whole corp market cap being you know, 1.3trn or some shit. who
the fuck pays 5% of
the market cap as a fine already, what is
this, Совет Экономической Взаимопомощи ?
shinohai: I awake
to
the screeches of `PREEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEET`
this morning with coffee. Hail
to
the
Trumpreich.
mircea_popescu: ben_vulpes
there is an ancient observation (toqueville)
that slavery is not unbearable
to peoples in proportion
to its intensity, but in proportion
to
the velocity of its reduction. he supports it by showing
that
the germans, more abject slaves in 1700
than any central asian people, found
their situation
tolerable ; whereas
the french, significantly freer ~and becoming freer~ found
the uninstantaneous speed of
the change INTO
a111: Logged on 2017-03-11 00:16 asciilifeform: if some unknown d00d had not written
to me last night, even now i might be doing it
Framedragger will check
tomorrow if
the insane size was from his shitty c. but actually, probably not - in ext3/ext4, a folder is an inode and an inode points
to unique data block - minimum size of which is 4k. given an expansive recursive
tree, you get what you get.
☟︎ Framedragger: (btw
the 'creation' is not bottlenecked by python or w/e; straight syscalls and simple c, and
the random hex generator is a small footprint)
Framedragger: (~60k
to store 'transaction' (excluding symlink itself)!!
this is
the price of deeper fs
trees)
Framedragger: (would advise against deep folder structure assuming no concrete reason
to prefer it. just-symlinks (+/-) seems better. but
then not sure if
to
think much of fs anyway.)
Framedragger: oh lord. *creation* of seven-levels-deep directories (in
the format of "6d56/a6f4/f1d5/67a3/505c/a7d0/9c72/6fff/e75e/a482/e36b/6b5b/7421/f9cf/e36a/", with last 8th level being symlink)
takes a long
time, and is also space-wasteful on ext4.
to 'store' 1k
transactions it
takes ~0.41s and
takes up 59M of space.
this without actual symlinks (should be fast but should check later). *removal* is ~0.45s
mircea_popescu: phf entirely idle example, no more important
than strand of hay, which is why chosen here, illustrates beautifully.