3000+ entries in 0.019s
Framedragger: i actually did check what was available before deciding on clock measurement yknow. but yah, need to try rdtsc
Framedragger: incrementor where, in another thread so that one may have more context switches for fun? those calls are blocking..
Framedragger: ah that i've heard of. not without its problems iirc
Framedragger: now they gonna talk about how one cant measure anything and need phuctor-made superclocks. kk. sure. but this is as accurate as it gets, sir
Framedragger: allegedly this is what folks use for high precision profiling. i checked and wrote trivial version of that. but yah, clock_gettime()
Framedragger: ACID is not a bad idea yknow. i wonder if you could accomplish what you wanted with dirty slave reads via streaming replication or sth. but, eh. bbl, food
Framedragger: also just ftr, a b-tree by definition has every part of it be of the same depth. 'unbalancing b-tree' doesn't even make sense. but you could have 'just a normal tree', and one can implement custom indices. just pointing out
Framedragger: (also just for poterity it's not like you can't tell a decent db which indexing mechanism it should use. but i agree with main 'general tools are ~meh' sentiment)
Framedragger: ftr i didn't every say it was the right thing for bitcoin with uniformly distributed hashspace
Framedragger: ah, you're probably right. lots of fs-specific cache madness, too
Framedragger: (and yah i'm sure there's lots of crud in the former worth dropping..)
Framedragger: kernel driver would work around userland fs specific stuff. but certainly not against disk seek, disk cache, etc. just to clarify
Framedragger: mircea_popescu: (i thought with "sounds great on paper" you were responding to possible kernel module workarounds. but if you're against the *whole* idea, fair enough.)
Framedragger hears wind swoosh as goalposts shifted at c velocity
Framedragger: but yeh, maybe more exposure to userspace shite if not driver - maybe
Framedragger: if you have shitty code, you have shitty code. shitty code in kernel wreaks more havoc.
Framedragger: (or, use linux cap to allow raw access, but that's maybe meh)
Framedragger: not that it's necessarily the way to go. but consider what asciilifeform was saying - one could just pass an already-opened file handle. handle to *whatever*. no ring0 driver, no root permissions.
Framedragger: mircea_popescu: but if you read further down, we were saying that it may be possible to just access a raw block device without kernel module :)
Framedragger: well, *in principle* deeper traversal should really not be a problem at all. in practice - guess we shall see
Framedragger: kk as long as we're clear - i've seen those "but what would happen with 1000s of symlinks in single folder!1" comments so thought that should be tested, etoo.
Framedragger: but on the other hand this'd benchmark/test directory traversal more.
Framedragger: so here's the thing, if folder depth is increased, the actual numbers of symlinks per directory will be basically ~0, as per those graphs from earlier.
Framedragger: mod6: thanks - was supposed to be busy with other stuff but this cute insane asylum is concerningly quite attracting :)
Framedragger: apparently calling fclose() while syscall is running on that descriptor in another thread is 'not a good idea' (outcome maybe platform/implementation specific), but that's an avoidable corner case.
Framedragger: that said, i'm pessimistic about this whole ext4 thing just like asciilifeform. no reason not to direct time into his latest proposal
Framedragger: oh wait, i was wrong re thread-safe functions. kernel takes care of that. has internal locking mechanism when dealing with fread/fwrite etc.
Framedragger: in terms of thread-safety, yes (but not in benchmarking tool as it's not using thread-safe posix functions, but then it's not writing anything, either). however, parallel performance not at all certain.
Framedragger: (anyway, will share tool i wrote if only because couldn't find anything fitting the task out there, but need to polish a bit first, etc.)
Framedragger: given time, i was also meaning to run lots of strace to see what is in actuality happening with multiple symlinks etc, but maybe sisyphus
Framedragger: no disagreement tbh, i probably know your reasoning
Framedragger: asciilifeform: you're right, i was meaning to reset caches (`echo 3 >/proc/sys/vm/drop_caches` should be enough i think), but then forgot
Framedragger: (ftr: sata 3gbps, ext4, intel SSDSA2CW30, mirror raid.)
Framedragger: (5ms read on an ssd ain't too pretty, but would have to check distribution of these kinds of tails, i guess.)
Framedragger: (fread() was 4k. files are short, with just the number of of 'block' for cross-confirmation and list of 'transactions' as contents.)
Framedragger: fclose() times given 100000000 iterations: min = 645 ns, max = 739202 ns, avg = 741.62 ns, stddev = 1647.59 ns, sum = 74161953610.0 ns, minloc @ 19564589, maxloc @ 67051179
Framedragger: fread() times given 100000000 iterations: min = 1189 ns, max = 53879016 ns, avg = 1552.73 ns, stddev = 6917.04 ns, sum = 155273497933.0 ns, minloc @ 2462746, maxloc @ 126806
Framedragger: fopen() times given 100000000 iterations: min = 1761 ns, max = 1506074 ns, avg = 2326.51 ns, stddev = 3003.91 ns, sum = 232650578514.0 ns, minloc @ 72539767, maxloc @ 19225446
Framedragger: Symlink resolve (readlink()) times given 100000000 iterations: min = 949 ns, max = 1353644 ns, avg = 1688.99 ns, stddev = 2502.42 ns, sum = 168899426822.0 ns, minloc @ 41772957, maxloc @ 20095033
Framedragger: Uniform randomness report: min hits seen = 871, max hits seen = 1160, avg = 1000.00, sum = 100000000.0, minloc @ 85798, maxloc @ 88837
Framedragger: Processed 1000000 lines (min/max target numbers seen: 1, 100000) - beginning benchmarks now... [i.e., 'target block' numbers are in this range. 1m symlinks point to 100k files randomly.]
Framedragger: some (very initial) symlink stats - more stuff will have to wait - given a "here are 1mn 'transactions' which symlink to files, resolve links and read from linked files in random order for 100mn times" setup, with one-folder-deep structure, like so: "simple_f1/e5/e5edc34c57d5ea2ea99cfe16d04655aa000c3d7f268022d2b21f95928fa34674 -> /files_f1/99997.txt", most basic stats are:
☟︎ Framedragger: yeah, i see what you mean, and everything. i'm not convinced that phuctor db is using the most it can from postgres, but neither you nor me have time to investigate this presently, i guess (and i may not be able to do a great job there anyway)
Framedragger: of course, i agree, i'm just saying it's not "inherent problem of all db omg"
Framedragger: tree rebalance is a feature of a balance binary tree which is sometimes the right tool for the job, cmon :)
Framedragger: (i guess depends on nature of parallel work. e.g. if it's some kind of sequential re-confirm, launch threads with different segments, etc.)
Framedragger: yes, but how to cheaply ensure the latter. you won't have a shitload of locks now will you
Framedragger: ah yeah, iirc some applications even use this 'descriptor passing' technique so minimise high permissions level exposure (but don't recall). quite robust
Framedragger: well, 'disk' could just be partition which isn't a bad thing anyway, right?
Framedragger: also, wonder if there could be a relevant linux cap'ability for allowing raw access to given /dev/block. but maybe not.
Framedragger: asciilifeform: yeah! i may have time / want to do this in ~april. just to be clear, there's really no way to have underlying userland fs allocate a contiguous block, right?
Framedragger: no, of course, you just mentioned possible cheats, but you probably meant something more 'robust'.
Framedragger: that's a horrible idea for sure, but hey, free space to cheat in
Framedragger: heh you can even use things like nx bit if not wanting to reorganize table, i would guess!
Framedragger: aha, so there's high probability that there will *be* a collision across the entire space.
Framedragger: hard to believe, but then, i should read logs at least
☟︎ Framedragger: attempted.. citing internal intelligence of course, or something
Framedragger: and yeah, this 'generate collisions' scenario seems like a harder version of 'just mine'
Framedragger: i'm guessing that an interested state could totally do that
Framedragger: while at it, and i'm guessing it's in the logs.., i wonder when was the last time some tried to come up with an approximation what the costs to get to >50% network hashing power would be
☟︎ Framedragger: yeah. hmh - i guess that's it. literally. nice scheme