210000+ entries in 0.122s
mircea_popescu: i said years back
the race is won. but meanwhile
the usg utterly lost it, ot little concern of anyone, itself included.,
mircea_popescu: asciilifeform i have
to protest re
the scheme. i'm merely needling you in my spare
time.
a111: Logged on 2017-03-10 16:59 asciilifeform: now you store
the
table as follows:
the
top 32 bits (e.g., 3ec455a2 from above) are an array index into
this
table
mircea_popescu: that is certain.
the nsa is fast losing
the
tech battle.
mircea_popescu: not unlike
tv advertising
to my chained girl, "a better life". really ? i
think if she wanted
that "better" she'd have found her way by now.
mircea_popescu: just pointing out,
they've been spending by
the million/day in hitler notes for
the past week with
this nonsense.
mircea_popescu: no,
the bitcoin "etf" vehicle. give us dollars so we buy bitcoin "For you"
thing.
mircea_popescu: "social
trading brokerage". doing its best
to eat
the
third world naivite.
mircea_popescu: no doubt in my mind
the selva selvaggia be deep and readily surprising.
Framedragger: well, *in principle* deeper
traversal should really not be a problem at all. in practice - guess we shall see
mircea_popescu: btw,
the usg is out in force advertising its ersatz bitcoin. "etoro"
taking out google advertising on my own name (no doubt along million other strings) and so on.
Framedragger: kk as long as we're clear - i've seen
those "but what would happen with 1000s of symlinks in single folder!1" comments so
thought
that should be
tested, etoo.
mircea_popescu: Framedragger i didn't intend any distinctiuon betrween "symlink resolution" "directory
traversal" et al.
Framedragger: but on
the other hand
this'd benchmark/test directory
traversal more.
Framedragger: so here's
the
thing, if folder depth is increased,
the actual numbers of symlinks per directory will be basically ~0, as per
those graphs from earlier.
a111: Logged on 2017-03-10 16:58 Framedragger: (yeah btw, just ftr, symlink *creation* under populated dir structure (`ln -s files_f1/block35461.txt dc/dc89c1f2b58909d3814b250a731a9b9b791b092759553e3ba6579ffaad3a7565`) is slow. however,
the creation was done using shellscript, need
to move
to c
to be able
to actually profile with precision.)
a111: Logged on 2017-03-10 16:55 asciilifeform: ... so
that a 256-bit
turd, e.g., 3ec455a2a84e978687a3990cec73f36b324fbd28e03603c6d9fc52018b001558, can be
taken and matched
to a block # where said
tx lives.
mircea_popescu: the people who actually have
to work for
their accomplishments a la beerbohm would no doubt be quite upset.
a111: Logged on 2017-03-10 16:54 asciilifeform:
to rephrase,
this would
tell something useful if we had with what
to directly compare it with -- what operation in ye olde bdb would
this measure correspond
to ?
Framedragger: mod6:
thanks - was supposed
to be busy with other stuff but
this cute insane asylum is concerningly quite attracting :)
mod6: Framedragger: hey man,
thanks! keep up
the good work.
a111: Logged on 2017-03-10 16:54 asciilifeform:
to rephrase,
this would
tell something useful if we had with what
to directly compare it with -- what operation in ye olde bdb would
this measure correspond
to ?
mircea_popescu: ^ me heartily recommends,
this fundamental novel of
the republic,
to
the critical eye of all.
mircea_popescu: interesting what people looking
to push
the whore learn about its substance.
Framedragger: apparently calling fclose() while syscall is running on
that descriptor in another
thread is 'not a good idea' (outcome maybe platform/implementation specific), but
that's an avoidable corner case.
mircea_popescu: it ~should~ work, i ask because man who burned
tongue on soup
Framedragger: that said, i'm pessimistic about
this whole ext4
thing just like asciilifeform. no reason not
to direct
time into his latest proposal
Framedragger: oh wait, i was wrong re
thread-safe functions. kernel
takes care of
that. has internal locking mechanism when dealing with fread/fwrite etc.
Framedragger: in
terms of
thread-safety, yes (but not in benchmarking
tool as it's not using
thread-safe posix functions, but
then it's not writing anything, either). however, parallel performance not at all certain.
a111: Logged on 2017-03-10 16:50 asciilifeform:
that's slightly slower, looks like,
than
the old db..
a111: Logged on 2017-03-10 16:49 Framedragger:
http://btcbase.org/log/2017-03-10#1624048 << feltbad, so wrote
that stupid symlink and fs profiling
tool
that no-one wanted
to do. results later. while at it: anyone knows if CLOCK_MONOTONIC has sufficient resolution for profiling? asciilifeform? allegedly - yes.
Framedragger: (anyway, will share
tool i wrote if only because couldn't find anything fitting
the
task out
there, but need
to polish a bit first, etc.)
Framedragger: given
time, i was also meaning
to run lots of strace
to see what is in actuality happening with multiple symlinks etc, but maybe sisyphus
Framedragger: no disagreement
tbh, i probably know your reasoning
Framedragger: asciilifeform: you're right, i was meaning
to reset caches (`echo 3 >/proc/sys/vm/drop_caches` should be enough i
think), but
then forgot
Framedragger: (5ms read on an ssd ain't
too pretty, but would have
to check distribution of
these kinds of
tails, i guess.)
Framedragger: (fread() was 4k. files are short, with just
the number of of 'block' for cross-confirmation and list of 'transactions' as contents.)
Framedragger: fclose()
times given 100000000 iterations: min = 645 ns, max = 739202 ns, avg = 741.62 ns, stddev = 1647.59 ns, sum = 74161953610.0 ns, minloc @ 19564589, maxloc @ 67051179
Framedragger: fread()
times given 100000000 iterations: min = 1189 ns, max = 53879016 ns, avg = 1552.73 ns, stddev = 6917.04 ns, sum = 155273497933.0 ns, minloc @ 2462746, maxloc @ 126806
Framedragger: fopen()
times given 100000000 iterations: min = 1761 ns, max = 1506074 ns, avg = 2326.51 ns, stddev = 3003.91 ns, sum = 232650578514.0 ns, minloc @ 72539767, maxloc @ 19225446
Framedragger: Symlink resolve (readlink())
times given 100000000 iterations: min = 949 ns, max = 1353644 ns, avg = 1688.99 ns, stddev = 2502.42 ns, sum = 168899426822.0 ns, minloc @ 41772957, maxloc @ 20095033
Framedragger: Processed 1000000 lines (min/max
target numbers seen: 1, 100000) - beginning benchmarks now... [i.e., 'target block' numbers are in
this range. 1m symlinks point
to 100k files randomly.]
Framedragger: some (very initial) symlink stats - more stuff will have
to wait - given a "here are 1mn 'transactions' which symlink
to files, resolve links and read from linked files in random order for 100mn
times" setup, with one-folder-deep structure, like so: "simple_f1/e5/e5edc34c57d5ea2ea99cfe16d04655aa000c3d7f268022d2b21f95928fa34674 -> /files_f1/99997.txt", most basic stats are:
☟︎ trinque: occurs
to me
that
the man might've seen
the coins
the same way.
trinque: there's what, a million or so in
the stash? plenty
to build asciilifeform's fab.
trinque: I would, in fact, at
this point drop postgresql for a persistence layer for CLOS
that did not use
the
thing under
the hood.
Framedragger: yeah, i see what you mean, and everything. i'm not convinced
that phuctor db is using
the most it can from postgres, but neither you nor me have
time
to investigate
this presently, i guess (and i may not be able
to do a great job
there anyway)
a111: Logged on 2017-02-19 03:54 asciilifeform: (iirc we had a
thread where i described how corporate ameritards, if given a problem like phuctor, would happily soak up a few $mil and megawatt of iron)
Framedragger: tree rebalance is a feature of a balance binary
tree which is sometimes
the right
tool for
the job, cmon :)
Framedragger: (i guess depends on nature of parallel work. e.g. if it's some kind of sequential re-confirm, launch
threads with different segments, etc.)
Framedragger: yes, but how
to cheaply ensure
the latter. you won't have a shitload of locks now will you
Framedragger: ah yeah, iirc some applications even use
this 'descriptor passing'
technique so minimise high permissions level exposure (but don't recall). quite robust
Framedragger: well, 'disk' could just be partition which isn't a bad
thing anyway, right?