210400+ entries in 0.135s
Framedragger: re. fs nodes, couldn't sleep + not sure if
this makes sense, so just
throwing
these out - barebones super simplistic (function is `n_objects_to_store ^ 1 / folder_depth`) plots showing expected average number of nodes per folder (assumptions are no bias in hashspace and also equal share of hash bits per folder level) - it may not be intuitive how low
the averages are until you look:
jhvh1: asciilifeform:
The operation succeeded.
Framedragger: assuming equally distributed
transaction hashspace, if you want your
tree
to fill up with 1000 nodes on average per given depth, you'd be storing 10^24
transactions. but
this assumes
that every folder depth gets assigned equal number of bits
to represent, of course.
Framedragger: for symlink fs
testers (or maybe selfnote for later): note
that if you allow for sufficient folder
tree depth,
the "1000s of symlinks per dir" won't realistically happen when storing, say, bitcoin
transaction hashes.
the latter have 256 bits => 64 hex chars. if you allow for depth of 8 where last level (8) is symlink itself, you get 32 bits per folder level.
mircea_popescu: the quality of service in
this restaurant is unparalleled. but yes, vupen.
mircea_popescu: this is like
the superlative of "what do i do now ?" "drink."
ben_vulpes: (not
that
the barf in question makes any goddamn sense in
the first place...)
ben_vulpes: well how am i supposed
to advise without being able
to repro?
phf: ben_vulpes: inlining svg works, but now it
throws errors about javascript. please advise.
ben_vulpes: hey
trinque didja hear
that one about cups and
the symlinks?
mircea_popescu: asciilifeform well,
tradeoffs. he wants
to store
the block in parts, he wants
to store it whole, what can i say
a111: Logged on 2017-03-09 20:54 mircea_popescu: nothing generally forces you
to keep a block other
than a collection of
transaction-files, for instance.
the exact implementation is up in
the air for exactly such reason\
mircea_popescu: Framedragger yes,
the fs is a major pile of dubious, as asciilifeform well points out.
diana_coman: needs
to reimplement
to get out on
the other side really (I did NOT dig deep into Blender but I wouldn't be surprised if it were
terribly bloated at
the very least)
a111: Logged on 2017-03-09 18:14 mircea_popescu: meanwhile i ended up with a python dependency via blender ;/ i dun
think it's going anywhere.
diana_coman:
http://btcbase.org/log/2017-03-09#1623846 <-- fwiw
the only way I can see
this dependency on Python going anywhere would be if someone makes a sane replacement basically - however, artists need it but won't do it and otherwise people who are able
to do it have a huge list of *other*
things
that need
to be done as far as I can see; moreover (and as usual already),
the whole steaming pile is deep so I can't even say how much one
☝︎ Framedragger: also, """But I appear
to have a lingering effect
that seems
to have started from
the
time my /tmp directory had
the millions of files in it.
mircea_popescu: nothing generally forces you
to keep a block other
than a collection of
transaction-files, for instance.
the exact implementation is up in
the air for exactly such reason\
☟︎ Framedragger: yeah, okay; as long as it's not fixed-width
trb-i, no way around
this.
mircea_popescu: and if fixed widths, you would actually "use" offsets in
the sense of making files.
mircea_popescu: you could use
them
to connect
transactions
to blocks, addresses
to blocks, whatever index you're
trying
to keep
Framedragger: would
this be performant enough even
theoretically, given no way
to use offsets?
a111: Logged on 2016-12-22 06:41 mircea_popescu:
this way you don't actually have
to ~index~ anything, if you wish
to see where
txn 1234567890 was included in a block, you go
to /12/34/56/7890 which points
to block x
Framedragger: (but maybe you covered
that,
too, and i forgot in logs.)
Framedragger: but with
transactions, you can be sure
that once it returns, it will have written
to disk. fsync can still be on. (iirc).
Framedragger: (and if you now say 'db is lost cause anyway' while not linking
to code/config *again*, i'll grit
teeth angrily)
shinohai: Actually lemme check one more
thing before I do ......
Framedragger: but i do hope you're doing
the former, i mean i assumed so.
that's
the lowest-hanging fruit re. 'how do i do batch writes
to db'
Framedragger: asciilifeform: on
top of 'transactions', postgres has 'checkpoint' parameter. but you probably won't like it because of
the whole 'not
turning off fsync'
thing
mircea_popescu: or who knows how
the fuck else -- rewriting
the committer, even.
mircea_popescu: not as such, no. but you can wedge it in
through, eg, making it commit in batches
mircea_popescu: how large a part is an open question, but it obviously comes after establishing whether
the would-be airplane is even larger
than a breadbox in
the first place
shinohai: asciilifeform: still getting bad signature when
trying
to verify your patchset ... (imported your key from deedbot)
mircea_popescu: a large part of
the "not writing properly" can actually be fixed via configs for
the fs, rather
than using
the "middle of
the road" stuff shipped with eg ubuntu.
a111: Logged on 2017-02-26 19:27 mircea_popescu:
the other problem is
that a good db fix is a very large project, because bitcoin is written insanely. and our fs db isn't moving, last i heard a month ago someone was going
to
try and profile an extx
mircea_popescu: linking would be better
than
this. having digested what was said better still.
mircea_popescu: you repeat halfway of
the discussion, "here is what i said lalala i can't hear anyone"
Framedragger: instead, it's "just" a matter of having a however-deep directory
tree with symlinks as
the leaves.
mircea_popescu: ah. no,
that is correctly within
the limit, as designed we'd have under 8 anyway
Framedragger: for a minute i
thought (don't know why)
that what is *additionally* needed is
the capability
to have paths of /symlinks/to/symlinks/.
Framedragger: oh wait, i phrased
this incorrectly while at
the same
time horribly mis-reading: sorry,
this is about max depth of path composed of symlinks.
mircea_popescu: something's amiss, im sure i have directories with more
than 40 symlionks
a111: Logged on 2016-12-22 06:41 mircea_popescu:
this way you don't actually have
to ~index~ anything, if you wish
to see where
txn 1234567890 was included in a block, you go
to /12/34/56/7890 which points
to block x
phf: (if nobody else steps up, i'm going
to bring one up in a day or
two)
☟︎ ben_vulpes: relatedly, from #elsewhere, "dude shows up with a
thousand farmers on camels. what,
tell him
to fuck off?"
ben_vulpes: onlooker: close
the
tab and read
the logs proper
mircea_popescu: it was
the beginning of
the end of
the british colonial empire. it crashed, where empires come
to day, in
that god forsaken asshole of
the world where fork-bearing regiments are cut
to shreds and don't run off on contact. or at all.
mircea_popescu: they had
to burn
the crops, kill
the cows,
tear villages stone from stone. losses -- immense, on both sides.
mircea_popescu: at some point lytter i
think it was decided
the afghanis are befallen under russian spell and invaded.
mircea_popescu: well,
the "great race" aha
this run
to
the pacific by russia and
the uk settled down eventually with
the brits holding india and
the russians holding siberia. afghanistan-buffer state.
mircea_popescu: anyone familiar with
the story of
the war in afghanistan ?
ben_vulpes: 'some regiments may be cut
to ribbons and run on contact, but
that does not make
them useless'