log
▁⏐
phf: there's no "escape" as such. instead it generates an adhoc escape (say replace dot with ZZ or whatever) and then patches that one line using s///.
asciilifeform: phf: 'ed-style' diff outputs are the Right Thing, but done the ~proper~ way, with NO INBAND MAGIC, and not the monkey way.
asciilifeform: ick.
phf: well, that's not inband magic! still "ick"
asciilifeform: it's magic
asciilifeform: if there's a forbidden char or string --- that is called magic.
asciilifeform: the alternative, the correct one, is 'next N bytes are payload'.
asciilifeform: and THEY CAN BE ANY OCTET
asciilifeform: the '.' operator in 'diff -e' is the magic.
phf: it's \n.\n
asciilifeform: one can debate whether the persians are right to cut hands off thieves. but the hands of folx who write programs like this, i cannot see any reason why they should stay attached.
asciilifeform: phf: USES nonprintables in the magic ?! even deeper retardation.
asciilifeform 'can't even', takes break, off to play with 10kg joystick, and then with pet
phf: newline is not a non-printable
phf: ffs, i resent being placed in a position of defending something that i'm not responsible nor care for. diff -e would've been closer to "teco macros", but it's the "sane teco macros" we're talking about here, etc. etc. etc.
mircea_popescu: in other news, obama's been doing more reality tv work than the entire kardashian extended family these past few days.
mircea_popescu: it's the irony of all time that while the progre press was derping about how trump will lose the election and try to turn it around into a television show, the reality turned out to be he won the election AND OBAMA is trying to turn his losership into a tv personality
ben_vulpes: eh, obab's just wants to grab some pussy
ben_vulpes: s/'s//
mircea_popescu: like the golfer guy what's his name
ben_vulpes: woods?
mircea_popescu: asciilifeform no the classification proposed is quite serviceable.
mircea_popescu: aha yeah woods
mircea_popescu: !!key mlcmolina
deedbot: http://wot.deedbot.org/8450B3D7AD8AE9D9FB2A0FCBCB5A7CD47D66FE7B.asc
mircea_popescu: uh why do i get a 404
mircea_popescu: trinque ^ ?
ben_vulpes: trinque ^^
ben_vulpes: C-k, not C-j, kid...
mircea_popescu: wut ?
mircea_popescu: phf i guess we will sooner or later have to actually formulate the patch format yeah.
mircea_popescu: re octopus - no other effect individually ; but living long is bad for the species, as hillary well exemplifies.
mircea_popescu: http://akphoto2.ask.fm/7a8/29f1b/e255/41d8/a375/1905a754e8d6/original/559106.jpg << here's something for phf. cheer up!
asciilifeform: http://btcbase.org/log/2016-12-30#1592962 << news to me☝︎
a111: Logged on 2016-12-30 00:04 phf: newline is not a non-printable
trinque: mircea_popescu: it'll be up there in a sec, let ya know
mircea_popescu: kk
trinque: gotta finish writing the thing that triggers granular (per-nick) updates, leaning on a full rebuild for now.
trinque: ben_vulpes and I had an interesting conversation yesterday about how to handle static sites generated from a db. idea we ended up with was that we'd have triggers which emit a pg_notify signal when the "dirty bit" has flipped for any page.
trinque: db operator has to go through and manually apply these triggers. for example, updates on either a "to" or "from" rating will require a rebuild of the nick pages in both directions.
ben_vulpes: i am very much a fan of this design
asciilifeform: trinque: why would you generate the site from a db? more than once, i mean
ben_vulpes: please to never have code near an httptron
asciilifeform: why have 2 representations of same thing on disk ?
trinque: there's a listening process consuming these pg_notify events, of the form (gen-nick-page "trinque") which debounces them according to some sane interval, i.e. if a few updates hit the same user in a short span, it will result in one single rebuild
trinque: asciilifeform: because people update their ratings
asciilifeform: so update the html?
trinque: ben_vulpes: and yes, app code needn't be anywhere near www
trinque: asciilifeform: that is precisely what I just said
ben_vulpes: asciilifeform: exactly :D
asciilifeform: yes but why have the sql db then
trinque: wtf
asciilifeform: at all
asciilifeform: anywhere
trinque: this model has nothing to do with sql
trinque: and everything to do with shitty html flowing from properly structured data
asciilifeform: the mention of pg_notify
ben_vulpes: trinque: just sed and awk the html you have in place to update ratings
trinque: oh ok
asciilifeform: hence i ask 'why have 2'
trinque: I am not using html as a data storage format what the hell
trinque: lol
ben_vulpes: no it's fine
ben_vulpes: deduplication
ben_vulpes: parsing html isn't even hard
trinque: asciilifeform: the question answered here is "why regenerate the same idiot HTML every www request?"
trinque: and the answer is don't
asciilifeform: i've seriously considered reimplementing phuctor in this shape. as it is, it loses more from the slow writes idiotically queuing up, and the wedged reads that result, than it wins from fast structured queries.
ben_vulpes: "this shape" being selective html regeneration? or mutation in place?
trinque: wot.deedbot.org will likely result in two genesis patches down the line. one for the tool, other for the particular site
trinque: if asciilifeform should ever want it
asciilifeform: ~displaying a www page~ oughta be an O(1) op, really
asciilifeform: in ~all~ cases.
ben_vulpes: mhm.
asciilifeform: trinque: mircea_popescu will probably barf if the 2 genesisen get displayed on same screen. it isn't an eggog in vtronics, but is in his head.
asciilifeform: ben_vulpes: 'this shape' being 'html as-seen-by-reader as the only storage format'
asciilifeform: i've developed a loathing, inexpressible in words, for postgres and all things like it
trinque: I am not moving from it until I have another database in hand
asciilifeform: why the FUCK should a READ block because ~wholly unrelated datum is being written~
asciilifeform: and why the FUCK is 10,000 writes/sec 'too much'
asciilifeform: is it 1971? am i on a drum disk?
trinque: yeah yeah. /dev/null is even faster
trinque: but the right answer is as much or as little structure enforcement as you like.
asciilifeform: my fs can eat GB/s r/w without breaking a sweat
trinque: this is not the direction from which to come at this subject
trinque: conflated in "database" is both a user interface and a programming environment
asciilifeform: elaborate?
trinque: as the shell is a user interface, in the first case
trinque: recall I conceded in the bitcoinfs thread that what I consider to be "database" is actually many orthogonal tools related to data manipulation, atomicity of operations, adherence to type constraints ~if desired~
trinque: ^ programming environment
asciilifeform: all of the orthogonal tools must be mercilessly cut apart, yes.
asciilifeform: but so must orthogonal ~operations~. under nk circumstances is blocking on UNRELATED op, tolerablr
asciilifeform: tolerable
asciilifeform: it is monumentally retarded, as a mere possibility in a system
trinque: that sounds like a bug, really, but the kind of bug the complexity of the thing makes hard to remove
asciilifeform: imagine your cock didn't work if 10,000 others in town were in use.
mircea_popescu: trinque i think that's how pretty much everyone ends up doing things.
trinque: mircea_popescu: ah you know what, excluding nicks with no ratings breaks this, because writing the keyfile happens inside that logic.
trinque: I'll instead write the page and exclude from the index
trinque: noobs can aspire to be named there
asciilifeform: http://btcbase.org/log/2016-11-19#1570951 << see also☝︎
a111: Logged on 2016-11-19 18:52 asciilifeform: Framedragger: db being hammered 24/7 with 'do we have this hash' 'do we have this fp' 'add this and this' 1000/sec is the bottle.
mircea_popescu: asciilifeform there would be two trees not "two genesises" in his idea, he's making a tree for the site and a tree for the tooling.
trinque: aha
asciilifeform: aite
mircea_popescu: and gtfo with the inept simile, the correct comparison is "imagine if your cock didn't work if there was already one in the target woman". which i bet yours doesn't, so really.
trinque: asciilifeform: we poor bastards who made money living inside SQL find ways to make it work.
trinque: but if not sooner, I have orthogonal organs of database given independent life as a retirement project
mircea_popescu: trinque ok so can i get this guy's key he's kinda weaiting for an acct.
asciilifeform: trinque: my understanding is that these ways typically involve clusters of machines, duplicate db, and very elaborate/failure-prone synchronizers
trinque: mircea_popescu: yep works now
asciilifeform: mircea_popescu: no, it is specifically 'doesn't work because ~nearly there is a cock-shaped shadow~
trinque: !!key mlcmolina
deedbot: http://wot.deedbot.org/8450B3D7AD8AE9D9FB2A0FCBCB5A7CD47D66FE7B.asc
asciilifeform: *nearby
mircea_popescu: ty
trinque: and I'm tweaking the site generator
mircea_popescu: aaa+++ would use again.
trinque: asciilifeform: aside "write a data manipulation environment that provides what trinque demands" I do not see a path here.
asciilifeform: how much of 'what trinque wants here' is unobtainable by simply abusing the fs ?
trinque: when I build things to ask the fs how many customers in new york placed orders on the weekend it starts to look quite like a relational db
asciilifeform: yeah but one that doesn't motherfucking grind to a halt when read 1000/sec omfg
trinque: could be. I'd like to see the design for such a thing.
mircea_popescu: we still didn't see the profiling for the symlink thing\
mircea_popescu: i'd expect it'd be informative.
asciilifeform: picture reiserfs , but without the idiot fortranistic hard limits on nodes, lengths, etc
mircea_popescu: if it starts with picture it can't be a part of this discussion.
asciilifeform: then take standard reiser. but in that case you grunt under entirely arbitrary procrusting .
phf: picture a spherical horse in vacuum (tm) (c)
deedbot: http://trilema.com/2016/disgrace-never-mind-note-that-we/ << Trilema - Disgrace - Never mind. Note that we
phf: asciilifeform: i will have to concede, (graphic-char-p #\newline) NIL
asciilifeform: if drum printer dun go 'bang' when it gets the octet - it ain't 'printable.'
asciilifeform: even if it ~does~ go whrrrrr.
phf: well, (graphic-char-p #\space) T
asciilifeform: ullage!1111
mircea_popescu: lmao
mircea_popescu: phf for my own curiosity, does anything in the rest of your life piss you off as much in aggregate as one session with these assholes ?
phf: hehe, as much? no. but that's only because if there's one lesson i learned from naggum, all this rage is not healthy.
mircea_popescu: tru.
phf: also i learned long time ago that americans* aren't taught how to argue properly, so when they do they have a really hard time keeping the thread, keeping more than one point, developing an argument, bringing it back to original point, etc. consensus intelligentsia are all very civil, so when you do get them railed up, they just flail, sort of discourages from even trying
mircea_popescu: hm.
deedbot: http://phuctor.nosuchlabs.com/gpgkey/18B7C700E3D732F13B498BE6777E8C50040F0BE90CC086A78993AEB30CCB803B << Recent Phuctorings. - Phuctored: 3209...6799 divides RSA Moduli belonging to '41.178.64.157 (ssh-rsa key from 41.178.64.157 (13-14 June 2016 extraction) for Phuctor import. Ask asciilifeform or framedragger on Freenode, or email fd at mkj dot lt) <ssh...lt>; ' (Unknown EG)
deedbot: http://phuctor.nosuchlabs.com/gpgkey/05654F6204A0FB615199D4DB4E442FFFCA74621F46A18B5E74CB978504E480F8 << Recent Phuctorings. - Phuctored: 3209...6799 divides RSA Moduli belonging to '41.178.64.148 (ssh-rsa key from 41.178.64.148 (13-14 June 2016 extraction) for Phuctor import. Ask asciilifeform or framedragger on Freenode, or email fd at mkj dot lt) <ssh...lt>; ' (Unknown EG)
deedbot: http://phuctor.nosuchlabs.com/gpgkey/763930076E165275AF7659105A2A31F61AE3FBF931A7E188B2DC146EFFC7CF49 << Recent Phuctorings. - Phuctored: 3209...6799 divides RSA Moduli belonging to '41.178.64.147 (ssh-rsa key from 41.178.64.147 (13-14 June 2016 extraction) for Phuctor import. Ask asciilifeform or framedragger on Freenode, or email fd at mkj dot lt) <ssh...lt>; ' (Unknown EG)
deedbot: http://phuctor.nosuchlabs.com/gpgkey/7E840C2FC34B3F9B8B95E6D12E18267E717ACBE1494C34DFA9055710347573A7 << Recent Phuctorings. - Phuctored: 3209...6799 divides RSA Moduli belonging to '116.105.59.120 (ssh-rsa key from 116.105.59.120 (13-14 June 2016 extraction) for Phuctor import. Ask asciilifeform or framedragger on Freenode, or email fd at mkj dot lt) <ssh...lt>; ' (Unknown VN 64)
deedbot: http://trilema.com/2016/disgrace-it-is-raining/ << Trilema - Disgrace - It is raining.
trinque: lol, but then russians take the thread wherever they like without telling anyone.
trinque: phf: when I brought to you "whence the disjunction between the practical and theoretical sides of subj" your output was "OP == faggot"
trinque: lets not speak of flailing
deedbot: http://trilema.com/2016/disgrace-you-say-you-have-not/ << Trilema - Disgrace - You say you have not
ben_vulpes: http://btcbase.org/log/2016-12-29#1592918 << ftr i hate the solution for this that i have on disk☝︎
a111: Logged on 2016-12-29 23:46 phf: http://btcbase.org/log/2016-12-29#1592904 << i just pasted diff so that i didn't have to do two lines :} corresponding +++ --- lines are
ben_vulpes: a) capture output of `patch' to determine which files were patched
ben_vulpes: b) when working through the list of each patch's children, search through the list of patched files until the patched filepath is a subsequence of the filename as recorded in the vpatch
ben_vulpes: c) hash that file and compare to the vpatch contents
ben_vulpes: the only other thing that i can think to do here is to grab the set of parents and children, match them up, and then get the lowest common denominator (if you will) file path for each patched file directly from the vpatch
ben_vulpes: standby, let me test a third approach
ben_vulpes: the third approach is to apply each patch to an empty directory and determine what files would have been patched, had it applied cleanly.
ben_vulpes: but i hate 3 for obvious reasons
ben_vulpes: http://p.bvulpes.com/pastes/FnxeR/?raw=true
phf: ben_vulpes: well, we're kind of constrained by the hardcode -p1 behavior, but i've no idea if that's an implementation detail or a spec
phf: of v that is
ben_vulpes: i've been frowning at -p1 for a bit now
phf: so the simplest solution would be to at least parametrize an equivalent of -p1 on lisp side
mircea_popescu: what is -p1 again ?
ben_vulpes: how many prefixes to strip
phf: -p1 means a/foo/bar gets pressed as foo/bar
phf: with p2 same guy gets pressed as bar
deedbot: http://trilema.com/2016/disgrace-at-first-they-do-not/ << Trilema - Disgrace - At first they do not
ben_vulpes: phf: patch will apply something cleanly with mismatched depths, won't it?
trinque: patch will ignore the number of levels you specified with -p
trinque: of path
phf: i think it treats one of the names as canonical
ben_vulpes: http://logs.bvulpes.com/trilema-mod6?d=2016-12-29#3 << thread in #trilema-mod6
phf: actually patch seems to do ... something magical
ben_vulpes: RIGHT?!
ben_vulpes: would a smallest common substring test suffice here?
phf: that's what btcbase does basically. it finds position where common suffix starts and then works from there..
phf: but i'm starting to think it's an overkill anyway, because it doesn't accommodate for all possible insane patch inputs.
ben_vulpes: what's an insane input that breaks the shortest common substring test?
phf: patch/diff lets you have a patch with --- foo +++ bar in which case it seems to ~check if foo exists, then try and press against foo, otherwise press against bar~
phf: so if you were to produce a patch with a/old-veh.lisp and b/veh.lisp. existing vtrons will happily press it, though it's a total clusterfuck
ben_vulpes: gross.kpeg
phf: actually that's a bad example because that'll work, but a/old.lisp and b/veh.lisp
mod6: anyway, yeah, as I said in #trilema-mod6, i see this as low-priority and SUPER high risk. but I'm open to suggestions how to implement this properly and safely.
ben_vulpes: phf: i actually can't get patch c/old.lisp and d/veh.lisp to apply derp.vpatch
ben_vulpes: standby 1
ben_vulpes: http://p.bvulpes.com/pastes/O687q/?raw=true
phf: ben_vulpes: where'd that 0 come from?
ben_vulpes: i have nfi
phf: also you're missing genesis
mod6: i think you maybe mean '-F' instead of '-f', it thinks 0 is a file
ben_vulpes: COOL
phf: http://wotpaste.cascadianhacker.com/pastes/udP9e/?raw=true
ben_vulpes: http://p.bvulpes.com/pastes/FF1IA/?raw=true
ben_vulpes: 1 thing of note is that your example patches an extant file in holyfuq/
ben_vulpes: would it be unreasonable to calve this scenario off?
ben_vulpes: oh
ben_vulpes: hm
ben_vulpes: okay i see
ben_vulpes: good to know that terminal prompt of yours survived the trip through the pastebin
phf: also the old school way of making a genesis http://p.bvulpes.com/pastes/JQxyU/?raw=true
ben_vulpes: what is `p'?
ben_vulpes: "tee p" just papers your house with the output or what
ben_vulpes: i would also like to see the vdiff into which you're piping diff, mostly out of curiosity
phf: tee writes piped input to stdout and to a file
ben_vulpes: ah <p gotcha
ben_vulpes: eager juvenile algos are eager and juvenile
phf: vdiff here http://p.bvulpes.com/pastes/r01nj/?raw=true it's the same old vdiff except if you pipe into it, it assumes you're piping in a patch, otherwise it acts as normal vdiff
deedbot: http://phuctor.nosuchlabs.com/gpgkey/D2FDFCE450D0A058B385E0E94E0E57E611A3EBF385B30F77CF915C96BDE19B97 << Recent Phuctorings. - Phuctored: 3209...6799 divides RSA Moduli belonging to '95.215.85.243 (ssh-rsa key from 95.215.85.243 (13-14 June 2016 extraction) for Phuctor import. Ask asciilifeform or framedragger on Freenode, or email fd at mkj dot lt) <ssh...lt>; ' (host-95.215.85.243.ongnet.ru. RU)
ben_vulpes: ah i see
phf: http://btcbase.org/log/2016-06-20#1485519☝︎
a111: Logged on 2016-06-20 04:23 phf: which is handy if you're using something else to produce the patch, or if you need to use a non-trivial diff command. for example i sometimes need to exclude files from diffing, so a command might look like diff -x foo -x bar -x qux -ruN a b | grep -v '^Binary files ' | vdiff > foo.vpatch
ben_vulpes: aaaaah
ben_vulpes: BLOOD SPORTS
ben_vulpes: phf: when i crack my v again in the morrow, i'm going to implement hash-checking against longest common directory tree
ben_vulpes: ^^ mircea_popescu asciilifeform mod6 trinque and any other vtronicists pls to opine
phf: will probably cover majority of cases(tm)(c)
phf: huh, apparently plan 9's diff/patch doesn't even implement unified diff format
ben_vulpes: tight
ben_vulpes: in other unixisms: http://p.bvulpes.com/pastes/Lwefz/?raw=true
ben_vulpes: felipelalli, vexare, wyrdmantis, Sinclair6, luke-jr please fix your bouncers
phf: i noticed that btcbase supports filenames with spaces in them: if you start a filename with " it will read until a closing ". i have no idea where i got this from, because gnu diff/patch don't support spaces in names.
ben_vulpes: wait phf hang on no i don't think i'm going to do the largest common, i think i'm just going to use the output of patch to figure out what was actually patched
ben_vulpes: !up luke-jr well did you read the link or what
phf: ben_vulpes: you mean like ~parse~ the output of patch?
ben_vulpes: phf: when it works, it outputs the list of patched files
phf: aah
ben_vulpes: one can patch an empty directory with an arbitrary patch and extract the filenames patch wanted to hit
ben_vulpes: luke-jr: i know you're awake and reading this because you pm'd me. don't pretend otherwise, it's downright foolish.
ben_vulpes: !!up luke-jr
deedbot: luke-jr voiced for 30 minutes.
luke-jr: I saw the link. didn't see a problem.
ben_vulpes: top 10 in disconnects over the year? no problem?
luke-jr: no problem besides AT&T
ben_vulpes: you want to matter in crypto but "nah bro, it's just my isp" at me?
luke-jr: want to donate $30k so I can get a better ISP? :p
ben_vulpes: what part of bumfuckistan do you live in that prevents colo access?
luke-jr: I don't trust colos.
luke-jr: not that IRC is all that important
ben_vulpes: it's like omfg even aws is barely 10us/mo for a vps you don't have to trust
ben_vulpes: nah, park your boat on the lawn, who cares
ben_vulpes: wear a shirt with last weeks sweat stains on it, it's not like you're that important
davout: luke-jr: it's not like you *have* to idle in the chan, logs are public and if you have something to say, it's a /join away
luke-jr: true
ben_vulpes: a join and an up, which is predicated on the obvious
ben_vulpes: for those who *still* miss the point, joining and parting is opening and closing the squeaky doors on a hall where 5 people are arguing and 500 muffling laughter and groans
ben_vulpes: it is visible and annoying. stop it.
mod6: fwiw you can turn off those messages in your client too.
ben_vulpes: i will not spend the time to figure out how to mute everyone but noobs i've never seen before
ben_vulpes: i see a new face at the back of the hall, i'm going to give them the opportunity to at least say hello and introduce themselves.
mod6: https://weechat.org/files/doc/weechat_faq.en.html#filter_irc_join_part_quit
ben_vulpes: mod6: how does that help me filter luke-jr's spam and not noobs?
ben_vulpes: amusing innit that the father of the since-aborted 'blockchain spam' meme is now spamming irc
davout 's trb node is now up and apparently syncing at 62.210.206.141
ben_vulpes: davout: congrats!
ben_vulpes: davout: you will sync far more quickly if you -connect to a single, high reliability, high bandwidth node during sync
ben_vulpes: otherwise trb may decide to ask utter randos for blocks
ben_vulpes: and waste time negotiating connections
ben_vulpes: not that it cannot be done that way, but it is faster other ways.
davout: i have a recent prb node on the same machine, but i'm not sure it's going to work, re http://btcbase.org/log/2016-12-29#1592875☝︎
a111: Logged on 2016-12-29 23:08 asciilifeform: also i see some 'connect() failed after select(): Connection refused' which iirc is bleeding edge prb kicking trb out
davout: ben_vulpes: any suggestions for such a node?
ben_vulpes: either asciilifeform's or mine
ben_vulpes: i use whirling rust, he -- ssd's.
ben_vulpes: i believe that mod6 has a solid one as well, pete_dushenski's has been blackholed of late
ben_vulpes: i'm off, later davout
davout: allright, ty!
deedbot: http://www.contravex.com/2016/12/30/malibus-most-wanted/ << » Contravex: A blog by Pete Dushenski - Malibu’s Most Wanted.
davout: OSX, totally the platform sane people develop on "valgrind: This formula either does not compile or function as expected on macOS" hurrrr
davout: "plox to use xcode, where everything works differently, because reasons"
davout: out of curiosity, how long did it take trb node operators to fully sync?
BingoBoingo: davout: 30 days to Februrary 6th 2015 block 34236 in latest sync
BingoBoingo: syncing from wild caught peers
davout: didn't fully sync?
BingoBoingo: not yet, started November 30th
davout: ok
BingoBoingo: need to remind self blockchain only gets longer
davout: i guess patience is in order here
davout: was there a discussion of the use case where one wishes to create, and sign transactions from an arbitrary set of unspent inputs?
Framedragger: http://btcbase.org/log/2016-12-30#1593070 << http://btcbase.org/log/2016-11-21#1571809☝︎☝︎
a111: Logged on 2016-12-30 01:20 asciilifeform: yeah but one that doesn't motherfucking grind to a halt when read 1000/sec omfg
a111: Logged on 2016-11-21 12:48 Framedragger: asciilifeform: since i'm fiddling around with postgres for work anyway, i'm curious, if you find a moment, could you maybe send me the postgresql.conf file on phuctor's machine? i'd take a look (it's very possible you know much more re. what's needed there, but i'm just curious about a coupla parameters, doesn't hurt to check)
Framedragger: would still be interested to take a look, wouldn't hurt.
Framedragger: (i'm thinking about things like size of shared buffers etc)
asciilifeform: http://btcbase.org/log/2016-12-30#1593233 << all knobs set to defaults☝︎
a111: Logged on 2016-12-30 12:53 Framedragger: (i'm thinking about things like size of shared buffers etc)
Framedragger: defaults are shit.
Framedragger: (not defending postgres re. this.)
Framedragger: asciilifeform: do you have an idea how much memory you could allow postgres to eat up? i know you have that other super hardcore thing eating lots of memory on the side
Framedragger: it's something that can be very easily changed and tested without sweat or breaking things.
asciilifeform: say, 64GB
asciilifeform: what'll it give
asciilifeform: disk is not the bottleneck on this box
Framedragger: aha right. i'm doing sth else but i could later ping you with a sample postgres file which you could try out (would need db restart)
Framedragger: things like sorting
Framedragger: if you run EXPLAIN ANALYZE
Framedragger: before query
Framedragger: it'll show what it's doing
Framedragger: (you've probably done this tho)
asciilifeform: aha
asciilifeform: did.
Framedragger: i'm thinking,more memory could help with certain things that db is busy with, incl insertion, even. i'm not sure.
asciilifeform: i even spoke with career dbists, answer was 'your application is monstrous abuse and you need a cluster'
Framedragger: this isn't rigorous, but easy to try.
Framedragger: oh. that's sad. :(
Framedragger: just know that some defaults are really low.
asciilifeform: soo which knob should i max, Framedragger ?
asciilifeform: worth a try
Framedragger: busy for a bit, i don't want to cite you sth without thinking about it
Framedragger: sec
asciilifeform: and what effect will this have on the consequences of yanked mains cord
asciilifeform: because nobody cancelled physical reality, write-caching means vulnerability to mains yanking
asciilifeform: i absolutely can't have a db req returnig before disk is written.
Framedragger: right. i just thought about checkpoint_completion_target (set to say 0.9) which may help with inserts, but ultimately you're right, physical reality
asciilifeform: *returning
Framedragger: none of that.
Framedragger: to be 100% certain, i'd have to check. i see your concerns.
asciilifeform: now if only i had a pill against http://btcbase.org/log/2016-07-18#1505027☝︎
a111: Logged on 2016-07-18 18:08 asciilifeform: i know of no file system that would not choke.
asciilifeform: then -- static html phuctor.
Framedragger: that would be neat..
Framedragger: asciilifeform: i take it you are certain that main bottleneck and 'hogger' is the numerous inserts?
mircea_popescu: goood mornin!
asciilifeform: correct
asciilifeform: guten morgen mircea_popescu !
Framedragger: mircea_popescu: it's dark here in the northern hemisphere, god it's deperessing :( mornin'..
mircea_popescu: ah, it's rainy here, so.
asciilifeform: Framedragger: they aren't only inserts, every key turns into half a dozen to a dozen queries interleaved with inserts
Framedragger: ahh right, i assume those include in-memory sorts
Framedragger: or requirement for sorts anyway
asciilifeform: Framedragger: as per http://btcbase.org/log/2016-11-19#1570951☝︎
a111: Logged on 2016-11-19 18:52 asciilifeform: Framedragger: db being hammered 24/7 with 'do we have this hash' 'do we have this fp' 'add this and this' 1000/sec is the bottle.
asciilifeform: and no, no sorts there
Framedragger: work_mem (used for in-memory sorts) is 4 MB default. 4 MB. (9.5 anyway). set it to 50 MB as per advisory at least.
asciilifeform: the 'do we haves' could benefit from bigger read cache
Framedragger: ah hm. tbh i'd still change work_mem because it's ridoinculously low by default, but i hear ya.
asciilifeform: i'ma try it soon.
Framedragger: (some of those settings don't require db restart (but may require to 'flush' params), some of them do, best to restart db after all changes are made.)
Framedragger: (do note, 'work_mem' is per user / per request. so may be easier to DoS. thought i should mention this for completeness)
asciilifeform: that means i won't be trying it for couplea weeks. i don't restart db until a current parcel is through submitting.
Framedragger: ah right, yes i see
Framedragger: asciilifeform: any JOINs in those multiple queries for each 'insert'? (if yes, this param should help.)
asciilifeform: (parcels are eaten by script that, presently, has no convenient pause button. and, because unix was dropped as a baby, suspending a process doesn't yield locks, so ~that~ doesn't safely work)
Framedragger: :(
asciilifeform: Framedragger: nope
Framedragger: ok
Framedragger: asciilifeform: docs advise heavily on enabling write cache (but (sanely) insist on battery backup in that case) for your 'loads of inserts per sec' use case..
asciilifeform: the machine isn't at my house and i have no control over the mains supply.
Framedragger: "For situations where a small amount of data loss is acceptable in return for a large boost in how many updates you can do to the database per second, consider switching synchronous commit off. This is particularly useful in the situation where you do not have a battery-backed write cache on your disk controller, because you could potentially get thousands of commits per second instead of just a few hundred."
Framedragger: << need to understand just what does that imply..
asciilifeform: Framedragger was not yet here, but phuctor-1 was quite vulnerable to pulled mains cord ( would lose weeks of work ) and as soon as this became publicly known,
asciilifeform: it was ~immediately~ and ~permanently~ afflicted by 'mysterious' reboots
mircea_popescu: http://kwiva.ru/protiv-hard-forkov/ http://kwiva.ru/protiv-hard-forkov-2/ << apparently russians also exist.
Framedragger: "lose weeks of work" is insane :( i'm sorry to hear that. *this* would not expose you to that scenario. but one would have to pin down still-possible data loss scenarios, if any.
asciilifeform: mircea_popescu -- switched hosts, i -- slowly, painfully, rewrote the thing...
asciilifeform: Framedragger: well it doesn't lose work nao.
asciilifeform: but could again if i gave it a multi-GB writecache
Framedragger: here's what i'm thinking: disable synchronous_commit , but set 'checkpoints' so that results are flushed to db every $n inserts/updates. i can see however how you may barf from such an idea, "it's either reliable, or isn't".
Framedragger: but this way you would constrain any losses to particular known amounts.
asciilifeform: mircea_popescu: that's a straight (and spam-encrusted) translation of an english article that was posted here ('against hardforks' iirc) a while back.
asciilifeform: Framedragger: no losses plox.
mircea_popescu: http://btcbase.org/log/2016-12-30#1593121 << dude the problems jus' keep on coming. wtf is this, we have hashes, why THE FUCK would we care about directory and holy shit who came up with the idea of using path as hash☝︎
a111: Logged on 2016-12-30 05:22 phf: i think it treats one of the names as canonical
mircea_popescu: asciilifeform hey, i only said they exist, i didn't say their brains work.
asciilifeform: mircea_popescu: tbh i had a 'what the hell is all this' reaction to reading ben_vulpes and phf vtron problemz
mircea_popescu: it's not even baseless, which is the saddest part.
asciilifeform: they must've moved some king-sized cockroach sofa.
mircea_popescu: his problems i mean
Framedragger: "Note that open_sync writing is buggy on some platforms (such as Linux), and you should (as always) do plenty of tests under a heavy write load to make sure that you haven't made your system less stable with this change. Reliable Writes contains more information on this topic. " oh god. more inserts/sec but zero data loss => probably can't help you much. documentation doesn't encourage me :/
mircea_popescu: sigh
mircea_popescu: Framedragger data loss is catastrophic to a degree that can't be described, as far as phuyctor goes. if you have to also check, your workload goes up 3x at least.
asciilifeform: ^^
Framedragger: mircea_popescu: consider a scenario in which you knew how much data you could lose ("up to 100 last rows"), and you could check if you lost any (last row id == last-id-processed.txt ? false : true). that being said, this way things become more wibbly-wobbly, so probably fuck that. :(
mircea_popescu: http://btcbase.org/log/2016-12-30#1593130 << holy crap.☝︎
a111: Logged on 2016-12-30 05:40 phf: so if you were to produce a patch with a/old-veh.lisp and b/veh.lisp. existing vtrons will happily press it, though it's a total clusterfuck
asciilifeform: gnupatch, as i warned long ago, MUST die.
mircea_popescu: Framedragger see here's what graybeard means : i see that statement, and I KNOW there's a footnote somewhere you don't know about / bother to mention which says "except when abendstar in conjunction with fuckyoustar when it's 105th to 1095th column".
mircea_popescu: because it is not computer-possible to have what you describe without what i describe.
mircea_popescu: consider the simple case of "check values, actuate machinery" in article linked here a few months ago. it is quite fundamentally informative.
Framedragger: right. either it's completely-reliable, or NP-complete complex dragons in a cave
mircea_popescu: well not exactly like that, but i guess that may work for a heuristic early on.
mircea_popescu: the issue is the magic numbers. you said "100". why did you say "100" and how did you [think you] knew ?
asciilifeform: moreover, it's either ~completely-readable~ or 'dragons in a cave'.
mircea_popescu: that's another, but just as important, issue.
asciilifeform: i'm not convinced that they are separable.
Framedragger: mircea_popescu: because you could tell postgres to flush rows (forcing all caching layers) every 100 rows, not every 1 row as currently specified
asciilifeform: Framedragger: may as well run whole thing off a ramdisk then
mircea_popescu: what you do is not supposed to be predicated on what you can do at any point in your existence.
Framedragger: (i'm not saying 100 is not barf-magical.)
Framedragger: i suspect then that the inserts/sec slowness is due to postgres currently making really damn sure that *all* layers of cache are forced. this "full forcing of cache for every row" is what makes things slower; but it's also the only really-super-reliable approach for the case at hand (remote box).
asciilifeform: the thing is a 1,001-layer shit sandwich
mircea_popescu: Framedragger think for a second : modulus gets added, it's in cache, other modulus gets added, they don't get checked against each other because one was in cache, now we have two unpopped poppables in db.
mircea_popescu: does this entirely subvert what phuctor is ?
jurov: just a side note - if using filesystem, for ACID guarantee you'd need to flush caches, too
mircea_popescu: yes.
mircea_popescu: wich is why fs sql is contemplated for bitcoin, not really for phuctor's woes.
asciilifeform: jurov: fs does have the journal
Framedragger: brb
mircea_popescu: asciilifeform actually a portion on a ramdisk may even be judicious.
mircea_popescu: i take it you were considering this for next rewrite
asciilifeform: mircea_popescu: linux by default uses all spare ram as disk readcache
jurov: asciilifeform: journal is not some magic allowing you to have 100k transactions.second without possibility of data loss
asciilifeform: writecache, on other hand, is a major Do Not Want here, for reasons described above
mircea_popescu: yeah it has to catch on eventually.
mircea_popescu: asciilifeform ah, i didn't realise you were happy with linux readcache.
asciilifeform: jurov: there is always 'possibility of data loss', machine could be stolen (as it once was!) or burn down
asciilifeform: problem is ~silent~ data loss.
asciilifeform: when you do not know that you oughta restore from backup
asciilifeform: the one obvious optimization i was considering was to avoid all dupe checks on key submit and simply deduplicate prior to each bernsteining. but this has serious cost in ui consistency, no more could submitters expect to see a result that is guaranteed to make sense after they submit.
mircea_popescu: on the other hand submitter support is not mandated, they fail to produce a significant portion of the input.
mircea_popescu: for that matter they fail to change their own diapers, either, end up having Framedragger write code for them etc.
asciilifeform: there is 1 serious reason why gotta check for existing key/fp:
mircea_popescu: well ddos.
asciilifeform: because it is what 'phuctor as keyserver' stands on, also
asciilifeform: most of the time i paste in a key from somewhere, there it is, from april.
mircea_popescu: yeh.
mircea_popescu: incidentally, i would say deedbot also counts as tmsr keyserver now.
asciilifeform: it's a total replacement (if slow) for sks.
asciilifeform: deedbot key getter dun work in heathens, does it..?
mircea_popescu: sks was not that fast. or that complete.
mircea_popescu: asciilifeform hey, i'm not surfe i want it to work on heathens what.
asciilifeform: well here's one typical scenario, i find a pgp-signed historic patch (e.g., linus) and want to see what vintage key etc
mircea_popescu: sure. i'm not saying it must be standardized. just, there.
asciilifeform: 'was this linus-key or linus-flipolade key'
mircea_popescu: note that as time rains on, this sort of query becomes less and less interesting
asciilifeform: it will.
mircea_popescu: "was gcc 5.x rms approved or not ?" "dude... who cares. it fails to build eulora."
asciilifeform: btw while we're on this subj, does eulora eschew cpp11?
asciilifeform: because cpp11 is how folx typically end up reluctantly grunting in the stake of gcc5
mircea_popescu: diana_coman ^
mircea_popescu: but yes, afaik it does.
asciilifeform: brings own set of problems. it is where monstrous thigs such as 'boost' came from - lack of cpp11
mircea_popescu: it dun has boost either.
diana_coman: asciilifeform, it does, yes
diana_coman: and omfg no boost, no
diana_coman: ideally it will eschew cpp alltogether but so far not ideal
asciilifeform: mircea_popescu, diana_coman : so you folx wrote own 'boost'-like horror? fwiw most games firms did, in the golden olden days. my brother's co, for instance, did.
mircea_popescu: nope.
diana_coman: asciilifeform, no
asciilifeform: so what do the data structures look like ?
mircea_popescu views with mind's eye diana_coman 's beard growing inches/second in the minds of alf
asciilifeform: how about iterators? they are all explicit? million temp vars?
asciilifeform: i'ma have to gather the courage and read this thing with own eyes, at some point.
diana_coman: asciilifeform, atm we are still slowly, slowly extricating ourselves from the swamps of ps code
jurov: it does not use much c++stdlib, but the crystalspace reimplementation of
asciilifeform: ps?
asciilifeform: postscript?!
mircea_popescu: the server code's not published, and the client code is mostly legacy.
diana_coman: planeshift asciilifeform , a swamp not worth gettting stuck into
mircea_popescu: planeshift.
asciilifeform: mircea_popescu: presently i have nfi what part is published, or where
asciilifeform: ah hm. i have nfi what is 'planeshift'. 3dengine?
diana_coman: planeshift is an open source mess that was used to jumpstart eulora basically
jurov: planeshift is opensource game, crystalspace is the engine
asciilifeform: aah
mircea_popescu: asciilifeform planeshift is a mmorpg that the many-eyes beast took 10 years to make. it uses cs which is a sort of game engine, which is built on cal3d which is a gfx lib.
mircea_popescu: the quality of code is uneven in the usual foss sense ; its main virtue is that being old, it is mostly not new.
diana_coman: and it manages to have some half million lines of code doing the job of maximum 100k by the looks of it
mircea_popescu: (they did decide to move over to unity last year, then they abandoned the plan._
asciilifeform: and looks like jurov answered my q earlier. eulora folx are using crystalspace's adhoc boostron.
diana_coman: asciilifeform, planeshift does that, yes
jurov: asciilifeform: https://gcc.gnu.org/wiki/C11Status btw, they claim c++11 is fully done in gcc 4.9 (as is my experience) . maybe you meant c++14 ?
asciilifeform: in the old days, every major cpp project had one
mircea_popescu: asciilifeform the client.
mircea_popescu: there's two parts to this mess.
asciilifeform: jurov: i am guilty of referring to anything that wasn't in my childhood borland3.1 as cpp11
jurov: lol
mircea_popescu: heh
asciilifeform: jurov: notice how some of the more appealing 11isms (e.g., bounds checking) dun work
asciilifeform: 'library issue'
mircea_popescu: that's ok, the planeshift implementation leaks at pretty much every other rivet
asciilifeform: btw , the unusability of naked cpp was also why we got horrors like 'qt'
asciilifeform: (which is only half gui toolkit, it is also datastructure lib)
asciilifeform brb
diana_coman: basically crystalspace has its own "boost" implementation yes, leaking as expected and on top of that planeshift uses it all over the place quite without any rhyme or reason, adding further to the swamp;
diana_coman: fwiw I can confirm that current code compiles perfectly fine on gcc 4.4 in any case
mod6: mornin'
diana_coman: morning mod6
mod6: how ya been!
diana_coman: heh, not bad
mod6: good deal
diana_coman: got to some snow, sledged downhill, even bruised a knee , got back to peaceful coding now , lol
diana_coman: how's that eulora-box coming in 2017 mod6 ? :D
mod6: nice! i haven't done any sledding yet. gotta do that one of these times.
mod6: oh, hey, actually. so I've got a box.
mod6: i had obsd on it like for nearly all of '16... but wasn't doing anything with it. so i threw linux on there.
jurov: For the uninitiated, there's already c++17 underway. With folks gearing up to c++23, when We Will Finally Reach Parity With Haskell(noshit).
mod6: when I get a free moment, i'll throw the latest eulora on there. can be my mining box. :]
diana_coman: sounds good mod6 :)
diana_coman: jurov, make cpp haskell again I gather?
mod6: heh
jurov: yay
mircea_popescu: jurov what is parity with haskell even ?!
jurov: i was just paraphrasing, don't remember the exact word
diana_coman: asciilifeform, to round off: atm eulora code is basically c99 (even that rather reluctantly when we moved over to 64bit)
asciilifeform: jurov: догоним и перегоним ! (tm) (r) (hruschev)
asciilifeform: but yes, cpp standardization is beginning to resemble that of PL/I
asciilifeform has a relative who, until recently retiring, programmed in PL/I ! i shit thee not
asciilifeform: jurov: lulzily, searching for 'c++17' i get... airplanes
asciilifeform: 'boeing c-17'
mircea_popescu: oh also, friendly reminder today's last day of eulora hackathon. closes in ~11 hours.
mircea_popescu: http://btcbase.org/log/2016-12-30#1593170 << he's not even kidding, i'm going to start banning.☝︎
a111: Logged on 2016-12-30 06:58 ben_vulpes: felipelalli, vexare, wyrdmantis, Sinclair6, luke-jr please fix your bouncers
mircea_popescu: oh hey, check it out, irc isn't that important. smart move, barely tolerated fraudster.
mircea_popescu: and for the record : the dood colluded with sonny vleisides / the rest of the 'ndrangheta running "bfl" scam (which, obviously, the usg hasn't ever prosecuted, in spite of loud violation of, eg, parole termas, because hey, partners in crime) to falsely claim that he received a miner delivery so as to scam bitbet into misresolving a bet, on which they had ~500 btc.
mircea_popescu: if he weren't a total fucking retard on top of being a consumate conman, he'd actually have 30k to buy an isp now.
mircea_popescu: http://btcbase.org/log/2016-12-30#1593220 << depending on your setup about 40 to 60 days in the wild, about half with ben_vulpes recommended method.☝︎
a111: Logged on 2016-12-30 09:13 davout: out of curiosity, how long did it take trb node operators to fully sync?
mircea_popescu: http://btcbase.org/log/2016-12-30#1593228 << yes, periodically since 2014.☝︎
a111: Logged on 2016-12-30 12:28 davout: was there a discussion of the use case where one wishes to create, and sign transactions from an arbitrary set of unspent inputs?
mircea_popescu: http://btcbase.org/log/2016-12-30#1593252 << these people. if phuyctor is not THE usecase then wtf is. wwwrot ffs.☝︎
a111: Logged on 2016-12-30 14:20 asciilifeform: i even spoke with career dbists, answer was 'your application is monstrous abuse and you need a cluster'
mircea_popescu: http://btcbase.org/log/2016-12-30#1593286 << actually workmem should be 256mb especially as you can afford it so totally, go for it.☝︎
a111: Logged on 2016-12-30 14:30 asciilifeform: i'ma try it soon.
mircea_popescu: http://btcbase.org/log/2016-12-30#1593288 << pretty sure phuctor is 1 user that recycles db connections.☝︎
a111: Logged on 2016-12-30 14:32 Framedragger: (do note, 'work_mem' is per user / per request. so may be easier to DoS. thought i should mention this for completeness)
asciilifeform: http://btcbase.org/log/2016-12-30#1593456 << iirc it took me ~3 days to sync via direct eatblock☝︎
a111: Logged on 2016-12-30 16:06 mircea_popescu: http://btcbase.org/log/2016-12-30#1593220 << depending on your setup about 40 to 60 days in the wild, about half with ben_vulpes recommended method.
asciilifeform: or hm, nm, 1 night
mircea_popescu: direct eat block is fast yes.
mircea_popescu: but you have to have the food.
mircea_popescu: incidentally asciilifeform since we're now doing open source db optimization shared_buffers is probably a larger concern. what is it ? defaults to 125mb but i'd readily see it 1-4gb in your case.
asciilifeform: the fastest sync method, supposing one has access to a synced node, but also supposing that it won't do to simply copy the blocks (and it won't, you want to verify) is an eater-shitter system
asciilifeform: mircea_popescu: probably. i'ma run with new knob settings as soon as it is safe to reset the db.
asciilifeform: but i do not expect miracle.
mircea_popescu: alrighty then let's make a full plan here.
mircea_popescu: 1) shared_buffers is to be per spec "25% of available ram" ; but it does diminish returns in the gb. you probably have it as 128mb, make it 2gb say.
asciilifeform: (because apparently 'thousands of queries / sec is abuse, get a cluster' is the 'state of the art')
mircea_popescu: 2) huge_pages should probably be on.
mircea_popescu: asciilifeform do you use a lot of temp tables ?
asciilifeform: mircea_popescu: i'd first like to know what this'll do to integrity-on-mains-failure
asciilifeform: mircea_popescu: 0 temp tables
asciilifeform: 0 anything fancy.
mircea_popescu: and work_mem should prolly be larger than 4mb, but hard to guess how large without a profile, and this is a major resource consumption ticket, so you actually want to do some maffs.
asciilifeform: 0 joins. 0 anythings. just plain old queries and inserts.
mircea_popescu: yes but what it uses it for is sorts, select by index may use it if the index is composite.
asciilifeform: ah
mircea_popescu: do you actually just recycle one connection or do you keep making connections ?
asciilifeform: makes connections.
mircea_popescu: ah then not nearly as important.
mircea_popescu: asciilifeform all these are memory usage ops, what they do is establish when it should go on disk. they do not significantly affect cord-yank robustness. there are other specs you can make for the background writer for instance that do.
mircea_popescu: eg bgwriter_delay you may want to be set low for this reason.
mircea_popescu: it's 200ms by default.
asciilifeform: the db absolutely has to be in a consistent state at all times, or 0 phuctoring takes place.
asciilifeform: (this scenario actually played out once.)
mircea_popescu: you can also set bgwriter_lru_maxpages to 0 and disable background writing altogether
asciilifeform: (and, painfully, i had to find the offending garbage by hand!)
asciilifeform: also did i mention that the entire db get shat out every time we bernstein ?
asciilifeform: (the moduli have to turn into an array of bignum*)
asciilifeform: this is easily 10% of the load on the db
trinque: might be faster to do in the db
asciilifeform: oh and then, factors are found, largely the same set every time (how bernsteinization works) and each one is queried to the db
trinque: I am sadly, quite good at SQL if you want the thing translated
asciilifeform: trinque: i need random-access in O(1) to them for bernsteining
asciilifeform: so no, they can't 'live in db' while it happens
trinque: ok
asciilifeform: have to live in MY data structure, in ram.
asciilifeform: where i have O(1) access to them.
asciilifeform: the whole thing working at all is predicated on these seemingly 'abusive' design choices
asciilifeform: and postgres is ~the~ albatross.
phf: so you basically snapshot your entire dataset back into the database at certain times, and snapshot is an equivalent of set merge?
asciilifeform: phf: nope. the only thing that happens to db as a result of bernsteinization is N queries 'do we already know this factor'
asciilifeform: if answer is 'no', it is inserted in 'factors' table.
asciilifeform: this, by all rights, ought to be a batch query. and probably will be in next version.
mircea_popescu: trinque 's idea, bernstein as prepared queries, may be a gain.
mircea_popescu: though i am unaware anyone ever implemented this ; because, of coruse, i am unaware anyone used the guy's algo for any other purpose than gawking.
mircea_popescu: but he might be interested to hear about it.
asciilifeform: mircea_popescu: it's a screaming nope
asciilifeform: mircea_popescu: algo ~demands~ O(1) random access to the bignums.
asciilifeform: the individual bignums.
mircea_popescu: so ?
asciilifeform: so holy shit is this not screamingly obvious
mircea_popescu: you implement bernstein IN the db. it is actually a programming language.
asciilifeform: i need ~less~ access to db, not moar.
phf: asciilifeform: oh so you do insert to a set, every time there's a result, and you query for the whole set before you start a cycle of process?
asciilifeform: mircea_popescu: sql doesn't have a bignumatron
trinque: asciilifeform: a temp table is in RAM
asciilifeform: much less an optimized one
trinque: but I am not arguing for something here; you'd know what you want
asciilifeform: phf: bernstein's algo operates on ~all known moduli simultaneously~
asciilifeform: there is no way around this.
asciilifeform: by definition, that's what it does.
asciilifeform: it is, by lightyears, the best known algo for batch gcd, also.
mircea_popescu: and you do it as prepared queries, which get precompiled to a degree
phf: asciilifeform: i'm just trying to establish the dataflow here, for my own curiosity
asciilifeform: the querying of 'do-we-have-this-factor' is maybe 1% of the load.
asciilifeform: so it has not been a priority, because batching will tremendously complicate the moving parts.
mircea_popescu: not a matter of access
asciilifeform: http://btcbase.org/log/2016-12-30#1593516 << recall, i wrote to bernstein himself.☝︎
a111: Logged on 2016-12-30 16:28 mircea_popescu: but he might be interested to hear about it.
asciilifeform: 0 answer.
asciilifeform: and at this point it is imho unlikely that he has not heard of phuctor.
asciilifeform: so that leaves 1 plausible explanation.
mircea_popescu: you're not addressing the idea. currently you use a pile of c code you labeled for purely personal reasons "a db" to store some data for you, and another pile, you labeled phuctor, to bernstein and do other things on the db-stored data. because the interface is the bottleneck, it then becomes clear you must merge this. one way is to merge by lifting the db code and putting it into phuctor, making it you know, its own db like
mircea_popescu: bitcoin wants its own fs. ANOTHER way, is to use the means the db already offers for this.
asciilifeform: mircea_popescu: if you think trb is a hard nut to crack, picture reading, grasping postgres.
mircea_popescu: which yes takes some work, but not quite as much as the other variant.
asciilifeform: i for one do not expect to live long enough to make a serious attempt at such a thing.
mircea_popescu: yes but it has this convenient hole through which you can go in, which is - implement bernstein IN sql.
asciilifeform: a sql or similar db system with built-in bignumatron could be useful and interesting. but no such thing exists. nor would it solve the actual bottleneck in phuctor if it were to be discovered tonight.
mircea_popescu: rather than in c.
asciilifeform: because the actual bottleneck is '1000s of queries AND inserts / second AND guaranteed realtime consistent'
mircea_popescu: it has ~some~ ability to precompile your queries, which is somewhat like linking object code.
asciilifeform: and not the bernsteining.
asciilifeform: understand, the only reason why the thing works at all, is that this one small part of it, the bernsteinization, can be made ~entirely~ independent from the db locking idiocy
asciilifeform: if it somehow had to happen inside postgres, it would not bypass the lock.
asciilifeform: mircea_popescu is seeing it through the naive vertically integrating rockefeller eyes, 'power plant expensive? let's put it right in my mansion'
asciilifeform: this does not always help.
phf: well, a more practical approach would be to adapt phuctor c part to a postgresql loadable module interface. in which case you he will eliminate the cross-boundary overhead (serialize/deserialize over the "wire").
asciilifeform: phf: what part of 'this isn't the bottleneck' was unclear
phf: asciilifeform: did you understand what i said?
asciilifeform: i think so?
mircea_popescu: well, it was a thought.
phf: what i'm saying is that a significant fraction of "1000s of queries AND ..." is the cross-boundary. you compile queries on c side, you send them to psql, it then parses, prepares results, serializes, sends it to c side, c side has to now parse all over again
asciilifeform: phf: actually the wwwtronic piece of phuctor is in python and does the precompiled queries thing
asciilifeform: that that's probably not it.
asciilifeform: believe or not, i actually put some work into this thing
asciilifeform: the current iteration is, iirc, the third from-the-ground rewrite.
asciilifeform: (or second, depending how to count)
mircea_popescu: pretty surreal.
phf: precompiled queries are a fraction of cross-boundary issue
phf: i'm not even arguing with you, i'm saying that the ~full extent~ of what "move it to psql" is going to do is ~eliminate cross-boundary issue~ that is all. so it'll shave some significant overhead, but it's not a silver bullet.
asciilifeform: when i profiled it, 99% of the time is spent in 'do we have this key hash? no? insert; do we have these fp's? no? insert...'
asciilifeform: phf: understand also, postgres can't store bignums as such, it stores strings
mircea_popescu: so a lot of hash search.
asciilifeform: these end up parsed into operable bignums every shot. but, surprisingly, this never takes > 3 minutes !
asciilifeform: on entire set.
phf: asciilifeform: right, i was going to get to that :}
mats: fbi evidence of ru hacking 10/10 lulz as expected
asciilifeform: (in fact, dumping out the entire db, and properly bignumizing, takes about 3min total for the current db.)
asciilifeform: mats: i especially loved the 1 single av signature offered
asciilifeform: (which was for some 'script kid' php kit)
mircea_popescu: asciilifeform if that's where it spends most time then a) http://btcbase.org/log/2016-12-30#1593462 is very likely to help and b) preparing your whole query as ONE single sorted item will help also.☝︎
a111: Logged on 2016-12-30 16:11 mircea_popescu: http://btcbase.org/log/2016-12-30#1593286 << actually workmem should be 256mb especially as you can afford it so totally, go for it.
mircea_popescu: i gather you already do b. is it index-sorted ?
asciilifeform: mircea_popescu: actually i do not. because it will require 100x more complex mechanism.
asciilifeform: which is to say, rewrite of WHOLE thing. we had a thread.
mircea_popescu: uh. then why do you put the keys in in batches if you're not... putting them in in batches ?
asciilifeform: they get thrown into same hole as if human submits.
asciilifeform: that way there is exactly one procedural path for key submission, and no duplicate logic.
mircea_popescu: would you stop with these bizarro deflections, they neither impress nor persuade, but they do give you an ugly image.
asciilifeform: i dun give half a shit about 'image'. laying out the fact of why the thing is as it is.
mircea_popescu: yes, well, that's then the problem. they should go in as a single query the size of the batch, with the items sorted within it
mircea_popescu: this is a piddly excuse, "no duplicate logic", case of luser with wwwform and case of 100k keys in batch form are different enough to warrant duplicate logic. that's why computers even exist, to account for such level of difference in code.
mircea_popescu: otherwise we'd just use hardware everywhere.
asciilifeform: it is the most obvious unmassaged piece , aha. the correct algo is , imho, to have separate 'nursery' (gcism term of art) table for the batch submits.
mircea_popescu: possibru.
asciilifeform: but what this adds up to is to have ~two~ quite separate phuctors. we wouldn't query the nursery, for instance, when someone keys in a url with a hash
asciilifeform: only the 'adult' db
asciilifeform: otherwise we get same speed as now.
mircea_popescu: no, have one phuctor with one db and two intakes.
mircea_popescu: and yes we would query.
asciilifeform: well 1 db, 2 sets of key/fp/factor tables.
phf: a sort of impolite question, but is there's an index on hash column?
asciilifeform: and no, you can't query the nursery every time somebody loads a url, or you get SAME performance as now, omfg
Framedragger: (and follow-up, does explain analyze show the use of that index)
asciilifeform: phf: there is
asciilifeform: mircea_popescu: but yes, for next version (presently only exists in my notebook) there is a nursery and it gets merged into main table at night. but this makes for considerably more complex system, where there are two very distinct types of submission, 'realtime' and 'scripted' , and they get treated quite differently.
asciilifeform: (and bernsteinization requires access to ~all~ moduli, as i think is obvious, and not simply 'most recent ones')
phf: is it a hash index? it has the least overhead (it isn't logged amongs other things, so you have to rebuild it on crash, but conversly it's kept in memory and only supports = operation) indexes will make your queries more cheap, but writes more expensive, so you want to make sure it's the cheapest possible
asciilifeform: phf: what means 'rebuild it on a crash'
asciilifeform: this has to be done programmatically ?
asciilifeform: mircea_popescu: the other obvious thing would be to dispense with 'real time submission' entirely, and when someone dumps in a key, it goes into next batch. but we discussed this earlier in this thread, it would mean that the thing cannot be used as sks-like tool.
Framedragger: docs say would need to get rebuilt only if there were any unwritten changes. which there shouldn't be as asciilifeform is not using write cache
deedbot: http://phuctor.nosuchlabs.com/gpgkey/0CCA49DE4C9967BFAE78ACF9D1AD438154B75B9700A055CA3DAC80F1714A2AA0 << Recent Phuctorings. - Phuctored: 7 divides RSA Moduli belonging to 'Connie Main <cma...om>; ' (host-95.215.85.243.ongnet.ru. Unknown)
deedbot: http://phuctor.nosuchlabs.com/gpgkey/289FCBF68419984FD484C7EF7823AB7C114193224DD2733C7A60A20BC118F5A6 << Recent Phuctorings. - Phuctored: 7 divides RSA Moduli belonging to ' <spaf@mac.com>; <spa...du>; Gene Spafford <gene@spaf.us>; Gene Spafford <spaf@acm.org>; Gene Spafford <spaf@mac.com>; Gene Spafford <spa...du>; Gene Spafford <spa...rg>; Eugene H. Spafford <spaf@mac.com>; Eugene H. Spafford <spa...iz>; Gene Spafford <SPA...du>; Gene Spafford <spa...
deedbot: http://phuctor.nosuchlabs.com/gpgkey/A07FCCF0D46AC8B25EB8F0982629537817E0CEA47BCC6C8B800A06F4F4647160 << Recent Phuctorings. - Phuctored: 7 divides RSA Moduli belonging to 'Todd A. Outten <out...om>; ' (host-95.215.85.243.ongnet.ru. Unknown)
mircea_popescu would not be particularly surprised if served with 100mb of query as per above postres wouldn't just fall over.
asciilifeform: aaaaah lel
asciilifeform: holy shit was that a ... user submission?!
phf: asciilifeform: no, it's a command that you run, like REINDEX index_of_things; it simply queries what's already in DB and warms up the cash
asciilifeform: looks like flipolade
asciilifeform: (won't import in gpg, but ~will~ in js www-based shitpgptrons)
asciilifeform: http://usagl.com << from above, lulzy, old-school voice telecom co.
mircea_popescu: http://btcbase.org/log/2016-12-30#1593602 << no. this is nonsense, and not what was at any point either suggested or discussed.☝︎
a111: Logged on 2016-12-30 16:48 asciilifeform: well 1 db, 2 sets of key/fp/factor tables.
mircea_popescu: you are getting to where it is in principle not worth anyone's time to talk with you, because your response is random nonsequitur.
asciilifeform: mircea_popescu: suggest algo, i'm all ears.
mircea_popescu: if this is the path you must walk to go from solipsist-alf to socially-integrated-alf i can see it, but hurry it up already it's irritating.
asciilifeform: mircea_popescu might be social-integrated-genius but often recommends algo that adds up to escherian skyscraper. and so results in headache thread.
deedbot: http://phuctor.nosuchlabs.com/gpgkey/2398E0817D454688D06524E1B99CCE125A5E4D5E4DB5FBEFBE1BBE65BDA99AB4 << Recent Phuctorings. - Phuctored: 1730...1787 divides RSA Moduli belonging to '150.187.4.208 (ssh-rsa key from 150.187.4.208 (13-14 June 2016 extraction) for Phuctor import. Ask asciilifeform or framedragger on Freenode, or email fd at mkj dot lt) <ssh...lt>; ' (Unknown VE A)
deedbot: http://phuctor.nosuchlabs.com/gpgkey/9AC623C503B5F6FF091E7B5819FAD4EE293D03B779770C1959FD3C159D6653FB << Recent Phuctorings. - Phuctored: 1036...2769 divides RSA Moduli belonging to '77.37.28.20 (ssh-rsa key from 77.37.28.20 (13-14 June 2016 extraction) for Phuctor import. Ask asciilifeform or framedragger on Freenode, or email fd at mkj dot lt) <ssh...lt>; ' (Unknown DE HE)
mircea_popescu: so : if loading the whole batches of keys through the user-wwwform process is what 99% of the machine time goes to, then yes, put the batches into a single, sorted query, make the workmem should be 256mb or 2gb or w/e it is you actually need to cover your query (yes this can be calculated, but can also be guessed from a few tries) and then run bernstein after every such query, on the db not on "nursery" (which yes, it's a ter
mircea_popescu: m of art in whatever, am i not impressed!)
asciilifeform: how to make the www piece respond at all while this runs ?
asciilifeform: (and yes, it is the obviously correct way to process thous. of keyz, no question)
asciilifeform: mircea_popescu: point of 'nursery' was to do the 'do we have this fp? how about this? ...' a few thou. at a time, is all.
asciilifeform: if there is some other way of doing it, i'm all ears
mircea_popescu: asciilifeform as to "how to make www respond", you use the method we were discussing last time, whereby www is a cached image and if out of date tough for viewer ; as to nursery "do we have this ? how about this?" you really want the db to do that for you, it's ~the only thing it;s good for.
asciilifeform: phf, mircea_popescu , et al : one thing that would immediately make a very palpable difference in speed is if there were a permanent way to order postgres to perform all reads immediately, disregarding all locks.
mircea_popescu: i dunno if it supports dirty reads.
mircea_popescu: mysql doesn't lock reads on write locks ; i expect any rmdbs should be capable via config.
asciilifeform: mircea_popescu: it is currently a cached image, i implemented it. the cached snapshots however last for a limited time (iirc i have it set to half hr per url)
asciilifeform: mysql is a shitsandwich, and i will not touch it (it fucking CRASHES)
mircea_popescu: you don't in general want the frontend to be able to expire your cache, let the backend do it whenever it feels like it.
asciilifeform: for all of the cruel things i have said about postgres : it crashed 0 times.
mircea_popescu: i wasn't proposing mysql in any sense.
asciilifeform: aah ok
mircea_popescu: but - dirty reads should be possible on any db system
asciilifeform: if someone knows , from memory, the relevant knob: please write in.
phf: pretty sure not on postgresql, they are strict about their acid
mircea_popescu: uh.
asciilifeform: yeah when i put on my shit diving suit and went down into the docs, i found none.
asciilifeform: (doesn't prove that it is absent)
mircea_popescu: that's pretty sad.
trinque: asciilifeform: https://www.postgresql.org/docs/current/static/transaction-iso.html
asciilifeform: trinque: that looks potentially useful, i'ma look at it in detail when i come back from meat .
asciilifeform bbl, meat
mircea_popescu: enjoy
phf: typically you handle it by not making your query lock the entire table, using a where clause of some sort. like if you're inserting things in batches, you can use a batch counter, and you query against max last known batch counter or less (or a variation of)
mircea_popescu: kinda halfway solution innit
jurov: this is decided by transaction isolation level trinque posted. by default, table gets locked only if you explicitly "select for update"
mircea_popescu: it really should be up to operator wtf, if i want to read dirty let me read dirty what sort of decision is this for designer to make.
phf: jurov: well, it's not clear where "disregard all locks" comes from in the original request. if the actual operations are as asciilifeform describes, i.e. sporadic inserts, and sporadic selects, then there will be no locks. my point is that there's no "disregard all locks" in postgresql, you solve it by knowing what lock you're hitting, and then designing your query to sidestep the lock
mircea_popescu: this makes sense to someone ?
jurov: Flly lockless dirty read is likely a security hazard (due to race conditions, you may end up reading memory you ough not to)
jurov: So, you have to live with locks and know them
phf: mircea_popescu: sop outside of mysql world. dirty read is considered a liability, so whole point of db systems design is to ensure that you don't hit locks when you shouldn't.
mircea_popescu: does either of you see how this is the db writer outsourcing his incompetence on the user ?
davout: http://btcbase.org/log/2016-12-30#1593458 <<< vaguely rings a bell, does anyone have some pointers at hand so i can go read/re-read?☝︎
a111: Logged on 2016-12-30 16:07 mircea_popescu: http://btcbase.org/log/2016-12-30#1593228 << yes, periodically since 2014.
mircea_popescu: nevermind "mysql world" and "security" claptrap. the point of fact is you want me to cut off my hand so my helmet will fit.
jurov: This is not incompetence, but unsolved problem in CS.
mircea_popescu: oh i see.
phf: mircea_popescu: yes, ~having to deal with locks~ happens past the limit of db designer's competence
mircea_popescu: exactly how the statements {"do not allow anyone else to write here until i say" ; "let anyone read anything at any time"} amount to an "unsolved problem in cs" ? and wtf cs is this we speak of, sounds more like chewinggum-science.
davout: http://btcbase.org/log/2016-12-30#1593472 <<< trb -> trb only possibru, or could prb -> trb also work?☝︎
a111: Logged on 2016-12-30 16:14 asciilifeform: the fastest sync method, supposing one has access to a synced node, but also supposing that it won't do to simply copy the blocks (and it won't, you want to verify) is an eater-shitter system
phf: mysql solution is to creatively relax acid and hope things will "just work", which is the flip side of "mysql crashes all the time"
davout: phf: iirc mysql's innodb lets you choose your isolation level per transaction
mircea_popescu: phf think for a second : the whole FUCKING POINT of a semaphore, of any kind, is that user can't know what the other item involved is doing. if they could know, they wouldn't "avoid the locks", they'd avoid the bad write outright.
jurov: davout: you can sync only from prb 0.10.4 and earlier
davout: jurov: because prb's dumped block format changed?
mircea_popescu: pretty much.
mircea_popescu: they have a new protocol.
jurov: davout: the ondisk format (not blocks, but index) changed much earlier, oorc at 0.9 or so
jurov: i was talking about network syncing
davout: ah yeah, i'm currently syncing off ben_vulpes, i was wondering if dumping blocks from prb and then eating them with trb would work
mircea_popescu: prb has dumpblock ?
davout is currently checking
davout: found "preciousblock" command. wtf
mircea_popescu: lol
davout: hahah god, check this out http://wotpaste.cascadianhacker.com/pastes/8CwCf/?raw=true
davout: so yeah, prb does have a way to dump a block to hex from a block hash, and a way to get a block hash from a height, looks like this could work
mircea_popescu: o look, a precious block!
mircea_popescu: davout keep us posted.
davout: did you just assume my diligence?
davout: maybe i'm too lazy to script this and can live with waiting a month to sync!
phf: mircea_popescu: i'm not quite groking what the bad write is. are you saying that instead of intermingling writes and reads, you should batch them, and not write while you're reading?
davout: looks pretty trivial tbh, will probably end up doing it
davout: what's the syncing bottleneck on trb's side?
davout: actually fetching the block data?
mircea_popescu: phf i am saying that if you imagine the user can be relied on to "know where the locks are and read around them" then you are therefore necessarily saying "locks are useless - user can always know what he wanted locked and simply not write there hurr"
davout: mircea_popescu: seems to me like it would reduce to 'moving the problem'
mircea_popescu: because, again, a semaphore exists because the user does not know what the user is doing.
phf: mircea_popescu: that's not quite what i'm saying.
mircea_popescu: davout this is entirely my argument : they've moved the problem and call this "modern db"
mircea_popescu: and im supposed to be so cowed by the risk of being called mysql-sometrhing that i'm not going to say anything or i dunno
phf: that's not even close to what i'm saying though.
mircea_popescu: say it again, mebbe it sticks this time.
mircea_popescu: davout neh, block verification.
davout: but then, how can it vastly improve sync time to feed blocks from same machine instead of letting trb suck them from the network?
jurov: mircea_popescu: you would trade speed for occassionally getting garbage when you call read()?
davout: and re locking, how's a RDBMS to provide ACID guantees without locking?
mircea_popescu: whether i would or i wouldn't IS NOT THE DB'S DECISION, jurov .
phf: you have your basic database requirements: atomicity, consistency, isolation and durability. these are axiomatic, you either expect them to hold or there's no point in further elaboration. at least SQL from the conception guaranteed the four requirements. "dirty read" violates consistency. your table might be half way through an update, you do a "dirty read", which is necessarily faster than update, and you have half the results with
phf: old values, other half with new values. you ~can~ guarantee four requirements ~without~ using locks
davout: (not saying that locking should be mandatory ofc)
mircea_popescu: phf here's the problem : moder(field) consists of take field, redefine it in a practically useless but superficially persuasive way, then bad_words() to whoever dares ask if your "field" solves any important questions in the field. because of course it doesn't, MIT is the premier institution in science(*) and technology(*) in the werld.
phf: but that's the slowest option, so you have strategies for increase of speed that involve strategic placement of locks
mircea_popescu: whether i want consistency as arbitrarily defined by you is my decision, not yours.
mircea_popescu: one cuts and the other picks. if you cut db field into "acid" i pick you out of existence.
jurov: mircea_popescu: in this case asciilifeform categorically claimed he decided to have consistency, or are you deciding otherwise?
mircea_popescu: you are confusing two consistencies. the problem here discussed is dirty read by www ; its consistency with the actual db is not seriously contemplated.
mircea_popescu: meanwhile inconsistency within the actual db are a different matter.
jurov: so you're fine if wwwtron occassionally read mangled pointer and returned for exampel contents of /etc/passwd?
jurov: this is what it boild down into
jurov: *boils
mircea_popescu: and here's exactly the problem of superficiality : "you either expect consistency or there's no point in discussing". there's LEVELS. maybe i expect all my writes to be consistent and don't care by A CLASS of reads being consistent. this is a consistency model that's consistent.
mircea_popescu: jurov no ; but i am fine with wwwtron ocasionally reading a field that has meanwhile been updated, and giving old, of an unspecified age but less than x time.
davout: jurov: i think it's more like nobody gives a shit if static wwwtron is out of sync with DB
mircea_popescu: and THIS is what i mean re "problems in the field". whopee, idiots who can't code still want to be "at the forefront of computing" so they made a modern db that doesn't work.
mircea_popescu: and i'm supposed to care about the fact that they don't know how to write a db that doesn't spit out passwd ?
mircea_popescu: and if i don't im racist and rapist ? really ?
jurov: mircea_popescu: this is the problem with c machine, that everythign is pointer, and without preemptive locking, you can't distinguish your pointer points to merely stale data vs. garbage
mircea_popescu: oh i see. it's the c machine. ok then.
mircea_popescu: how about prostgres===mysql because guess what, c machine. hm ?
jurov: cuz buth postgres and mysql are in C?
mircea_popescu: YOU JUST SAID THIS!
phf: well, "you either expect" is because ~sql~ as a db language is specified to have acid. there are databases that support dirty reads/writes they are just not "sql"
mircea_popescu: phf you'll have to link me to this.
davout: phf: you're saying postgresql doesn't have a "read uncommitted" transaction isolation level like innodb?
jurov: phf no mircea imagines he can order c machine "read me this without any locking, but it must be in some class or!"
phf: davout: no, nor does oracle :o
ben_vulpes: http://btcbase.org/log/2016-12-30#1593697 << scripting is too much work, just manually dump every block and then manually load it into trb once the previous eat completes. nice meditative activity☝︎
a111: Logged on 2016-12-30 17:40 davout: maybe i'm too lazy to script this and can live with waiting a month to sync!
mircea_popescu: davout no he's saying it's not in the sql spec! which, considering how specwork goes, he might be even right about some version.
phf: i'm wrong, sql92 allows dirty read in read uncommitted
mircea_popescu: alright.
mircea_popescu: !#add precious point
asciilifeform: http://btcbase.org/log/2016-12-30#1593682 << prb aint got dumpblock☝︎
a111: Logged on 2016-12-30 17:33 davout: jurov: because prb's dumped block format changed?
mircea_popescu: apparently he found something, dunno.
mircea_popescu: we'll see how it goeth.
davout: asciilifeform: it does have a command that shits hex at me given a block hash
mircea_popescu: well that's the next best thing.
davout: oic, dumpblock dumps binary?
asciilifeform: aha
asciilifeform: whole block
mircea_popescu: davout the suspicion is that relevant data may be missing from the thing, but we really dunno.
asciilifeform: eatblock -- eats it
davout: well, either a block verifies, or it doesn't
mircea_popescu: the bar is higher, but anyway, yes.
asciilifeform: all of my nodes, fwiw, descend from the eatblock experiment
asciilifeform: which, in turn, ate mircea_popescu 's vintage block set
davout: i'm still curious what would make this kind of setup where i script "prb dumpblock | hex2bin | trb eatblock" much faster than syncing from network if the bottleneck is indeed the block verification?
mircea_popescu: filtering a chain out of the soup outside like BingoBoingo is not without merit.
mircea_popescu: davout block verification is the bottleneck in the dump-eat block process.
ben_vulpes: davout: my node is for example, busy sometimes serving blocks to other people
mircea_popescu: otherwise, the bottleneck is the shitsoup outisde.
ben_vulpes: sometimes verifying a new block
phf: davout: you don't get consistent, uninterrupted, sequential chain of blocks. the actual distribution pattern is a mess, that "orphanage" was bandaiding
davout: ok. imma give it a shot
asciilifeform: davout: mempool operation slows sync 100x
asciilifeform: ditch it, and ditch randos and their shitblocks, and 0--current sync takes 6 or so hrs.
asciilifeform: hence eatblock on airgapped box.
asciilifeform: http://btcbase.org/log/2016-12-30#1593675 << mircea_popescu nails it: postgres is crippled 'for yer own good'☝︎
a111: Logged on 2016-12-30 17:29 mircea_popescu: exactly how the statements {"do not allow anyone else to write here until i say" ; "let anyone read anything at any time"} amount to an "unsolved problem in cs" ? and wtf cs is this we speak of, sounds more like chewinggum-science.
asciilifeform: it is precisely chewing gum, 1000 tonned of it
phf: not just postgresql mind you. oracle definitely, mssql as far as i know
asciilifeform: *tonnes
asciilifeform: all of'em
asciilifeform: liquishit.
asciilifeform: 'keep monkeys from injuring self and others'
jurov: asciilifeform: so is there any database in existence that allows it? without occassional garbage?
asciilifeform: http://btcbase.org/log/2016-12-30#1593712 << why should i ever get garbage when reading ~unrelated~ datum?!!☝︎
a111: Logged on 2016-12-30 17:49 jurov: mircea_popescu: you would trade speed for occassionally getting garbage when you call read()?
mircea_popescu: dude the fact that every other girl in your class is a slut isn't going to feed you or your baby.
asciilifeform: or wait, 'spittoon is in one strand' ??
jurov: because the db was in the middle of balancing some datastructure?
mircea_popescu: so let it use locks :D
asciilifeform: it's malignantly retarded, and i'ma burn it down.
asciilifeform: all of it.
mircea_popescu: sadly chewing gum is not flammable.
jurov: my point is, if you want to read arbitrary stuff at arbitrary time, you must *carefully design* for it. you don;t get it for free on c machine
asciilifeform: hence the slow methodical spray of petrol
mircea_popescu: this point has some merit, but we're reading "arbitrary" stuff not arbitrary stuff, it's addressed to the db abstraction which is allowed to handle it, not directly to pointers.
mircea_popescu: i don't demand db hand over real memory addresses.
asciilifeform: i pissed on 'db' concept as a student, and i piss today: custom data structure for each job! the year ~is~ 1972.
asciilifeform: prolly forever
mircea_popescu: this is not so differeny from my "if you absolutely must hire shaman, hire mysql, it's cheapest"
asciilifeform: sql is exactly the infamous vice-grip: 'the wrong tool for every job'
trinque: sure, but the tool for vast piles of relational time series data looks very much like relational database, but not for idiots.
asciilifeform: and sometimes, cheap shaman is a disaster, and you want an actual surgeon.
mircea_popescu: what you want matters as much as what you hope.
mircea_popescu: you get what there is.
asciilifeform: well yes, you get cut open by the butcher you have, not the surgeon you wish you had
mircea_popescu: anyway. this has been, at least to me, an informative excursion.
asciilifeform brb
phf: dirty read would definitely solve me a lot of headache now, though not enough motivation to switch to mysql. not so much when i worked on oracle for a g-sib where you want acid, so instead "avoid bad writes"
mircea_popescu: there's a reason mysql owns the web, and that has to do with this very specific www-powered profile described above, http://btcbase.org/log/2016-12-30#1593729☝︎
a111: Logged on 2016-12-30 17:56 mircea_popescu: jurov no ; but i am fine with wwwtron ocasionally reading a field that has meanwhile been updated, and giving old, of an unspecified age but less than x time.
mircea_popescu: which is just about what the web MEANS. www = "that data exploration mechanism which ocasionally puts out old data, of an unspecified age but younger than x".
mircea_popescu: to go into trinque 's mullings about the meaning of things and items.
phf: well, it's also reason why the kind of stuff you could run on a beefy dreamhost now requires $5k/mo amazon rds instance
mircea_popescu: im not sure i follow this one ?
jurov: mircea_popescu: yes, but most of www can live with crashing database (even innocent reading can cause sigsegv, sadly)
phf: mircea_popescu: rds is amazon's hosted relational database solution(tm) which is a postgresql on a unixbox
mircea_popescu: jurov trilema db never crashed, in 9+ years and however many quintillion queries.
trinque: the reason I flip the process and say that db writes static www, rather than www reads db, is that the write of www matter can be transactional with db state update.
trinque: consider the case where a form is generated based on db state, and the validity of that form depends on db state
mircea_popescu: phf yes, but a fine approach to answering "what is the basis of alf's value as an engineer" is pointing out that he runs phuctor on the phuctor box, which fails to cost 5k/mo.
trinque: you otherwise get a case where one user can't submit his form, because mismatch between UI and acceptable-insert
trinque: then from there you can optimize and say "I don't care if one guy gets stale form, need moar speed"
mircea_popescu: of course random dorks go on about how no such labs will go out of business through the unlikely avenue of delivering what is clearing 1mn/year worth of services out of <1k/month, but hey.
trinque: and forgo the transactional write of the form
mircea_popescu: that's what the web is for.
phf: true, but alf also "won't touch the web"
mircea_popescu: except he does, evidently.
mircea_popescu: he's written more webfacing stuff than ~everyone else. he has teh red army spirit.
phf: for the republic, not for hitler. he lacks that certain "good german" spirit, ja
mircea_popescu: well, dunno about "than anyone else" in that very general form, but certainly more than you'd expect or he'd have hoped.
mircea_popescu: trinque the idea isn't without merit.
jurov: http://btcbase.org/log/2016-12-29#1592846 << I have synced and pressed makefiles.vpatch, but there's no C code, only makefiles☝︎
a111: Logged on 2016-12-29 22:54 ben_vulpes: jurov: would you be so kind as to update the lxr with makefiles.vpatch ?
jurov: ben_vulpes et al.: how you want to present this in lxr? or are there more steps?
mod6: wut
mod6: everything, if press happened correctly, should be under bitcoin/src
jurov: ok, i'll rather start again from the beginning. what's the newest v.pl version?
mod6: jurov: http://thebitcoin.foundation/trb-howto.html
davout: jurov: werked for me :3
davout: asciilifeform: why is there a specific -caneat flag? is there something specifically dangerous about eating blocks?
jurov: me was working with V-20151014.tar.gz :)
ben_vulpes: jurov: have you run msft updater today? :P
mod6: jurov: in that howto, you'll find a series of both offline steps, and online steps. you can choose your own adventure.
jurov: i see, ty
davout: also i'm getting a "Flushing wallet.dat" after each eatblock, eats ~50ms each time
ben_vulpes: davout: yup
ben_vulpes: because it has to keep track of the inputs for all of the addresses it made ten minutes ago
asciilifeform: http://btcbase.org/log/2016-12-30#1593841 << it's part of the not-being-prb business, not to foist changes that have ~any~ potential sharp edges on operator☝︎
a111: Logged on 2016-12-30 19:04 davout: asciilifeform: why is there a specific -caneat flag? is there something specifically dangerous about eating blocks?
davout: asciilifeform: you mean it files rough edges off every single time?
asciilifeform: eatblock is a specialist tool
asciilifeform: and yes, it is only accessible via rpc anyway
asciilifeform: but i saw no reason not to give it a red flip cover.
davout: right
davout: ok, ~/blox/ass_to_mouth.rb is working, we'll see how that goes
mircea_popescu: nice.
asciilifeform: davout: why do you have a prb node, out of curiosity?
davout: because createrawtx et al.
asciilifeform: waiwat
asciilifeform: wassat
davout: what do you mean? i create transactions from arbitrary unspent outputs, sign them, and broadcast them
asciilifeform: ah
davout: hence my previous questions about the state of this particular functionality in trb
mod6: working on it :]
mod6: well, was, anyway. once the new changes for V are complete/tested/released, will be back on it.
davout: yeah, ben_vulpes told me in your very chan, if i can help it i'd be happy to, it does sound like a pretty good starting point for me to hack on trb
davout: "scratch your own itch"
jurov: http://btc.yt/lxr/satoshi/source/src?v=makefiles << mod6 asciilifeform ben_vulpes
davout: pretty much the only thing i personally need to be able to rm -rf all traces of prb from my boxen
mod6: well, ... feel free. but i think the coding part aside, which isn't going to be horribru, since a lot of it is backport anyway. but the testing is gonna be gnarly.
ben_vulpes: ty jurov
mod6: and am going to try to build tools, if needed, to help test this.
davout: mod6: i think it would actually be the least painful part to test
mod6: really? why do you think so?
ben_vulpes: mod6: "test[ing] this" is actually how i got on the alpha centauri miner quest
davout: try to craft a bunch of transactions, sign them, it either works or doesn't work, testing this functionality doesn't seem to depend on a lot of external, hard to reproduce, state
davout: s/state/context/
ben_vulpes: create and sign at least may be testable via the boost testing framework that's already in place
davout: yeah, that's something that seems to me pretty easily testable in a "isolated unit tests" way
asciilifeform: http://btcbase.org/log/2016-12-30#1593869 << pretty neat, ty jurov .☝︎
a111: Logged on 2016-12-30 19:25 jurov: http://btc.yt/lxr/satoshi/source/src?v=makefiles << mod6 asciilifeform ben_vulpes
ben_vulpes: http://btc.yt/lxr/satoshi/source/src/test/transaction_tests.cpp?v=makefiles#0004
ben_vulpes: davout: ^^
mod6: those are not really unit tests, those are functional tests. but yeah.
asciilifeform: ben_vulpes: this won't , as i understand, help him, he wants to ~craft~ tx, not merely broadcast-raw
mod6: i think over all it's a decent approach. have some pre-crafted transactions, and see how it goes. this is minimum. i wanna make sure we don't just capture "happy-path" but, all edge cases too.
ben_vulpes: davout's a rubyist, don't expect rigor in terminology from him mod6 :P
asciilifeform: as in, y'know, the thing that wallet ~ought to have done from day 1~
asciilifeform: instead of the 'accounts' and 'wallets' idiocy
mod6: no room for error here, lest someone sends all their coins out as a large fee, or some crazyness.
mod6: anyway! glad to have the help, and the experience from someone who uses this end of bitcoin quite a bit.
mod6: :]
ben_vulpes: asciilifeform: sorry, what?
mod6: we'll be discussing more in the near future i do suspect, Sir.
asciilifeform: mod6: error can be tolerated in ~autopilot that user can disable at all times~, i.e. it ~recommends~ a tx, user can review before firing
asciilifeform: ben_vulpes: as i understand, davout was asking for sane-wallet, rather than merely raw-tx-hopper
mod6: im not fixing the wallet, in this case, ftr. i'm just putting in the ability to create and send a raw tx.
mod6: and some other tools like 'listunspent' etc.
ben_vulpes: asciilifeform: he's still digesting "how to cut the wallet", let him ask for the things for which he's going to ask
asciilifeform: mod6: what means, in this case, 'create' ?
mod6: eh. mis-spoke kinda.
mod6: "build" a rawtx by hand, send it.
mod6: anyway, im just working through the beginning stages here. so im certainly not an expert on rawtxns
mod6: <+asciilifeform> mod6: error can be tolerated in ~autopilot that user can disable at all times~, i.e. it ~recommends~ a tx, user can review before firing << umm, i dunno about this.
davout: asciilifeform: i'm simply after the functionality of crafting raw txes from an arbitrary of outputs that *I* select, not wallet functionality in the sense of letting the system work out the details of "send X bitcoin to Y address"
davout: s/arbitrary of/arbitrary list/
asciilifeform: mod6: 'create a tx' is np-complete (knapsack problem) so you can potentially end up with strange solutions. user MUST approve before firing.
asciilifeform: the current behaviour is nuts.
mod6: ok. i didn't grok your sentence above.
asciilifeform: but the probability of 'txtron suggests 'send all money to karpeles' or 'send a million btc as fee' ought to be 0.
mod6: i read 'errors are tolerated'. and freaked.
davout: asciilifeform: my opinion is that the system doesn't even have any business *attempting* to select which outputs should be spent, let the user plug whichever system he wants on top of the low level "raw tx from arbitrary inputs" tool set
mod6: yeah, a valid rawtx is valid, but yeha, should be approved, somehow by user, before sending.
asciilifeform: davout: by all means it oughta have manual knobs.
davout: i content that these should be the ~only~ knobs at trb level
asciilifeform: but there is no reason i ought to have to enter 8 decimal points BY FUCKING HAND 10,001 times to make a tx.
asciilifeform: there ~will~ be error.
asciilifeform: if i have to do that.
asciilifeform: and error in ~fired~ tx is intolerable.
davout: script it on top of trb, don't integrate it directly in there is what I think is the correct solution
asciilifeform: so, now what, the thing drags perl along with it into eternity? python ?
davout: i didn't say it had to come *with* trb
asciilifeform: if trb is not usable NAKED, it ain't trb !
mod6: <+jurov> http://btc.yt/lxr/satoshi/source/src?v=makefiles << mod6 asciilifeform ben_vulpes << cool! thanks
asciilifeform: not a ~reference~ !
asciilifeform: we had this 'lose the wallet!' thread.
asciilifeform: before. several times.
mod6: <+davout> mod6: i think it would actually be the least painful part to test << anyway, i hope so. im sure there will be more discussion in coming months.
davout: such a setup ~is~ usable naked
asciilifeform: davout: how ?
asciilifeform: unless i misunderstand, you suggested removing functionality that ~was~ there in 2009
asciilifeform: in favour of something yet to be written.
davout: in the same way a gun is usable "naked", just don't point it to your face!
davout: the "let program select outputs to spend" half works half of the time, like you said "knapsack problem"
asciilifeform: my contention is that a trb with entirely removed unspent-selector is not usable-naked.
davout: and why not? because dangerous?
asciilifeform: there has to be a basic mechanism where the thing can be used, in anger, 2009-style, sans perl/python/etc.
asciilifeform: davout: no, because i'm not about to calculate ecdsa by hand with pencil on grid paper.
mod6: i don't think that should be removed. i think that the user aught to have the option to select them if he wants, with rawtx.
asciilifeform: option -- yes.
asciilifeform: raw tx hopper -- also yes
asciilifeform: removal of old grandfather's pistol -- no.
asciilifeform: unless there is a clear and fully-capable replacement.
davout: imho a "warn-if-insane-fee" config knob is largely sufficient, and would allow removal of the "output selection" nonsense from the code
asciilifeform: this is what distinguishes us from the beasts of the fields, folx.
ben_vulpes: you two are using "output selection" to mean two different things
asciilifeform: what distinguishes surgeon from butcher.
ben_vulpes: one of you is using it to describe the process of selecting signable unspent transaction outputs and another using it to describe the new outputs created
mircea_popescu: davout it is a good starting point yes.
ben_vulpes: at issue here is "where do the coins go" and "how to select the utxos to sign"
davout: utxos to sign -> provided by user
ben_vulpes: davout: and you get this list of utxos how?
davout: trb being able to list utxos given a bunch of addresses would be pretty obviously needed
mod6: listunspent
ben_vulpes: loya
davout: but then we're going down the bitcoinfs rabbit hole
asciilifeform: i dun think 'listunspent' is escapable, no.
ben_vulpes: with the current mechanism you'll have to import those addresses and rescan some amount of the blockchain to find the utxos you want
ben_vulpes: index with symlinks on address too why not
asciilifeform: why? if you have the privkey, every incoming valid block is inspected for tx pertaining to $addr
asciilifeform: even in the oldest trb.
ben_vulpes: today, yes
asciilifeform: and in any sane future trb.
ben_vulpes: any sane trb that doesn't index tx on output address i suppose
davout: yeah that's pretty much the point
davout: indexing based on address seems the sanest to me
ben_vulpes: wallet boils down to 'index of txen paying to addresses i care about' anyways
asciilifeform: index'em however you like, if new blocks aren't inspected for pertinent-to-me tx, the thing's a turd
asciilifeform: there is 0 reason why any extra processing ought to be needed for this.
mircea_popescu: http://btcbase.org/log/2016-12-30#1593945 << this is the wrong approach. you want to not shit in soup, not to filter shit from soup prior to serving.☝︎
a111: Logged on 2016-12-30 19:43 davout: imho a "warn-if-insane-fee" config knob is largely sufficient, and would allow removal of the "output selection" nonsense from the code
asciilifeform: just walk the new blocks. as is done now.
davout: such index allows complete excision of wallet functionality, and reduces privacy concerns should your node somehow get buttraped
ben_vulpes: 'inspected for pertinent-to-me tx' is a subset of 'index blocks sanely'
davout: mircea_popescu: ok, "forbid-insane-fee" then
asciilifeform: davout: if your node is raped, anything that is fed into it is also visible to enemy, and you have 0 privacy of anything but (possibly) privkeys
asciilifeform: and that's supposing you are meticulous about using 'diode' etc.
mircea_popescu: http://btcbase.org/log/2016-12-30#1593954 << ben_vulpes is going that way from my understanding.☝︎
a111: Logged on 2016-12-30 19:46 davout: trb being able to list utxos given a bunch of addresses would be pretty obviously needed
davout: asciilifeform: privkey privacy is pretty much all that matters imo
asciilifeform: davout: which pubkeys you watch is also something enemy has 0 business knowing
mircea_popescu: there is that.
asciilifeform: even if not as catastrophic as privkey leak
davout: i might very well broadcast other folk's txes from my node, just as well as i might broadcast my own txes from arbitrary shitnodes
ben_vulpes: which is why index the whole blockchaaaaaain
davout: asciilifeform: ben_vulpes has it
ben_vulpes: fine i'm going to go scream in a corner where i'm sure i'm the only one listening
asciilifeform: this is actually one of the reasons i insisted on eatblock and dumpblock
mircea_popescu: is there something being discussed or we just shootin da breeze ?
asciilifeform: it lets you have airgapped nodes.
asciilifeform: mircea_popescu: as i understand, thread was originally about 'sane wallet mechanism'
ben_vulpes: mircea_popescu: we're cramming a year+ of wallet into davout's head now that he's paying attention again
davout: haha fuck you
mircea_popescu: yeah just parser failed to return anything in the $controversy construct
ben_vulpes: davout: cheeky
phf: everyone sensing there ought to be a fight, but everyone's agreeing
davout: anyway, i guess my position basically boils down to: "as far as trb proper is concerned, best wallet is no wallet. but sane indexing mechanisms"
ben_vulpes: ^
ben_vulpes: hey, the power rangers wanted 20mb blocks for whatever reason. i want indices.
mircea_popescu: davout this is pretty much agreed upon, provided you mean by the words what we mean by the words.
davout: let's elaborate i guess
davout: currently unspent outputs are indexed by address
mircea_popescu: and incidentally "knapsack" problem is a fucking overstatement. here's a very simple strategy : 1. sort available inputs by size ; 2. if current step != last step, select first input that is smaller than tx going out else select the input right before that ; recurse to 1.
mircea_popescu: user defines number of steps and that's it.
davout: trb should be able to shit list of unspent outputs, optionally filtered by address
mircea_popescu: no, actually, trb should apply the above scheme EXACTLY like how v applies patches : you populate a wot with acceptable addresses
mircea_popescu: which may be "all" or a subset at your option.
davout: crafts desired transaction therefrom, insert in mempool ~fin~
mircea_popescu: right.
mircea_popescu: well technically spits it out, it shouldn't insert by itself.
mircea_popescu: for one thing it may not have authority to sign tx.
ben_vulpes: still needs signing, yeah
davout: mircea_popescu: yeah, i meant it as a separate, user-initiated step
mircea_popescu: right.
davout: maybe you want to broadcast that transaction from mit prb nodes, up to you
davout: no more "oh, but the transaction you're attempting doesn't match min fee $magic_number"
mircea_popescu: right
mircea_popescu was pretty impressed with davout 's bitbet liquidation tx.
davout: no, but seriously, how many times did I beat prb into crafting txes it would give me unwarranted opinions about
davout: that ended up confirming just fine
mircea_popescu: aha.
davout: and equipped with such sanity, if your transaction doesn't confirm, double spend it, no big deal
mircea_popescu: anyway - go, code.
davout: no more "oh, but this transaction is already in my mempool, sfyl"
mircea_popescu is going to start recommending people go to barren islands after all. turns out boredom is promotive of sanity.
davout: i want this transaction broadcast, just fucking drop whatever's conflicting with it
asciilifeform: davout: the entire attempt to mechanically distinguish 'double' from normal spends is an evil prbism
asciilifeform: and Must Die
mircea_popescu: no dude! you wand some acid?
asciilifeform: a doublespend is, in sane planet, STRICTLY attempt to spend coin that was already spent IN A BLOCK
asciilifeform: not in mempool soup.
asciilifeform: and in the block case -- yes, mechanically rejectable.
davout: latest example of transaction "that shouldn't have confirmed": e73d40c1aa9147e426de43d64753c2318c234426f6efbd090a7a9313d87f95e6
davout: a couple of days ago
asciilifeform: davout: why 'shouldn't have confirmed' in this case ?
davout: ended up just perfectly sent to children in uganda
davout: asciilifeform: because it apparently didn't match whatever prb thought was a "minimum fee"
davout: nevermind that this transaction not only carried a fee, but also resulted in a net cleanup of the utxo set
mats: so much log
asciilifeform: http://btcbase.org/log/2016-12-30#1594027 << this is only half of the headache. the behaviour of heathen nodes, who think that they already have 'your' tx, and the new one is 'doublespend', is the other half.☝︎
a111: Logged on 2016-12-30 20:04 davout: i want this transaction broadcast, just fucking drop whatever's conflicting with it
asciilifeform: and those -- must be exterminated.
asciilifeform: because they are a malignancy.
mats: asciilifeform: best part is the tacit omission that usg can be owned by 6 year old with five bitcents to rub together
mats: lulululul
davout: asciilifeform: this ended up never being a problem in practice
asciilifeform: let'em live in the wild, with the wolves, with prb segshitness etc.
asciilifeform: davout: ahahaha recall the bbet shitstorm ?
davout: i'm not remembering until my lawyer is around
mircea_popescu: mats was that a quote ?
asciilifeform: lel
mats: mircea_popescu: no, i wish
mircea_popescu: irconfused.
mats: er, admission*
asciilifeform: davout: for whatever reason, there exist miners who SIT on a tx, right until the very millisecond that they see a 'doublespend' OF it, and then IMMEDIATELY mine the ~first~ one.
asciilifeform: why -- to this day i do not know.
phf: ftr, least i somehow become sql acid proponent, i'd like to point out that i'm the only person running tmsr infrastructure ~not on a sql database~
asciilifeform: somehow, this profits somebody, somewhere, or is perhaps a side-effect of some other idiocy.
asciilifeform: phf: i thought your logtron was fed by a standard db..?
phf: no wai
asciilifeform: by what, then
davout: asciilifeform: weird
asciilifeform: davout: they'll pick up high-S tx also, and sit on then RIGHT UNTIL you broadcast a doublespend with correct chirality
asciilifeform: and ~immediately~ fire the malleated one
asciilifeform: *on them
asciilifeform: i do not know why this is done, nor have any plausible hypothesis. vermin do what vermin do.
davout: since vermin can hardly be completely exterminated the correct approach seems to be "don't go live in sewer"
ben_vulpes: phf: i think this is pretty neat, have wondered about your x-referencer etc
asciilifeform: davout: atm we are at the medieval tech level where we have -- afaik -- nfi how to live without lice, fleas.
asciilifeform: miners.
asciilifeform: i, for one, would love to discover how.
davout: why would a lord want to live without at least ~some~ peasants around?
phf: asciilifeform: a log is just push-vector with checkpoints. entire thing barely takes up 200mb of in memory data
asciilifeform: theoretically a 'will go in node xxxxxxx --- yyyyyyy inclusive or NEVER' field in tx, would have been sane. but it is too late, this is not in bitcoin.
mircea_popescu: asciilifeform there is no such thing as "immediately mine".
asciilifeform: *in block
mircea_popescu: to mine a block you must have decided what it contains a long while in advance.
phf: sbcl right now is at 450mb on average, which drops to 270mb after full gc
asciilifeform: mircea_popescu: of course not. it gets disgorged.
asciilifeform: so presumably was mined 'in advance'.
asciilifeform: how ? i do not know.
asciilifeform: presumably the 'withholding algo' discussed earlier.
mircea_popescu: phf your honor is not in question sir.
davout: http://btcbase.org/log/2016-12-30#1594075 <<< iirc it's half there, see "locktime"☝︎
a111: Logged on 2016-12-30 20:16 asciilifeform: theoretically a 'will go in node xxxxxxx --- yyyyyyy inclusive or NEVER' field in tx, would have been sane. but it is too late, this is not in bitcoin.
asciilifeform: davout: locktime is promisetronic!
phf: har
asciilifeform: and even if it weren't, it does the exact OPPOSITE of what i asked for.
davout: asciilifeform: isn't it actually enforced?
asciilifeform: davout: understand, it is 'enforced' by miner cartel ONLY
asciilifeform: anybody with a few mil. usd to burn could rent the hash tonight to thermonuke it.
asciilifeform: (probably even less, i have not crunched exact number.)
asciilifeform: the locktime thing is simply a hint that says 'usg-compliant miners, PLEEEZ dun mine this until block X'
davout: asciilifeform: so you're saying a block including a transaction with locktime > block height is considered valid by trb ?
asciilifeform: they can say 'fuckyou' tonight, if they like.
asciilifeform: yes.
asciilifeform: absolutely.
asciilifeform: we had this thread.
davout: iirc there were two *different* locktime 'features'
davout: one that was there since day 1, the other that was 'soft-forked' in
davout: i shall research
mircea_popescu: davout trb does not implement any of the prbisms. this means that ANY innovation included by the power rangers is a "while it lasts" thing, and building on top of it is setting one up for tears.
mircea_popescu: sooner or latter prb WILL be unwound. this is a certainty. just a matter of when.
mircea_popescu: this is regularily reiterated just for good measure.
asciilifeform: mircea_popescu has it.
davout: mircea_popescu: iirc there was one locktime thing that was there from day 1
asciilifeform: davout: it applied to nonfinal txen strictly.
mircea_popescu: but as far as software design and business risk planning goes, importing any prb means locking in a certain future loss.
davout: asciilifeform: howso?
asciilifeform: davout: http://btc.yt/lxr/satoshi/ident?v=asciilifeform_add_verifyall_option&_i=nLockTime
asciilifeform: the ONLY branches on it, are here: http://btc.yt/lxr/satoshi/source/src/main.h?v=asciilifeform_add_verifyall_option#0435
asciilifeform: carefully follow the logic.
asciilifeform: observe also, http://btc.yt/lxr/satoshi/ident?v=asciilifeform_add_verifyall_option&_i=IsFinal
asciilifeform: afaik all real-life tx are 'final'.
asciilifeform: ergo no trb node will ever reject a tx for reasons pertaining to 'locktime' garbage.
asciilifeform: nor a block containing said 'violator' tx.
asciilifeform: http://btcbase.org/log/2016-07-27#1510547 << prev. thread re subj.☝︎
a111: Logged on 2016-07-27 18:45 asciilifeform: thestringpuller: what EXTANT miners CHOOSE to mine, and what COULD be mined, if there were sane folks mining, are quite distinct things.
asciilifeform: and observe, http://btcbase.org/log/2016-07-27#1510563 , the prb idiots did not ever dare to introduce eggoging-on-locktime-violated. because that there'd be a phork☝︎
a111: Logged on 2016-07-27 18:53 asciilifeform: http://btc.yt/lxr/satoshi/source/src/main.cpp?v=0.10.0#0722 << still quite the same in prb 10 !!
asciilifeform: because the bomb ~could~ drop ~tonight~.
davout: so it appears I'm a fucking idiot here
mircea_popescu: this so rarely happens in #trilema
ben_vulpes hands davout a beer
davout: asciilifeform: this method returns false if nLockTime is > blockHeight http://btc.yt/lxr/satoshi/source/src/main.h#0436
davout: amirite?
davout: and this one here: http://btc.yt/lxr/satoshi/source/src/main.cpp?v=asciilifeform_add_verifyall_option#1268
davout: raises an error should one of the transactions return false for IsFinal ?
asciilifeform: davout: all tx that live in a block must be 'final' yes
davout: so a block doesn't pass AcceptBlock if one of the transactions has nLockTime > blockHeight
asciilifeform: nope, that condition only gets unmet if any INPUT is nonfinal
asciilifeform: AND the blocktime thing. it recurses.
asciilifeform: the fallthrough case is 'true'.
mircea_popescu: there should be a word for computer code that does the opposite of what it "appears" to be wanting to do.
asciilifeform: mircea_popescu: it is called, in this case, 'underhanded c'
mircea_popescu: !dwim
mod6: or typical
davout: asciilifeform: oic
asciilifeform: and yes, this crapola is typical
asciilifeform: 'see! we enforce it!'
asciilifeform: 'ummm looksy here'
mircea_popescu: typical-underhanded-do-the-opposite-of-what-it-seems-to-mean
davout: also, from reading the source, AcceptBlock is an entirely different beast than CheckBlock
mircea_popescu: davout yes.
asciilifeform: this is the liquishit 'softness' of 'softforks'.
asciilifeform: the ~appearance~ of rules, where there is only promises.
mircea_popescu: and if you are one of those who can summon the mental energy to convince yourself this [sort of thing] is accident rather than mens rea, you have my admiration.
davout: asciilifeform: this IsFinal method is quite weird, if nLockTime < blockHeight, the further checks on the txin are simply skipped
davout: maybe i'm missing the semantics of "transaction finality"
mircea_popescu: which is the type of thing alf is usually loud about, and all thinking men concerned :
mircea_popescu: this sort of thing allows nonconforming txn to be put in!
asciilifeform: aha.
mircea_popescu: including eg btc creating txn
mircea_popescu: here's an extra mn
asciilifeform: that's the end of the line, where this train goes, yes.
davout: in all fairness, some of this stuff might also have been fixed later on
asciilifeform: davout: see a few min ago. link. it was not touched in prb10.
asciilifeform: they daren't. yet.
asciilifeform: because it's '3rd rail.' they change ~any~ semantics, and -- bang -- forkable.
asciilifeform: don't think enemy doesn't know which wires are hot. he -- knows.
mircea_popescu: none of this stuff CAN be fixed.
asciilifeform: ^
mircea_popescu: that's why the "pistols" discussion ends up where it does
asciilifeform: also if you'd like more joy in life, look at where 'nBlockTime' comes from.
asciilifeform: (spoiler: it comes from your system clock +/- the voodoo delta that trb comes up with using peers)
asciilifeform: arbitrarily, the comparison is only even performed if the magic is below LOCKTIME_THRESHOLD ( http://btc.yt/lxr/satoshi/source/src/main.h#0042 )
asciilifeform: otherwise interpreted as epochtime.
asciilifeform: but at any rate, this threatrical blinkenlight assemblage is simply to distract from the fact that the check IS NOT ENFORCED!
asciilifeform: it is textbook heartbleediste misdirection.
mircea_popescu: so it is.
asciilifeform: or, far earlier, basic stage magic.
asciilifeform: the exact equivalent of meatspace circus sleigh-of-hand.
asciilifeform: *sleight
asciilifeform: (for n00bz) http://btcbase.org/log/2016-12-30#1594162 << it is important to actually go the gedankenexperiment, in one's mind, and understand why it cannot be fixed.☝︎
a111: Logged on 2016-12-30 20:45 mircea_popescu: none of this stuff CAN be fixed.
asciilifeform: as soon as you touch the hot wire, you now have a 'schrodinger's blockchain'
asciilifeform: that may or may not get forked at a particular time, but now ~is~ forkable, leaving you and anyone dumb enough to use your patched btctron on the -- almost certainly -- losing end.
asciilifeform: (specifically, a node where somebody touched the hot wire, will accept blocks that other, traditional nodes, will not. and/or reject blocks with they ~will~. definitionally - forktronic.)
mircea_popescu: this is a best case view not supported in practice (by which practice we mean the repeated etherape, symbolic as it is of the chances of the premier science and technology institution in the world in front of a loose assemblage of things that don't, supposedly, exist.)
asciilifeform: it is best-case. average-case will also involve a forest fire that singes ~everybody.
asciilifeform: but it is strange to speak of an 'average case' for phorkwarz.
asciilifeform: we haven't really had many of'em, to learn from.
mircea_popescu: no, actually - a forest fire that singes every ~law abiding~ participant and them only.
mircea_popescu: that "and them only" trailer is mostly why we haven't had the pleasure. just as soon as hitler figures out how to remove it, he WILL burn his citizenry into a crips, as he always does.
mircea_popescu: but hey, citizenry dun wanna believe.
asciilifeform: mircea_popescu: hitler dun wanna settle for 'burn own citizenry', that's trivial already. he wants to dekulakize mircea_popescu et al.
asciilifeform: drain the battery, if you will, that mircea_popescu et al charged.
asciilifeform: it is how hitlers are powered, after all.
asciilifeform: drainin' other folxs' batteries.
mircea_popescu: so far the winds are not too favourable. consider the situation in the field : hitler has ~exactly one trick~, and it is the following trick : you know i shall fall, and i know i shall fall, and we both know we both know, but here's the thing -- the market can stay liquid for longer than you can stay solvent, especially if the liquid is liquid shit from my liquid shit pump. so come, take positions on my sotck exchange, reflec
mircea_popescu: tive of economic reality, why not.
mircea_popescu: to which mp reacts with http://trilema.com/2014/an-era-ends-today-a-new-era-starts-today/ ; which is notable in the form, but not in the result. hitler gets to do his usual manipulation, and "win", but a) hitler is stuck denying the venue where he won exists, and this hurts him so bad he spends a year trying to get kakobrekla to eventually agree "fiat is a better deal" and whatever other "modern science & democracy" nonsense
mircea_popescu: ; and b) that it happens when mp actively says.
mircea_popescu: then hitler tries to make "his own bitcoin", and pumps the battery, which mp drains at his leisure. but it is HIS, hitler figures, so he changes it. and changes it. each subsequent change resulting in erosion of his grasp over the battery in question. the wind blows the other way.
asciilifeform: the exact moment when hitler runs out of furniture to burn, is ~unknowable
asciilifeform: and so various folks, who oughta know better, bet on him.
mircea_popescu: mp can afford to lose however much mp volunteers to lose ; the exposure profile is not symmetrical, hitler keeps having to go all in.
mircea_popescu: moreover, the strategic profile is not symmetrical : the initative is strictly not with hitler.
mircea_popescu: and so here we are.
asciilifeform: dunno that hitler ever goes 'all in', the moneys (however counted) spent on, e.g., haskell, dwarf the ethertard lab budget 100+x
mircea_popescu: the "money" is not a point of interest here.
mircea_popescu: the currency of the reich is, as the original hitler well observed, a sort of sweat.
asciilifeform: easily 1000x the sweat went to haskell.
asciilifeform: (and 100,000x -- to rubyism and other unthinkable rubbish)
mircea_popescu: but in any case - ethertard budget exceeded 50mn in 2016 ; and these are turkey dollars. this is more than the aggregate expenditure across all software branches in the united states in that year.
mircea_popescu: asciilifeform mind that chickens do not sweat.
mircea_popescu: nor dogs (except for their tongue).
asciilifeform: academitards get paid in turkeydollar?! where?
mircea_popescu: asciilifeform the loss was extracted in the field of battle, and came as actual, isis-equipping dollars.
mircea_popescu: the sort that pay for the bullets that cut holes in retarded us-born kids' heads.
mircea_popescu: (to be perfectly clear - most of the eth budget went to propping the exchange rate, nothing else.)
deedbot: http://trilema.com/2016/disgrace-dont-the-dogs-get/ << Trilema - Disgrace - Don't the dogs get
asciilifeform: mircea_popescu: would that be in actual btc-that-usg-somehow-had ? or in 'could-have-hads' (riaa-style accounting..?)
mircea_popescu: whence it comes is not really my concern, for all i know it is running a charity on mars for poor usian children.
asciilifeform: imho actual moneys are in different conceptual category than coulda-shoulda-woulda-moneys, but perhaps that's just me.
asciilifeform: in other lulz, latest visitors to my www, and then to nosuchlabs: employees of Wolfram Co.
mircea_popescu: the dispositive criterion for the matter is the counterparty function.
asciilifeform pictures herr wolfram reading the article about him, and going 'neinneinneinneiennein!1111, banging head'
mircea_popescu: lol
mircea_popescu: carefuly, you'll get unhappened.
asciilifeform: but last i heard there's 300+ bodies in there
asciilifeform: so not necessarily Herr Doktor
asciilifeform: d00d has own cult, 'with blackjack! and hookerz!'
mircea_popescu: there's worse fates.
mircea_popescu: hopefully he doesn't decide to fuck any of the interns or anything.
asciilifeform: and, unrelatedly, asciilifeform reluctantly admits that (american! early 2000s) 'cougar' game joystick is mighty fine. 1:1 sized replica of 'f16' controls, everything made from iron, even buttons. quite pleasantly surprising. almost like piece from different era, when even toys were made honestly.
asciilifeform: historically accurate to the point of wtf -- even radio controls (what good are they in a sim?) are there.
asciilifeform: thing shows up as ~three mice when plugged in!
asciilifeform: mircea_popescu: i bet herr dokror w fucks baby pandas , and gets away with it.
asciilifeform: he's that kind of 3111117 d00d.
mircea_popescu: i've yet to find a hitler servant that fucked anything.
asciilifeform: mircea_popescu: i thought that normally they fucked little chillenz or similar
asciilifeform: (or, less commonly, household beasts)
mircea_popescu: world of warcraft much more common.
asciilifeform: that's for the young set
asciilifeform: ( 'supernumerary children' (tm) (r) ( mircea_popescu's articles ) )
mircea_popescu: tjhe
mircea_popescu: they're all supernumerary, and chyldren.
mircea_popescu: chyldren lol. children
asciilifeform: dyver∫e ∫orte∫ of chyldren!
deedbot: http://trilema.com/2016/disgrace-well-youre-welcome/ << Trilema - Disgrace - Well, you're welcome
ben_vulpes: in other eurolols https://www.youtube.com/watch?v=C12F4MM71q0
mircea_popescu: are they adopting the india currency plan yet ?
asciilifeform: ben_vulpes: there is context, 'kindling' is the catchword in ru penal code for 'hatespeech' ('kindling discord between nations')
mircea_popescu: so is this putin-supported or what ?
asciilifeform: in this film, 'we will kindle, fuck you, we will kindle, they will knife us and we will live, lands of the ancestors, etc, etc'
asciilifeform: mircea_popescu: not overtly.
mircea_popescu: aha. seems pretty clunky.
ben_vulpes: baaad videography
mircea_popescu: what's with the horses anyway ? don't tell me the russkis think themselves don cossacks or somesuch.
asciilifeform: nope
asciilifeform: that'd be ukrs.
mircea_popescu: irk?
asciilifeform: horses are just generic positive picture.
asciilifeform: https://www.youtube.com/watch?v=jc-KpbX-6w8 << better clip on same theme.
asciilifeform: ^ 'it means that soon, war'
asciilifeform has no ready translation , but imho ^ is 'best of genre'
mircea_popescu: eh, war.
mircea_popescu: they may butcher some random "foreign" dorks, especially if they're defenseless in an urban setting.
asciilifeform: http://amdm.ru/akkordi/kontrrevolyciya/137409/eto_znachit_chto_skoro_voyna/ << lyrics.
mircea_popescu: oh, counterrevolution no less.
mircea_popescu: fucking tedium.
asciilifeform: that was just the band's name.
asciilifeform: 'Если русский мужик в безысходности пьёт, / Если наглый джигит как добычу берёт / Дом отцовский его, и сестру, и жену — / Это значит, что нам объявили войну. / Это значит, что скоро война.' << sufficient sample.
asciilifeform: anyway, mentioned strictly because ben_vulpes unearthed .
asciilifeform: ( i did not even know he spoke ru )
ben_vulpes: don't
asciilifeform: then how the fuck did you pick up this rubbish
ben_vulpes wiggles eyebrows
asciilifeform: stuck to your shoe?
mircea_popescu: he's added a third to his harem, she came in the mail.
mircea_popescu: i mean, she arrived in the mail. she came everywhere else.
asciilifeform: ah then! then.
ben_vulpes: never seen anything like it in my life
mircea_popescu: get out of here ?!
asciilifeform: ben_vulpes: these 'motherland cried TO US, HER SAVIOURS!' clips are a dime a dozen.
asciilifeform: as mircea_popescu correctly said -- snoar
mircea_popescu: wtf do you do all day in cascadia, not buying drinks for artistic adolescents ? what is their life work, not this crapolade ?
ben_vulpes: mircea_popescu: buying drinks is one thing, girl soaking everything in sight less common
mircea_popescu: this is why you get a russian, gets the soak done.
asciilifeform: !#s zcash
a111: 17 results for "zcash", http://btcbase.org/log-search?q=zcash
asciilifeform: http://spectrum.ieee.org/tech-talk/computing/networks/the-crazy-security-behind-the-birth-of-zcash << quality circus material, via jurov .
ben_vulpes: that's the one with the magic secret data that peter todd nominally destroyed?
asciilifeform: aha, that one
asciilifeform: with great fires! and lights! and mega-persuasive camera crews!
asciilifeform: and audience!
asciilifeform: etc
ben_vulpes: so trust
ben_vulpes: moon landing of cryptocurrencies
asciilifeform: wtc-burning of currencies
ben_vulpes: ooh ooh or the ruskies who burnt hundies on camera for the transfer back to us treasury!
asciilifeform: or them
asciilifeform: no shortage of these
asciilifeform: or the jew from mircea_popescu's old tale, who dropped a cheque into a grave
asciilifeform: and asked for change
mats: http://thebulletin.org/evidence-shows-iron-dome-not-working7318 kinda old news, but for future reference, CIWS and such appears to be 90% propaganda 10% effectiveness
mats: idk about the claims re: rocketry but the insurance bit looks credible
mats: which has yet to be explained by .is even now
asciilifeform: mats: iirc thing ~worked against primitive (17th c.-style, unaimed) handmade orc rockets
ben_vulpes: can't exactly light the computer-controlled machine gun off at inbound targets over heavily populated areas
asciilifeform: ben_vulpes: thing used small rockets, neh ?
asciilifeform: rather than cannon
asciilifeform: the kind with autodestructors
mats: yes, there is a proximity fuse that disperses shrapnel
mircea_popescu: hey, they didn't say steel dome yes ? iron, sorta-worx.
mircea_popescu: "modern" warfare, like modern everything, is 90% gargle and posturing.
mats: seeing as how hamas is mobile and the dome is static, i don't see how it could possibly work 9/10 times like .is claims
mircea_popescu: who knows, maybe enemy is composed of sufficient % of modern folks, the sort that live life of the mind, and believes.
ben_vulpes: asciilifeform: what i mean to say is that it's about the only thing one could deploy around that many folks, is something that goes blooie in the air rather than a zillion rather-fast bullets most of which will come back down at lethal speeds
asciilifeform: or at the very least get the disbelievers into that one spot where the old machines work, and shred'em
ben_vulpes: the only actual anti-rocketry defense being "don't be anywhere near targets of interest"
asciilifeform: ben_vulpes: nah
asciilifeform: ben_vulpes: the actual defense is 'kill'em all'
asciilifeform: 'then salt the earth'
ben_vulpes: amusingly, .is actually wrote the book on rocketry-resistant fleets. turns out -- many small boats. hard to do that with the holy tent.
ben_vulpes: mats: does the dome at least handle steep ballistics decently?
ben_vulpes: eg mortars, that anti-ship device that pops up and comes right back down
mircea_popescu: ben_vulpes "many small boats" is shit for unrelated reasons.
ben_vulpes: wyrdmantis: ffs get a bouncer
ben_vulpes: i don't want to hear about your laptop status omg
mats: ben_vulpes: i expect iron dome actually only works effectively against mortar rounds
asciilifeform: ben_vulpes: often i wonder what the remoras are even doing in #t. why can't they read l0gz like normal peoples.
ben_vulpes: heh
mats: if only usg published c-ram data from iraq
mircea_popescu: ahaha don't be ridiculous.
ben_vulpes: mats: usg and data in the same sentence, hyuuu
mircea_popescu: they can start with the m3, any time they grow a pair.
ben_vulpes: i was making an m3 joke too! cool!
ben_vulpes: howabout u6
ben_vulpes: anyways, tty'alll
asciilifeform: m3?
mircea_popescu: monetary mass.