log☇︎
15200+ entries in 0.009s
ossabot: Logged on 2019-10-04 11:59:44 diana_coman: all 3 loggers seem synced atm; ossabot is back too, though still waiting to find out wtf happened exactly.
mp_en_viaje: http://logs.ossasepia.com/log/trilema/2019-10-04#1940091 <<->> http://logs.ossasepia.com/log/trilema/2019-10-04#1940092 those be some pretty lulzy timestamps then
ossabot: Logged on 2019-10-03 22:28:19 asciilifeform: i find it mighty 'coincidence'(tm) that for year+ we get 24/7 pipe, and then put a logger and nao dulap-I treatment.
mp_en_viaje: http://logs.ossasepia.com/log/trilema/2019-10-03#1939920 << i don't get it, what is the coincidence ?
asciilifeform: along with errything else in that cage.
asciilifeform: ftr i did not put this in genesis because naively supposed that ordinary workings of tcp will in fact throw a connection if the pipe were to unplug. but apparently this aint so
asciilifeform has internal policy of not regrinding a patch for mistakes in comments; folx oughta read the actual proggy before connecting to live missiles
asciilifeform: 'To disable forced disconnects, set disc_t to zero' is wrong btw. will give actual off switch on next patch.
asciilifeform: i noticed last night that on none of the piz outages did the thing actually realize it's dead. hence this patch.
asciilifeform: tested just nao w/ this item.
asciilifeform: (at the very least, when live conn, will have pingism from fleanoad erry ~45s )
asciilifeform: diana_coman: the new one simply demands that ~something~ come down the pipe erry disc_t seconds. recc'd value 180 .
diana_coman: it did reconnect properly the times when it just lost connection; hm, the log I saw for when unreachable, it was trying to connect
asciilifeform: diana_coman: the old one worx a++ on 'organic' i.e. fleanode-initiated disconnects. but sadly not on pipe death
diana_coman: asciilifeform: thanks, I'll have a look; fwiw so far bvt's fix actually worked well for ossabot
asciilifeform would deploy nao, if had a working box to deploy to...
asciilifeform: if deploying, make sure to see readme & example config re knob.
asciilifeform: tested (by setting absurdly small interval; throws, reconnects, as expected; then by setting 3min -- operates normally, pings typically come erry 45s or so)
asciilifeform: ACHTUNG, panzers! logotron tree updated w/ detect_disconnect.kv.vpatch . << lobbesbot , diana_coman
mp_en_viaje: aite, ima brb writing article then
asciilifeform meanwhile wrote patch for bot to make it actually detect disconnects. is in staging box locally.
asciilifeform: mp_en_viaje: it's a small ddos, as they come, but compounded by the hamfistedness of piz upstream. BB is in the cage atm bazaring w/em
mp_en_viaje: what is this halfass cvasi-dos ffs, i thought we're talking srs bzns
asciilifeform: mp_en_viaje: nope. but iirc you're on a phased array of euro pipes, not individually routable to from outside also eh
asciilifeform: mp_en_viaje: seems like only folx regularly throwing ips in the log, affected.
mp_en_viaje: amusingly, muh thing here unmolested apparently, i got complete scrollback even
mp_en_viaje: oh, that old thing
asciilifeform: mp_en_viaje: summary : rotating ddos, not even limited to piz
BingoBoingo: mp_en_viaje: Got flooded, appears to have been one of the lurkers havesting uncloaked ips and shooting
mp_en_viaje: a ok, ima get to it then
asciilifeform: mp_en_viaje: whole thing in log ( diana_coman's and lobbes's, naturally ), it's reasonably compact
BingoBoingo: asciilifeform: This terminal is on a radiomodem atm
BingoBoingo: I have stopped irissi, weechat, and znc services on the shared machine. I am about to request the portion of the IP space not known to be running uncloaked bouncers is restored shortly.
asciilifeform: BingoBoingo: i thought whole cage were nullrouted ?
BingoBoingo: asciilifeform: ty in the basement on junkyard wars connection
asciilifeform: incidentally, anyone else think this is not entirely accidentally coinciding with the start of the festivities ?
ericbot: Logged on 2019-10-04 17:32:32 diana_coman: asciilifeform: uhm, how does that reasoning go? they fire less at my box because one box or what? anyways, is there at least any concrete communication from piz's data centre re what they are doing otherwise exactly?
asciilifeform: http://logs.ericbenevides.com/log/trilema/2019-10-04#1940177 << i oughta expand re this. ddos is cheap but not free, if the 1e6 or whatnot winblowz boxen were evenly split to piss into 20 diff addrs, would not amount to much effect. so traditionally rotates, e.g. 3min to 1, then to next, and so on, in circle.
asciilifeform: does give a little support to the hypothesis that original demasking signal was via irc.
asciilifeform: ( either that, or simply outlandish margin of spare pipe bought from last time when he ~was~ included )
asciilifeform: mp's www fwiw doesn't seem visibly affected. so prolly not included on the magick list.
asciilifeform: diana_coman: i expect that before this is through, we'll find out just what tonnage of liquishit the moldavian folx are able to digest
asciilifeform: BingoBoingo: plox to post the next despatch from latch in cleartext, i see no reason to whisper these
BingoBoingo headed to the rack
diana_coman: not that "oh, so currently not under fire because they are firing at piz"
diana_coman: asciilifeform: ok, but the point here is that moldavians managed to mitigate it while under fire.
asciilifeform: diana_coman: if latech is to be believed, it's a rotating ddos
diana_coman: asciilifeform: uhm, how does that reasoning go? they fire less at my box because one box or what? anyways, is there at least any concrete communication from piz's data centre re what they are doing otherwise exactly?
asciilifeform: diana_coman: that being said, i also suspect that yer box is only standing because piz is taking 90% of the fire
asciilifeform: diana_coman: moldavia has advantage of not being on monkey continent and not relying on single pipe to civilization, i suspect
diana_coman: dunno, moldavians managed to mitigate it sanely without unplugging my logger
asciilifeform: diana_coman: seems to be that 'unplug customer' is the only way anybody mitigates ddos.
diana_coman had to follow quite a few things at the same time during this last hr so didn't focus on pizarro specifically.
asciilifeform: diana_coman: indeed they did, i thought it was clear from last hr of #t
diana_coman: o.O they mitigated by unplugging the paying customer, I see.
asciilifeform: what condition they were in prior to this event, i do not know
asciilifeform: diana_coman: well since latech appears to have in fact pulled our plug, it is no surprise that they're unreachable ~nao~
diana_coman: asciilifeform: yes re piz house; I was just noting that those have nothing to do with cloaks and irc at any rate.
BingoBoingo: It seems to depend on how big the incoming crapflood is and how far upstream they have to go to find a sufficiently wide pipe to cut off the crapflood.
asciilifeform: even so, if incoming e.g. 50, that 50 oughta consist of a random sampling of the incoming. not 0.
BingoBoingo: asciilifeform: We have an asymmetrical pipe. The 200Mb/s is traffic going out of the rack
asciilifeform: BingoBoingo: i guess this is an illuminative experiment re the q of what it is we actually get for that ( pretty hefty, quite enuff to keep three bus-fuls of orcs in empanadas erry month ) pipe fee. turns out not 200Mb/s , but 'what we feel like aint too hard' ??
asciilifeform: whole thing is offline
asciilifeform: diana_coman: they're both in piz house neh
BingoBoingo: asciilifeform: We have to start from somewhere
asciilifeform: soo, open all but the actually used boxes , lol ??
BingoBoingo: asciilifeform: I trying to start by getting the portions of the range without known, uncloaked IRC bouncers opened up again. From there we continue trying to open things up.
asciilifeform: BingoBoingo: at the risk of repeating, what's yer plan for getting the pipe back ?
ossabot: (pizarro) 2019-10-04 thimbronion: fwiw my bouncer (running on non-pizarro box) was disconnected yesterday, and a few days prior, and although I have a cloak, it does not get applied before bouncer rejoins channels.
asciilifeform: and no, per the docs, not optional, i.e. the naked ip is broadcast ~unless~ you log in 'in one shot' via their ssliquishit.
asciilifeform: BingoBoingo: there is already such delay.
BingoBoingo: asciilifeform: It does appear the SSLism for cloaks is still optional, but a delay needs to be inserted between authentication and joining channels
asciilifeform: this supports BingoBoingo's hypothesis of 'shrapnel addressed to occupant' (vs 'bullet w/ name on it')
BingoBoingo: Well, now with cloaks attacker HAS to reveal that they are targeting
asciilifeform: it does seem that the box where 'uncloaked' lobbesbot connects from , was on the nuke rotation, whereas e.g. asciilifeform's local pipe -- not
diana_coman: ok but the problem is still there, only ...better hidden, no/
BingoBoingo: I am saying have cloak, ever so slightly more protected in theory yet substantially more protected in practice
diana_coman: BingoBoingo: are you saying that "have a cloak -> no problem" or what exactly?
asciilifeform: BingoBoingo: i still dun see the matter of 'from where' as settled -- it aint as if the addrs of tmsr www boxen are anyffin but public
BingoBoingo: I suspect what happened is a long stretch of relatively calm weather got a lot of people complacent on the matter of IRC cloaks.
BingoBoingo: The cloak is not perfect protection, but it does frustrate the lazy sort of attack we appear to have been struck by
ossabot: Logged on 2019-10-02 14:08:12 asciilifeform: http://logs.nosuchlabs.com/log/trilema/2019-10-02#1939628 << seems to lurk in #a too. perhaps the current enemy spy, i have nfi
asciilifeform: i suppose this could explain what these are for
BingoBoingo: The omissions strongly suggest uncloaked IRC connections drew the aggro
BingoBoingo: asciilifeform: "The attacks are moving on the range. It started with .97, then move to .99, then to .200 and so on." This supports lazy attack targeting uncloaked irc connections.
asciilifeform: and in turn, piz subscribers are also paying for pretend-boxen ?!
asciilifeform: BingoBoingo: what's the plan on latech co's side ? we pay'em and they give us a pretend-pipe ?
BingoBoingo: asciilifeform: I am presently connected through not a bouncer
asciilifeform: BingoBoingo: through what are you connected ? whole piz house seems to be unplugged
diana_coman: moldavians seem to be doing fine so far ie attack still on from what they say but not making it through.
asciilifeform: ( and seems like ddos is only half the problem , the other half is pipe vendor's immunocascade ? )
diana_coman: ddos attack seems to be.
BingoBoingo: asciilifeform: I don't see a simpler narrative for why diana_coman's bot went down too.
feedbot: http://thetarpit.org/posts/y06/0a5-work-plan-vi.html << The Tar Pit -- Work plan for M10 2019
asciilifeform: BingoBoingo: i doubt any of this has anyffin to do with cheapo lurkbots or any similar
BingoBoingo: <diana_coman> that's why I didn't really bother re cloak because the ip is public anyway << Even the most trivial barrier does something, like prevent lurkbots from harvesting addresses as they connect.
asciilifeform: atm that meter's fucking running but the taxi aint movin'!!
BingoBoingo: asciilifeform: Someone appears to be hopping around and hitting addresses. DDoS is an unsolved problem.
asciilifeform: so BingoBoingo what exactly are we paying these people for ? 'isp but if not too many packets' ?!
BingoBoingo: asciilifeform: Apparently the full block down due to flooding