242400+ entries in 0.152s
Framedragger: cputime per process logging may help
to settle
this for future cases :p
mircea_popescu: also at issue is something called "fastwerker".
that
the same
thing ?
Framedragger: (how can
they even see
the name? kvm? i
thought it was baremetal?)
mircea_popescu: and unless maz actuallty reads
the logs instead of doing his work, I
TOLD YOU SO
mircea_popescu: asciilifeform "process fastwerker is killing
the box, would you like reboot"
Framedragger: asciilifeform: of course. and also don't
treat
this as high priority, i prolly wont look at it before wednesday anyway. but would be interesting
to
take a look - and i'd
take a look
Framedragger: stop web app and other stuff, copy /var/lib/pgsql/data, start web app again, use data
to set up separate db.
Framedragger: i'm
thinking whether it'd be worth it
to just have a static replica of db-as-it-currently-is, for now. as in for "i want
to
touch data,
there's an outdated html file on loper-os i guess?" cases.
Framedragger: i'm sorry but you didn't convince me in regards
to
the 'amount' of data. > 100mil row postgres with > 100 gb of data in a 8GB ram server ran fine. and while phuctor may be a more demanding beast, shouting '5 mil keys, MILLION!' doesn't convince
Framedragger: i have no doubt
that a sillicon valley version would be a datacentre of mongodb nodes which constantly fail and corrupt data, and cost millions. :)
Framedragger: quite sure
the diff'ing / updates were
thought out
thoroughly, i.e.
time complexity is constant.
a111: Logged on 2016-11-19 16:58 ben_vulpes: Framedragger: if you go down
the 'phuctor visualizer/dash' route, you might consider leaning on pg streaming replication
Framedragger: asciilifeform: have you profiled an
http request
to a permalinkable phuctor page? where's
the bottleneck? curious if you could insert a
thing into flask which crawls
through everything and stores locally.
mircea_popescu: "oh it's on phuctor and why do i need
to do anything", which is how we got "those debian experts are flyeyeing
the code so why should i have a clue".
mircea_popescu is pretty fucking annoyed
that
the MOMENT
the slightest disturbance in
the force occurs, we suddenly discover
there was really 0 defense in depth.
Framedragger: (again, many phuctor pages will simply
timeout, iirc. but maybe can adjust; and still worth doing.)
Framedragger: by 'it', i mean scriba submitting
to archive.is
Framedragger: mircea_popescu: right, will get it done. i had started on it, got sidetracked by
the python encoding problem, and
the got sidetracked by other stuff. need
to re-trace, and will first do
the archival bit.
mircea_popescu: that someone utterly failed at
that
task, as push coming
to shove proves.
Framedragger: (i
thought someone was supposed
to retroactively archive all links in all logs of all
times?)
mircea_popescu: we fucking need
this archival of links in chan, like years ago.
Framedragger: asciilifeform: and
the permalink pages identifiable via fingerprint, are
they generated by
the flask backend,
too?
mircea_popescu: Framedragger your scriba could crawl all links in chan and archive
them. phf said a while back he almost has
this but i've yet
to see it
Framedragger: asciilifeform: why can't a separate box be set up
to just crawl
through all of phuctor pages, and
then determine which of
them are 'static' / won't ever change, for starters. and re-query
the dynamic ones (at least
the /phuctored) every $x amount of
time
mircea_popescu: ie, you don't get
to see ip or link. which /.... so search, what's it do.
Framedragger: that would indeed be nontrivial and would
take quite a bit of your
time
a111: Logged on 2016-11-17 15:06 mircea_popescu:
trinque can deedbot rss parsing be unprincipledly altered so
that any succession of alphanum characters in excess of 16 spaces is replaced with first4[...]last4 ?
Framedragger: that's
the question - will you have
to actually do
that.
Framedragger: mircea_popescu: just fyi archive.is fails and/or
timeouts on some pages
Framedragger: asciilifeform: (instead of doing a more obscure "query-able phuctor-and-stuff db"
thing i could help with some kind of phuctor-public-display-vps infrastructure setup. just sayin'.
thing's not clear in my head.)
mircea_popescu: fucking worthless, really,
there's a
total of SIX links
to phuctor
that went in chan
this month ?
mircea_popescu: and all-phyuctor snapshots are like
this : 1 from nov 14.
three from nov 3 .
two from nov 1.
Framedragger: no doubt
that's priority asciilifeform, and will patiently wait
Framedragger: i've read, i am doubly interested due
to vagueness of said broadcasts :p
Framedragger: with regards
to vps i could help, if help/hands are needed. i know you have other priority stuff asciilifeform. also, don't know what
the meta-priority level here is. (i.e., compared
to other projects etc)
Framedragger: and
then imagine, deploying 'display' vps would become simpler still.
mircea_popescu: any collection of data will have
to consist of references
to other sources at
the edges
Framedragger: those sibling pages, why can't
they hit once, and html be generated,
too.
a111: Logged on 2016-11-19 15:52 asciilifeform:
the folx whose chumpnet we blew open, with
the debian boxes, i suspect --
trying
to make
their displeasure known.
Framedragger: have a way (rss or better version of rss, or whatever)
to sync it every $n hours.
Framedragger: even if more
than
that - all of
that shit can be cached, static html pages. maybe i'm oversimplifying.
mircea_popescu: which is k of records * 10kb or such, not
the end of world.
Framedragger: i don't know why you need
true real
time,
tbh.
Framedragger: i'm not saying
that going for cheapest ad-hoc option is accetabru. just, a display box showing static content needs much less.