228800+ entries in 0.14s
phf: asciilifeform: no, it's a command
that you run, like REINDEX index_of_things; it simply queries what's already in DB and warms up
the cash
Framedragger: docs say would need
to get rebuilt only if
there were any unwritten changes. which
there shouldn't be as asciilifeform is not using write cache
phf: is it a hash index? it has
the least overhead (it isn't logged amongs other
things, so you have
to rebuild it on crash, but conversly it's kept in memory and only supports = operation) indexes will make your queries more cheap, but writes more expensive, so you want
to make sure it's
the cheapest possible
Framedragger: (and follow-up, does explain analyze show
the use of
that index)
phf: a sort of impolite question, but is
there's an index on hash column?
mircea_popescu: this is a piddly excuse, "no duplicate logic", case of luser with wwwform and case of 100k keys in batch form are different enough
to warrant duplicate logic.
that's why computers even exist,
to account for such level of difference in code.
mircea_popescu: yes, well,
that's
then
the problem.
they should go in as a single query
the size of
the batch, with
the items sorted within it
mircea_popescu: would you stop with
these bizarro deflections,
they neither impress nor persuade, but
they do give you an ugly image.
mircea_popescu: uh.
then why do you put
the keys in in batches if you're not... putting
them in in batches ?
phf: asciilifeform: right, i was going
to get
to
that :}
phf: i'm not even arguing with you, i'm saying
that
the ~full extent~ of what "move it
to psql" is going
to do is ~eliminate cross-boundary issue~
that is all. so it'll shave some significant overhead, but it's not a silver bullet.
phf: what i'm saying is
that a significant fraction of "1000s of queries AND ..." is
the cross-boundary. you compile queries on c side, you send
them
to psql, it
then parses, prepares results, serializes, sends it
to c side, c side has
to now parse all over again
phf: well, a more practical approach would be
to adapt phuctor c part
to a postgresql loadable module interface. in which case you he will eliminate
the cross-boundary overhead (serialize/deserialize over
the "wire").
mircea_popescu: it has ~some~ ability
to precompile your queries, which is somewhat like linking object code.
mircea_popescu: yes but it has
this convenient hole
through which you can go in, which is - implement bernstein IN sql.
mircea_popescu: which yes
takes some work, but not quite as much as
the other variant.
mircea_popescu: bitcoin wants its own fs. ANOTHER way, is
to use
the means
the db already offers for
this.
mircea_popescu: you're not addressing
the idea. currently you use a pile of c code you labeled for purely personal reasons "a db"
to store some data for you, and another pile, you labeled phuctor,
to bernstein and do other
things on
the db-stored data. because
the interface is
the bottleneck, it
then becomes clear you must merge
this. one way is
to merge by lifting
the db code and putting it into phuctor, making it you know, its own db like
a111: Logged on 2016-12-30 16:28 mircea_popescu: but he might be interested
to hear about it.
phf: asciilifeform: i'm just
trying
to establish
the dataflow here, for my own curiosity
mircea_popescu: and you do it as prepared queries, which get precompiled
to a degree
trinque: asciilifeform: a
temp
table is in RAM
phf: asciilifeform: oh so you do insert
to a set, every
time
there's a result, and you query for
the whole set before you start a cycle of process?
mircea_popescu: you implement bernstein IN
the db. it is actually a programming language.
mircea_popescu: though i am unaware anyone ever implemented
this ; because, of coruse, i am unaware anyone used
the guy's algo for any other purpose
than gawking.
phf: so you basically snapshot your entire dataset back into
the database at certain
times, and snapshot is an equivalent of set merge?
trinque: I am sadly, quite good at SQL if you want
the
thing
translated
trinque: might be faster
to do in
the db
mircea_popescu: you can also set bgwriter_lru_maxpages
to 0 and disable background writing altogether
mircea_popescu: eg bgwriter_delay you may want
to be set low for
this reason.
mircea_popescu: asciilifeform all
these are memory usage ops, what
they do is establish when it should go on disk.
they do not significantly affect cord-yank robustness.
there are other specs you can make for
the background writer for instance
that do.