This is the Aug 2019 draft of S.NSA WWW logotron and IRC bot kit,
as can be seen presently at http://logs.nosuchlabs.com/log .

To make your own installation, you will need:

(1) Traditional 'python' 2.7.
(2) 'flask' lib for (1).
(3) 'psycopg2' lib for (1).
(4) 'postgres' (9 or 10).
(5) A WWW server that knows how to proxy.

To use the kit, you will first need to create a user and DB, e.g.:

su - postgres
psql

create user nsabot createdb;
alter role nsabot superuser;
create database nsalog;
grant all privileges on database nsalog to nsabot ;

... you can take 'super' away from this user after 1st run,
it is needed in order to let him load the pg_trgm indexer
plugin.

Next, run 'init_db.sh' (alter the constants to match the
names of your postgres user and the DB), this creates the schema.

Then see 'eat.sh' and the 'eat_dump.py' it uses, re how to
fill your log archive DB. 'eat_dump.py' eats in Phf's classical
format, e.g.:

1926177;1564727032;mp_en_viaje;in the meantime, everyone's invited on trilema & other blogs.

where 1926177 is absolute line index (in given chan), 1564727032
is unix epochal timestamp, mp_en_viaje is speaker (if he is
'actioning', there will be a * behind his name), and the remainder
of the line is the payload.

You WILL need to adjust the constants in 'eat_dump.py', it is not
currently capable of eating config file. Set these to your DB
and PG user.

Now, adjust the constants in 'nsabot.conf' (rename per taste)
to specify your IRC params, name of bot, host at which www
logger will reside, and other knob values.

Adjust the three 'flask' templates in 'templates' subdir to
give the desired look and feel for the www end. Currently we are using
Phf's classic style sheet, with minor modifications.

'reader.py' takes one mandatory command-line argument: full path
to the config above. Same for 'bot.py', which is the IRC bot.

Run these via e.g. nohup ./bot.py & ; nohup ./reader.py &
and let your proxying WWW server know how to reach the latter's port.

For bot.py you will need a registered nick on fleanode (or wherever
it is used.) There are no fleanode-specific hacks in the bot, ergo
it can be stood up behind ZNC (although this has not been tested.)

Certain important features are presently unimplemented, in no order:
(1) Backlinkage.
(2) Search result pagination.
(3) Double-quoted search terms.
(4) Paste archiving.
(5) Multi-headed IRC bot for weather resistance.
(6) Informative eggogology for bot commands.
(7) Automatic synchronization with mirrors

A ZNC log eater is also required, to properly fill in the archives.
This is not yet available at the time of this writing.

###############
Aug. 11 Update:
###############

(1) Multi-line selection is implemented.

(2) User may request export of raw log lines in Phf-classical format.
    E.g. :
    http://logs.nosuchlabs.com/log-raw/trilema?istart=1925000&iend=1925500


###############
Aug. 16 Update:
###############

(1) Bot: added Phf's algo for uniturd processing ('fallback to latin-1').
(2) Bot: May leave IRC password blank in config for use with unregged bots
(3) WWW: 'Checkerboard' colouring; footer links.
(4) WWW: Links in log lines now open in new browser tab (a la Phf's log)
(5) WWW: Can now be loaded 'in reverse gear', http://...../log?rev=1
(6) WWW: 'Random' knob, takes reader to an arbitrary moment in past log.

###############
Sep. 28 Update:
###############

(1) Bot: incorporated reconnector fixes.
(2) Bot: 'seen', 'seen-all', and 'version' knobs implemented.
(3) WWW: up/down arrows for mouse-driven operation on pocket comps
(4) WWW: double-quotes prevent breakup of search query
    (Note: still needs improvement to force matching on word boundaries)
(5) WWW: demarkation of "action" lines
(6) WWW: Raw 'Tape" knob:

e.g. http://logs.nosuchlabs.com/tape?istart=1937934&iend=1937941
returns:

trilema;1937934;1569444908;asciilifeform;achtung panzers! piz pipe down ?!
trilema;1937935;1569444920;diana_coman;seems so
trilema;1937936;1569444957;asciilifeform;paging BingoBoingo !
trilema;1937937;1569444974;diana_coman;well, apparetly he was connected from piz too, lol
trilema;1937938;1569444982;diana_coman;ossasepia.com (pizarro) is down too, yes
trilema;1937939;1569445010;asciilifeform;dulap unreachable for 1st time since year+ ago when bb elbowed the mains cord
trilema;1937940;1569445073;diana_coman;apparently back on
trilema;1937941;1569445079;diana_coman;wb BingoBoingo !

Still needs a variant of 'eater' that will eat these.


(1) WWW: Removed ill-conceived 'Tape' knob.
(2) Bot: Added silent-disconnection detector.

Now requires the following value in bot config e.g.:

[irc]
disc_t  = 180

This represents the currently-recommended interval: 3 minutes.
If inactivity on the incoming line (incl. pingism) exceeds this interval,
a disconnection is forced. The normal reconnection routine will execute.

To disable forced disconnects, set disc_t to zero.

#################
Jul. 2020 Update:
#################

(1) Hide by default chans where no activity in days_hide day (set in config.)
(2) Added 'join as guest' link (via 'kiwi' wwwtronic IRC client)
(3) Example Apache-WSGI configs.