Commit 953d41c4 authored by Gnus developers's avatar Gnus developers Committed by Katsumi Yamaoka
Browse files

Merge changes made in Gnus trunk.

nnir.el: General clean-up, and reimplementation of various bits.
nnir.el (nnir-search-engine): Ressurect variable, since it's used later in the file.
shr.el (shr-generic): The text nodes should be text, not :text.
nnir.el: Move defvars around to silence compiler warnings.
shr.el (shr-tag-img): Output "*" instead of "[img]".
parent 8a500a91
2010-10-30 Lars Magne Ingebrigtsen <larsi@gnus.org>
* shr.el (shr-tag-img): Output "*" instead of "[img]".
2010-10-30 Andrew Cohen <cohen@andy.bu.edu>
* nnir.el move defvar, defcustom around to keep file organized and keep
byte-compiler quiet.
(nnir-read-parms): accept search-engine as arg.
(nnir-run-query): pass search-engine as arg.
(nnir-search-engine): remove.
2010-10-30 Lars Magne Ingebrigtsen <larsi@gnus.org>
* shr.el (shr-generic): The text nodes should be text, not :text.
* nnir.el (nnir-search-engine): Ressurect variable, since it's used
later in the file.
2010-10-30 Andrew Cohen <cohen@andy.bu.edu>
* nnir.el: general clean up. allow searching with multiple
engines. allow separate extra-parameters for each engine. batch queries
when possible.
(nnir-imap-default-search-key,nnir-method-default-engines): add
customize interface.
(nnir-run-gmane): new engine.
(nnir-engines): use it. qualify all prompts with engine name.
(nnir-search-engine): remove global variable.
(nnir-run-hyrex): restore for now.
(nnir-extra-parms,nnir-search-history): new variables.
(gnus-group-make-nnir-group): use them.
(nnir-group-server): remove in favor of gnus-group-server.
(nnir-request-group): avoid searching twice.
(nnir-sort-groups-by-server): new function.
2010-10-30 Julien Danjou <julien@danjou.info>
 
* gnus-group.el: Remove gnus-group-fetch-control.
......
......@@ -32,161 +32,40 @@
;; TODO: Documentation in the Gnus manual
;; From: Reiner Steib
;; Subject: Re: Including nnir.el
;; Newsgroups: gmane.emacs.gnus.general
;; Message-ID: <v9d5dnp6aq.fsf@marauder.physik.uni-ulm.de>
;; Date: 2006-06-05 22:49:01 GMT
;;
;; On Sun, Jun 04 2006, Sascha Wilde wrote:
;;
;; > The one thing most hackers like to forget: Documentation. By now the
;; > documentation is only in the comments at the head of the source, I
;; > would use it as basis to cook up some minimal texinfo docs.
;; >
;; > Where in the existing gnus manual would this fit best?
;; Maybe (info "(gnus)Combined Groups") for a general description.
;; `gnus-group-make-nnir-group' might be described in (info
;; "(gnus)Foreign Groups") as well.
;; Where in the existing gnus manual would this fit best?
;; The most recent version of this can always be fetched from the Gnus
;; repository. See http://www.gnus.org/ for more information.
;; This code is still in the development stage but I'd like other
;; people to have a look at it. Please do not hesitate to contact me
;; with your ideas.
;; What does it do? Well, it allows you to index your mail using some
;; search engine (freeWAIS-sf, swish-e and others -- see later),
;; then type `G G' in the Group buffer and issue a query to the search
;; engine. You will then get a buffer which shows all articles
;; matching the query, sorted by Retrieval Status Value (score).
;; What does it do? Well, it allows you to search your mail using
;; some search engine (imap, namazu, swish-e, gmane and others -- see
;; later) by typing `G G' in the Group buffer. You will then get a
;; buffer which shows all articles matching the query, sorted by
;; Retrieval Status Value (score).
;; When looking at the retrieval result (in the Summary buffer) you
;; can type `G T' (aka M-x gnus-summary-nnir-goto-thread RET) on an
;; article. You will be teleported into the group this article came
;; from, showing the thread this article is part of. (See below for
;; restrictions.)
;; The Lisp installation is simple: just put this file on your
;; load-path, byte-compile it, and load it from ~/.gnus or something.
;; This will install a new command `G G' in your Group buffer for
;; searching your mail. Note that you also need to configure a number
;; of variables, as described below.
;; from, showing the thread this article is part of.
;; Restrictions:
;;
;; * This expects that you use nnml or another one-file-per-message backend,
;; because the others doesn't support nnfolder.
;; * It can only search the mail backend's which are supported by one
;; search engine, because of different query languages.
;; * There are restrictions to the Wais setup.
;; * There are restrictions to the imap setup.
;; * gnus-summary-nnir-goto-thread: Fetches whole group first, before
;; limiting to the right articles. This is much too slow, of
;; course. May issue a query for number of articles to fetch; you
;; must accept the default of all articles at this point or things
;; may break.
;; The Lisp setup involves setting a few variables and setting up the
;; The Lisp setup may involve setting a few variables and setting up the
;; search engine. You can define the variables in the server definition
;; like this :
;; (setq gnus-secondary-select-methods '(
;; (nnimap "" (nnimap-address "localhost")
;; (nnir-search-engine namazu)
;; )))
;; Or you can define the global ones. The variables set in the mailer-
;; definition will be used first.
;; The variable to set is `nnir-search-engine'. Choose one of the engines
;; listed in `nnir-engines'. (Actually `nnir-engines' is an alist,
;; type `C-h v nnir-engines RET' for more information; this includes
;; examples for setting `nnir-search-engine', too.)
;;
;; The variable nnir-mail-backend isn't used anymore.
;;
;; The main variable to set is `nnir-search-engine'. Choose one of
;; the engines listed in `nnir-engines'. (Actually `nnir-engines' is
;; an alist, type `C-h v nnir-engines RET' for more information; this
;; includes examples for setting `nnir-search-engine', too.)
;; You must also set up a search engine. I'll tell you about the two
;; search engines currently supported:
;; If you use one of the local indices (namazu, find-grep, swish) you
;; must also set up a search engine backend.
;; 1. freeWAIS-sf
;;
;; As always with freeWAIS-sf, you need a so-called `format file'. I
;; use the following file:
;;
;; ,-----
;; | # Kai's format file for freeWAIS-sf for indexing mails.
;; | # Each mail is in a file, much like the MH format.
;; |
;; | # Document separator should never match -- each file is a document.
;; | record-sep: /^@this regex should never match@$/
;; |
;; | # Searchable fields specification.
;; |
;; | region: /^[sS]ubject:/ /^[sS]ubject: */
;; | subject "Subject header" stemming TEXT BOTH
;; | end: /^[^ \t]/
;; |
;; | region: /^([tT][oO]|[cC][cC]):/ /^([tT][oO]|[cC][cC]): */
;; | to "To and Cc headers" SOUNDEX BOTH
;; | end: /^[^ \t]/
;; |
;; | region: /^[fF][rR][oO][mM]:/ /^[fF][rR][oO][mM]: */
;; | from "From header" SOUNDEX BOTH
;; | end: /^[^ \t]/
;; |
;; | region: /^$/
;; | stemming TEXT GLOBAL
;; | end: /^@this regex should never match@$/
;; `-----
;;
;; 1998-07-22: waisindex would dump core on me for large articles with
;; the above settings. I used /^$/ as the end regex for the global
;; field. That seemed to work okay.
;; There is a Perl module called `WAIS.pm' which is available from
;; CPAN as well as ls6-ftp.cs.uni-dortmund.de:/pub/wais/Perl. This
;; module comes with a nifty tool called `makedb', which I use for
;; indexing. Here's my `makedb.conf':
;;
;; ,-----
;; | # Config file for makedb
;; |
;; | # Global options
;; | waisindex = /usr/local/bin/waisindex
;; | wais_opt = -stem -t fields
;; | # `-stem' option necessary when `stemming' is specified for the
;; | # global field in the *.fmt file
;; |
;; | # Own variables
;; | homedir = /home/kai
;; |
;; | # The mail database.
;; | database = mail
;; | files = `find $homedir/Mail -name \*[0-9] -print`
;; | dbdir = $homedir/.wais
;; | limit = 100
;; `-----
;;
;; The Lisp setup involves the `nnir-wais-*' variables. The most
;; difficult to understand variable is probably
;; `nnir-wais-remove-prefix'. Here's what it does: the output of
;; `waissearch' basically contains the file name and the (full)
;; directory name. As Gnus works with group names rather than
;; directory names, the directory name is transformed into a group
;; name as follows: first, a prefix is removed from the (full)
;; directory name, then all `/' are replaced with `.'. The variable
;; `nnir-wais-remove-prefix' should contain a regex matching exactly
;; this prefix. It defaults to `$HOME/Mail/' (note the trailing
;; slash).
;; 2. Namazu
;; 1. Namazu
;;
;; The Namazu backend requires you to have one directory containing all
;; index files, this is controlled by the `nnir-namazu-index-directory'
;; variable. To function the `nnir-namazu-remove-prefix' variable must
;; also be correct, see the documentation for `nnir-wais-remove-prefix'
;; also be correct, see the documentation for `nnir-namazu-remove-prefix'
;; above.
;;
;; It is particularly important not to pass any any switches to namazu
......@@ -225,7 +104,7 @@
;; For maximum searching efficiency I have a cron job set to run this
;; command every four hours.
;; 3. find-grep
;; 2. find-grep
;;
;; The find-grep engine simply runs find(1) to locate eligible
;; articles and searches them with grep(1). This, of course, is much
......@@ -281,39 +160,7 @@
;; function should return the list of articles as a vector, as
;; described above. Then, you need to register this backend in
;; `nnir-engines'. Then, users can choose the backend by setting
;; `nnir-search-engine'.
;; Todo, or future ideas:
;; * It should be possible to restrict search to certain groups.
;;
;; * There is currently no error checking.
;;
;; * The summary buffer display is currently really ugly, with all the
;; added information in the subjects. How could I make this
;; prettier?
;;
;; * A function which can be called from an nnir summary buffer which
;; teleports you into the group the current article came from and
;; shows you the whole thread this article is part of.
;; Implementation suggestions?
;; (1998-07-24: There is now a preliminary implementation, but
;; it is much too slow and quite fragile.)
;;
;; * Support other mail backends. In particular, probably quite a few
;; people use nnfolder. How would one go about searching nnfolders
;; and producing the right data needed? The group name and the RSV
;; are simple, but what about the article number?
;; - The article number is encoded in the `X-Gnus-Article-Number'
;; header of each mail.
;;
;; * Support compressed mail files. Probably, just stripping off the
;; `.gz' or `.Z' file name extension is sufficient.
;;
;; * At least for imap, the query is performed twice.
;;
;; Have you got other ideas?
;; `nnir-search-engine' as a server variable.
;;; Setup Code:
......@@ -336,116 +183,27 @@
(gnus-declare-backend "nnir" 'mail)
(defvar nnir-imap-default-search-key "Whole message"
"The default IMAP search key for an nnir search. Must be one of
the keys in nnir-imap-search-arguments. To use raw imap queries
by default set this to \"Imap\"")
(defvar nnir-imap-search-arguments
'(("Whole message" . "TEXT")
("Subject" . "SUBJECT")
("To" . "TO")
("From" . "FROM")
("Imap" . ""))
"Mapping from user readable keys to IMAP search items for use in nnir")
(defvar nnir-imap-search-other "HEADER %S"
"The IMAP search item to use for anything other than
nnir-imap-search-arguments. By default this is the name of an
email header field")
(defvar nnir-imap-search-argument-history ()
"The history for querying search options in nnir")
(defvar nnir-get-article-nov-override-function nil
"If non-nil, a function that will be passed each search result. This
should return a message's headers in NOV format.
If this variable is nil, or if the provided function returns nil for a search
result, `gnus-retrieve-headers' will be called instead.")
(defvar nnir-method-default-engines
'((nnimap . imap)
(nntp . nil))
"Alist of default search engines by server method")
;;; Developer Extension Variable:
(defvar nnir-engines
`((wais nnir-run-waissearch
())
(imap nnir-run-imap
((criteria
"Search in" ; Prompt
,(mapcar 'car nnir-imap-search-arguments) ; alist for completing
nil ; allow any user input
nil ; initial value
nnir-imap-search-argument-history ; the history to use
,nnir-imap-default-search-key ; default
)))
(swish++ nnir-run-swish++
((group . "Group spec: ")))
(swish-e nnir-run-swish-e
((group . "Group spec: ")))
(namazu nnir-run-namazu
())
(find-grep nnir-run-find-grep
((grep-options . "Grep options: "))))
"Alist of supported search engines.
Each element in the alist is a three-element list (ENGINE FUNCTION ARGS).
ENGINE is a symbol designating the searching engine. FUNCTION is also
a symbol, giving the function that does the search. The third element
ARGS is a list of cons pairs (PARAM . PROMPT). When issuing a query,
the FUNCTION will issue a query for each of the PARAMs, using PROMPT.
The value of `nnir-search-engine' must be one of the ENGINE symbols.
For example, use the following line for searching using freeWAIS-sf:
(setq nnir-search-engine 'wais)
Use the following line if you read your mail via IMAP and your IMAP
server supports searching:
(setq nnir-search-engine 'imap)
Note that you have to set additional variables for most backends. For
example, the `wais' backend needs the variables `nnir-wais-program',
`nnir-wais-database' and `nnir-wais-remove-prefix'.
Add an entry here when adding a new search engine.")
;;; User Customizable Variables:
(defgroup nnir nil
"Search nnmh and nnml groups in Gnus with swish-e, freeWAIS-sf, or EWS."
"Search groups in Gnus with assorted seach engines."
:group 'gnus)
;; Mail backend.
;; TODO:
;; If `nil', use server parameters to find out which server to search. CCC
;;
(defcustom nnir-mail-backend '(nnml "")
"*Specifies which backend should be searched.
More precisely, this is used to determine from which backend to fetch the
messages found.
This must be equal to an existing server, so maybe it is best to use
something like the following:
(setq nnir-mail-backend (nth 0 gnus-secondary-select-methods))
The above line works fine if the mail backend you want to search is
the first element of gnus-secondary-select-methods (`nth' starts counting
at zero)."
:type '(sexp)
(defcustom nnir-method-default-engines
'((nnimap . imap)
(nntp . gmane))
"*Alist of default search engines keyed by server method"
:type '(alist)
:group 'nnir)
;; Search engine to use.
(defcustom nnir-search-engine 'wais
"*The search engine to use. Must be a symbol.
See `nnir-engines' for a list of supported engines, and for example
settings of `nnir-search-engine'."
:type '(sexp)
(defcustom nnir-imap-default-search-key "Whole message"
"*The default IMAP search key for an nnir search. Must be one of
the keys in `nnir-imap-search-arguments'. To use raw imap queries
by default set this to \"Imap\""
:type '(string)
:group 'nnir)
;; freeWAIS-sf.
(defcustom nnir-wais-program "waissearch"
"*Name of waissearch executable."
:type '(string)
......@@ -501,8 +259,8 @@ Instead, use this:
in order to get a group name (albeit with / instead of .). This is a
regular expression.
This variable is very similar to `nnir-wais-remove-prefix', except
that it is for swish++, not Wais."
This variable is very similar to `nnir-namazu-remove-prefix', except
that it is for swish++, not Namazu."
:type '(regexp)
:group 'nnir)
......@@ -552,13 +310,47 @@ This could be a server parameter."
in order to get a group name (albeit with / instead of .). This is a
regular expression.
This variable is very similar to `nnir-wais-remove-prefix', except
that it is for swish-e, not Wais.
This variable is very similar to `nnir-namazu-remove-prefix', except
that it is for swish-e, not Namazu.
This could be a server parameter."
:type '(regexp)
:group 'nnir)
;; HyREX engine, see <URL:http://ls6-www.cs.uni-dortmund.de/>
(defcustom nnir-hyrex-program "nnir-search"
"*Name of the nnir-search executable."
:type '(string)
:group 'nnir)
(defcustom nnir-hyrex-additional-switches '()
"*A list of strings, to be given as additional arguments for nnir-search.
Note that this should be a list. Ie, do NOT use the following:
(setq nnir-hyrex-additional-switches \"-ddl ddl.xml -c nnir\") ; wrong !
Instead, use this:
(setq nnir-hyrex-additional-switches '(\"-ddl\" \"ddl.xml\" \"-c\" \"nnir\"))"
:type '(repeat (string))
:group 'nnir)
(defcustom nnir-hyrex-index-directory (getenv "HOME")
"*Index directory for HyREX."
:type '(directory)
:group 'nnir)
(defcustom nnir-hyrex-remove-prefix (concat (getenv "HOME") "/Mail/")
"*The prefix to remove from each file name returned by HyREX
in order to get a group name (albeit with / instead of .).
For example, suppose that HyREX returns file names such as
\"/home/john/Mail/mail/misc/42\". For this example, use the following
setting: (setq nnir-hyrex-remove-prefix \"/home/john/Mail/\")
Note the trailing slash. Removing this prefix gives \"mail/misc/42\".
`nnir' knows to remove the \"/42\" and to replace \"/\" with \".\" to
arrive at the correct group name, \"mail.misc\"."
:type '(directory)
:group 'nnir)
;; Namazu engine, see <URL:http://www.namazu.org/>
(defcustom nnir-namazu-program "namazu"
......@@ -587,11 +379,83 @@ Instead, use this:
"*The prefix to remove from each file name returned by Namazu
in order to get a group name (albeit with / instead of .).
This variable is very similar to `nnir-wais-remove-prefix', except
that it is for Namazu, not Wais."
For example, suppose that Namazu returns file names such as
\"/home/john/Mail/mail/misc/42\". For this example, use the following
setting: (setq nnir-namazu-remove-prefix \"/home/john/Mail/\")
Note the trailing slash. Removing this prefix gives \"mail/misc/42\".
`nnir' knows to remove the \"/42\" and to replace \"/\" with \".\" to
arrive at the correct group name, \"mail.misc\"."
:type '(directory)
:group 'nnir)
;; Imap variables
(defvar nnir-imap-search-arguments
'(("Whole message" . "TEXT")
("Subject" . "SUBJECT")
("To" . "TO")
("From" . "FROM")
("Imap" . ""))
"Mapping from user readable keys to IMAP search items for use in nnir")
(defvar nnir-imap-search-other "HEADER %S"
"The IMAP search item to use for anything other than
`nnir-imap-search-arguments'. By default this is the name of an
email header field")
(defvar nnir-imap-search-argument-history ()
"The history for querying search options in nnir")
;;; Developer Extension Variable:
(defvar nnir-engines
`((wais nnir-run-waissearch
())
(imap nnir-run-imap
((criteria
"Imap Search in" ; Prompt
,(mapcar 'car nnir-imap-search-arguments) ; alist for completing
nil ; allow any user input
nil ; initial value
nnir-imap-search-argument-history ; the history to use
,nnir-imap-default-search-key ; default
)))
(gmane nnir-run-gmane
((author . "Gmane Author: ")))
(swish++ nnir-run-swish++
((group . "Swish++ Group spec: ")))
(swish-e nnir-run-swish-e
((group . "Swish-e Group spec: ")))
(namazu nnir-run-namazu
())
(hyrex nnir-run-hyrex
((group . "Hyrex Group spec: ")))
(find-grep nnir-run-find-grep
((grep-options . "Grep options: "))))
"Alist of supported search engines.
Each element in the alist is a three-element list (ENGINE FUNCTION ARGS).
ENGINE is a symbol designating the searching engine. FUNCTION is also
a symbol, giving the function that does the search. The third element
ARGS is a list of cons pairs (PARAM . PROMPT). When issuing a query,
the FUNCTION will issue a query for each of the PARAMs, using PROMPT.
The value of `nnir-search-engine' must be one of the ENGINE symbols.
For example, for searching a server using namazu include
(nnir-search-engine namazu)
in the server definition. Note that you have to set additional
variables for most backends. For example, the `namazu' backend
needs the variables `nnir-namazu-program',
`nnir-namazu-index-directory' and `nnir-namazu-remove-prefix'.
Add an entry here when adding a new search engine.")
(defvar nnir-get-article-nov-override-function nil
"If non-nil, a function that will be passed each search result. This
should return a message's headers in NOV format.
If this variable is nil, or if the provided function returns nil for a search
result, `gnus-retrieve-headers' will be called instead.")
;;; Internal Variables:
(defvar nnir-current-query nil
......@@ -609,43 +473,31 @@ that it is for Namazu, not Wais."
(defvar nnir-tmp-buffer " *nnir*"
"Internal: temporary buffer.")
(defvar nnir-search-history ()
"Internal: the history for querying search options in nnir")
(defvar nnir-extra-parms nil
"Internal: stores request for extra search parms")
;;; Code:
;; Gnus glue.
(defun gnus-group-make-nnir-group (extra-parms query)
(defun gnus-group-make-nnir-group (nnir-extra-parms)
"Create an nnir group. Asks for query."
(interactive "P\nsQuery: ")
(interactive "P")
(setq nnir-current-query nil
nnir-current-server nil
nnir-current-group-marked nil
nnir-artlist nil)
(let ((parms nil))
(if extra-parms
(setq parms (nnir-read-parms query))
(setq parms (list (cons 'query query))))
(let* ((query (read-string "Query: " nil 'nnir-search-history))
(parms (list (cons 'query query))))
(add-to-list 'parms (cons 'unique-id (message-unique-id)) t)
(gnus-group-read-ephemeral-group
(concat "nnir:" (prin1-to-string parms)) '(nnir "") t
(cons (current-buffer)
gnus-current-window-configuration)
(cons (current-buffer) gnus-current-window-configuration)
nil)))
;; Why is this needed? Is this for compatibility with old/new gnusae? Using
;; gnus-group-server instead works for me. -- Justus Piater
(defmacro nnir-group-server (group)
"Return the server for a newsgroup GROUP.
The returned format is as `gnus-server-to-method' needs it. See
`gnus-group-real-prefix' and `gnus-group-real-name'."
`(let ((gname ,group))
(if (string-match "^\\([^:]+\\):" gname)
(progn
(setq gname (match-string 1 gname))
(if (string-match "^\\([^+]+\\)\\+\\(.+\\)$" gname)
(format "%s:%s" (match-string 1 gname) (match-string 2 gname))
(concat gname ":")))
(format "%s:%s" (car gnus-select-method) (cadr gnus-select-method)))))
;; Summary mode commands.
(defun gnus-summary-nnir-goto-thread ()
......@@ -660,22 +512,27 @@ and show thread that contains this article."
(id (mail-header-id (gnus-summary-article-header)))
(refs (split-string
(mail-header-references (gnus-summary-article-header)))))
(if (eq (car (gnus-group-method group)) 'nnimap)
(progn (nnimap-possibly-change-group (gnus-group-short-name group) nil)
(with-current-buffer (nnimap-buffer)
(let* ((cmd (let ((value (format
"(OR HEADER REFERENCES %s HEADER Message-Id %s)"
id id)))
(dolist (refid refs value)
(setq value (format
"(OR (OR HEADER Message-Id %s HEADER REFERENCES %s) %s)"
refid refid value)))))
(result (nnimap-command
"UID SEARCH %s" cmd)))
(gnus-summary-read-group-1 group t t gnus-summary-buffer nil
(and (car result)
(delete 0 (mapcar #'string-to-number
(cdr (assoc "SEARCH" (cdr result))))))))))
(if (eq (car (gnus-find-method-for-group group)) 'nnimap)
(progn
(nnimap-possibly-change-group (gnus-group-short-name group) nil)
(with-current-buffer (nnimap-buffer)
(let* ((cmd
(let ((value
(format
"(OR HEADER REFERENCES %s HEADER Message-Id %s)"
id id)))
(dolist (refid refs value)
(setq value
(format
"(OR (OR HEADER Message-Id %s HEADER REFERENCES %s) %s)"
refid refid value)))))
(result (nnimap-command "UID SEARCH %s" cmd)))
(gnus-summary-read-group-1
group t t gnus-summary-buffer nil
(and (car result)
(delete 0 (mapcar
#'string-to-number
(cdr (assoc "SEARCH" (cdr result))))))))))
(gnus-summary-read-group-1 group t t gnus-summary-buffer
nil (list backend-number))
(gnus-summary-limit (list backend-number))
......@@ -711,22 +568,17 @@ and show thread that contains this article."
;; Cache miss.
(setq nnir-artlist (nnir-run-query group)))
(with-current-buffer nntp-server-buffer
(setq nnir-current-query group)
(when server (setq nnir-current-server server))