url.texi 38.2 KB
Newer Older
Stefan Monnier's avatar
Stefan Monnier committed
1
\input texinfo
Richard M. Stallman's avatar
Richard M. Stallman committed
2
@setfilename ../info/url
Stefan Monnier's avatar
Stefan Monnier committed
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
@settitle URL Programmer's Manual

@iftex
@c @finalout
@end iftex
@c @setchapternewpage odd
@c @smallbook

@tex
\overfullrule=0pt
%\global\baselineskip 30pt      % for printing in double space
@end tex
@dircategory World Wide Web
@dircategory GNU Emacs Lisp
@direntry
* URL: (url).                 URL loading package.
@end direntry

@ifnottex
This file documents the URL loading package.

24 25 26
Copyright @copyright{} 1996, 1997, 1998, 1999, 2002, 2004,
2005, 2006 Free Software Foundation, Inc.@*
Copyright @copyright{} 1993, 1994, 1995, 1996  William M. Perry
Stefan Monnier's avatar
Stefan Monnier committed
27 28

Permission is granted to copy, distribute and/or modify this document
29
under the terms of the GNU Free Documentation License, Version 1.2 or
Stefan Monnier's avatar
Stefan Monnier committed
30
any later version published by the Free Software Foundation; with the
Luc Teirlinck's avatar
Luc Teirlinck committed
31
Invariant Sections being
Stefan Monnier's avatar
Stefan Monnier committed
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
``GNU GENERAL PUBLIC LICENSE''.  A copy of the
license is included in the section entitled ``GNU Free Documentation
License.''
@end ifnottex

@c
@titlepage
@sp 6
@center @titlefont{URL}
@center @titlefont{Programmer's Manual}
@sp 4
@center First Edition, URL Version 2.0
@sp 1
@c @center December 1999
@sp 5
@center William M. Perry
@center @email{wmperry@@gnu.org}
@center David Love
@center @email{fx@@gnu.org}
@page
@vskip 0pt plus 1filll
Copyright @copyright{} 1993, 1994, 1995, 1996 William M. Perry@*
54
Copyright @copyright{} 1996, 1997, 1998, 1999, 2002, 2003, 2004,
55
2005, 2006 Free Software Foundation, Inc.
Stefan Monnier's avatar
Stefan Monnier committed
56 57

Permission is granted to copy, distribute and/or modify this document
58
under the terms of the GNU Free Documentation License, Version 1.2 or
Stefan Monnier's avatar
Stefan Monnier committed
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78
any later version published by the Free Software Foundation; with the
Invariant Sections being
``GNU GENERAL PUBLIC LICENSE''.  A copy of the
license is included in the section entitled ``GNU Free Documentation
License.''
@end titlepage
@page
@node Top
@top URL



@menu
* Getting Started::             Preparing your program to use URLs.
* Retrieving URLs::             How to use this package to retrieve a URL.
* Supported URL Types::         Descriptions of URL types currently supported.
* Defining New URLs::           How to define a URL loader for a new protocol.
* General Facilities::          URLs can be cached, accessed via a gateway
                                and tracked in a history list.
* Customization::               Variables you can alter.
Luc Teirlinck's avatar
Luc Teirlinck committed
79 80 81
* Function Index::
* Variable Index::
* Concept Index::
Stefan Monnier's avatar
Stefan Monnier committed
82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97
@end menu

@node Getting Started
@chapter Getting Started
@cindex URLs, definition
@cindex URIs

@dfn{Uniform Resource Locators} (URLs) are a specific form of
@dfn{Uniform Resource Identifiers} (URI) described in RFC 2396 which
updates RFC 1738 and RFC 1808.  RFC 2016 defines uniform resource
agents.

URIs have the form @var{scheme}:@var{scheme-specific-part}, where the
@var{scheme}s supported by this library are described below.
@xref{Supported URL Types}.

Juanma Barranquero's avatar
Juanma Barranquero committed
98
FTP, NFS, HTTP, HTTPS, @code{rlogin}, @code{telnet}, tn3270,
Stefan Monnier's avatar
Stefan Monnier committed
99 100 101 102 103 104 105 106 107 108 109 110 111 112 113
IRC and gopher URLs all have the form

@example
@var{scheme}://@r{[}@var{userinfo}@@@r{]}@var{hostname}@r{[}:@var{port}@r{]}@r{[}/@var{path}@r{]}
@end example
@noindent
where @samp{@r{[}} and @samp{@r{]}} delimit optional parts.
@var{userinfo} sometimes takes the form @var{username}:@var{password}
but you should beware of the security risks of sending cleartext
passwords.  @var{hostname} may be a domain name or a dotted decimal
address.  If the @samp{:@var{port}} is omitted then the library will
use the `well known' port for that service when accessing URLs.  With
the possible exception of @code{telnet}, it is rare for ports to be
specified, and it is possible using a non-standard port may have
undesired consequences if a different service is listening on that
Richard M. Stallman's avatar
Richard M. Stallman committed
114
port (e.g., an HTTP URL specifying the SMTP port can cause mail to be
Juanma Barranquero's avatar
Juanma Barranquero committed
115 116
sent). @c , but @xref{Other Variables, url-bad-port-list}.
The meaning of the @var{path} component depends on the service.
Stefan Monnier's avatar
Stefan Monnier committed
117 118

@menu
Luc Teirlinck's avatar
Luc Teirlinck committed
119
* Configuration::
Stefan Monnier's avatar
Stefan Monnier committed
120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145
* Parsed URLs::                 URLs are parsed into vector structures.
@end menu

@node Configuration
@section Configuration

@defvar url-configuration-directory
@cindex @file{~/.url}
@cindex configuration files
The directory in which URL configuration files, the cache etc.,
reside.  Default @file{~/.url}.
@end defvar

@node Parsed URLs
@section Parsed URLs
@cindex parsed URLs
The library functions typically operate on @dfn{parsed} versions of
URLs.  These are actually vectors of the form:

@example
[@var{type} @var{user} @var{password} @var{host} @var{port} @var{file} @var{target} @var{attributes} @var{full}]
@end example

@noindent where
@table @var
@item type
Richard M. Stallman's avatar
Richard M. Stallman committed
146
is the type of the URL scheme, e.g., @code{http}
Stefan Monnier's avatar
Stefan Monnier committed
147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185
@item user
is the username associated with it, or @code{nil};
@item password
is the user password associated with it, or @code{nil};
@item host
is the host name associated with it, or @code{nil};
@item port
is the port number associated with it, or @code{nil};
@item file
is the `file' part of it, or @code{nil}.  This doesn't necessarily
actually refer to a file;
@item target
is the target part, or @code{nil};
@item attributes
is the attributes associated with it, or @code{nil};
@item full
is @code{t} for a fully-specified URL, with a host part indicated by
@samp{//} after the scheme part.
@end table

@findex url-type
@findex url-user
@findex url-password
@findex url-host
@findex url-port
@findex url-file
@findex url-target
@findex url-attributes
@findex url-full
@findex url-set-type
@findex url-set-user
@findex url-set-password
@findex url-set-host
@findex url-set-port
@findex url-set-file
@findex url-set-target
@findex url-set-attributes
@findex url-set-full
These attributes have accessors named @code{url-@var{part}}, where
Richard M. Stallman's avatar
Richard M. Stallman committed
186
@var{part} is the name of one of the elements above, e.g.,
Stefan Monnier's avatar
Stefan Monnier committed
187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207
@code{url-host}.  Similarly, there are setters of the form
@code{url-set-@var{part}}.

There are functions for parsing and unparsing between the string and
vector forms.

@defun url-generic-parse-url url
Return a parsed version of the string @var{url}.
@end defun

@defun url-recreate-url url
@cindex unparsing URLs
Recreates a URL string from the parsed @var{url}.
@end defun

@node Retrieving URLs
@chapter Retrieving URLs

@defun url-retrieve-synchronously url
Retrieve @var{url} synchronously and return a buffer containing the
data.  @var{url} is either a string or a parsed URL structure.  Return
208
@code{nil} if there are no data associated with it (the case for dired,
Stefan Monnier's avatar
Stefan Monnier committed
209 210 211 212 213 214 215 216 217
info, or mailto URLs that need no further processing).
@end defun

@defun url-retrieve url callback &optional cbargs
Retrieve @var{url} asynchronously and call @var{callback} with args
@var{cbargs} when finished.  The callback is called when the object
has been completely retrieved, with the current buffer containing the
object and any MIME headers associated with it.  @var{url} is either a
string or a parsed URL structure.  Returns the buffer @var{url} will
218
load into, or @code{nil} if the process has already completed.
Stefan Monnier's avatar
Stefan Monnier committed
219 220 221 222 223 224 225
@end defun

@node Supported URL Types
@chapter Supported URL Types

@menu
* http/https::                  Hypertext Transfer Protocol.
Luc Teirlinck's avatar
Luc Teirlinck committed
226
* file/ftp::                    Local files and FTP archives.
Stefan Monnier's avatar
Stefan Monnier committed
227 228 229 230 231 232 233 234 235 236 237 238
* info::                        Emacs `Info' pages.
* mailto::                      Sending email.
* news/nntp/snews::             Usenet news.
* rlogin/telnet/tn3270::        Remote host connectivity.
* irc::                         Internet Relay Chat.
* data::                        Embedded data URLs.
* nfs::                         Networked File System
@c * finger::
@c * gopher::
@c * netrek::
@c * prospero::
* cid::                         Content-ID.
Luc Teirlinck's avatar
Luc Teirlinck committed
239
* about::
Stefan Monnier's avatar
Stefan Monnier committed
240 241 242 243 244 245 246 247 248 249 250 251 252
* ldap::                        Lightweight Directory Access Protocol
* imap::                        IMAP mailboxes.
* man::                         Unix man pages.
@end menu

@node http/https
@section @code{http} and @code{https}

The scheme @code{http} is Hypertext Transfer Protocol.  The library
supports version 1.1, specified in RFC 2616.  (This supersedes 1.0,
defined in RFC 1945) HTTP URLs have the following form, where most of
the parts are optional:
@example
253
http://@var{user}:@var{password}@@@var{host}:@var{port}/@var{path}?@var{searchpart}#@var{fragment}
Stefan Monnier's avatar
Stefan Monnier committed
254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276
@end example
@c The @code{:@var{port}} part is optional, and @var{port} defaults to
@c 80.  The @code{/@var{path}} part, if present, is a slash-separated
@c series elements.  The @code{?@var{searchpart}}, if present, is the
@c query for a search or the content of a form submission.  The
@c @code{#fragment} part, if present, is a location in the document.

The scheme @code{https} is a secure version of @code{http}, with
transmission via SSL.  It is defined in RFC 2069.  Its default port is
443.  This scheme depends on SSL support in Emacs via the
@file{ssl.el} library and is actually implemented by forcing the
@code{ssl} gateway method to be used.  @xref{Gateways in general}.

@defopt url-honor-refresh-requests
This controls honouring of HTTP @samp{Refresh} headers by which
servers can direct clients to reload documents from the same URL or a
or different one.  @code{nil} means they will not be honoured,
@code{t} (the default) means they will always be honoured, and
otherwise the user will be asked on each request.
@end defopt


@menu
Luc Teirlinck's avatar
Luc Teirlinck committed
277 278 279 280
* Cookies::
* HTTP language/coding::
* HTTP URL Options::
* Dealing with HTTP documents::
Stefan Monnier's avatar
Stefan Monnier committed
281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320
@end menu

@node Cookies
@subsection Cookies

@defopt url-cookie-file
The file in which cookies are stored, defaulting to @file{cookies} in
the directory specified by @code{url-configuration-directory}.
@end defopt

@defopt url-cookie-confirmation
Specifies whether confirmation is require to accept cookies.
@end defopt

@defopt url-cookie-multiple-line
Specifies whether to put all cookies for the server on one line in the
HTTP request to satisfy broken servers like
@url{http://www.hotmail.com}.
@end defopt

@defopt url-cookie-trusted-urls
A list of regular expressions matching URLs from which to accept
cookies always.
@end defopt

@defopt url-cookie-untrusted-urls
A list of regular expressions matching URLs from which to reject
cookies always.
@end defopt

@defopt url-cookie-save-interval
The number of seconds between automatic saves of cookies to disk.
Default is one hour.
@end defopt


@node HTTP language/coding
@subsection Language and Encoding Preferences

HTTP allows clients to express preferences for the language and
Richard M. Stallman's avatar
Richard M. Stallman committed
321 322
encoding of documents which servers may honour.  For each of these
variables, the value is a string; it can specify a single choice, or
323 324 325 326 327 328 329 330
it can be a comma-separated list.

Normally this list ordered by descending preference.  However, each
element can be followed by @samp{;q=@var{priority}} to specify its
preference level, a decimal number from 0 to 1; e.g., for
@code{url-mime-language-string}, @w{@code{"de, en-gb;q=0.8,
en;q=0.7"}}.  An element that has no @samp{;q} specification has
preference level 1.
Stefan Monnier's avatar
Stefan Monnier committed
331 332 333 334 335 336 337

@defopt url-mime-charset-string
@cindex character sets
@cindex coding systems
This variable specifies a preference for character sets when documents
can be served in more than one encoding.

Richard M. Stallman's avatar
Richard M. Stallman committed
338 339 340 341 342 343
HTTP allows specifying a series of MIME charsets which indicate your
preferred character set encodings, e.g., Latin-9 or Big5, and these
can be weighted.  The default series is generated automatically from
the associated MIME types of all defined coding systems, sorted by the
coding system priority specified in Emacs.  @xref{Recognize Coding, ,
Recognizing Coding Systems, emacs, The GNU Emacs Manual}.
Stefan Monnier's avatar
Stefan Monnier committed
344 345 346 347 348
@end defopt

@defopt url-mime-language-string
@cindex language preferences
A string specifying the preferred language when servers can serve
Richard M. Stallman's avatar
Richard M. Stallman committed
349 350 351 352 353
files in several languages.  Use RFC 1766 abbreviations, e.g.,
@samp{en} for English, @samp{de} for German.

The string can be @code{"*"} to get the first available language (as
opposed to the default).
Stefan Monnier's avatar
Stefan Monnier committed
354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395
@end defopt

@node HTTP URL Options
@subsection HTTP URL Options

HTTP supports an @samp{OPTIONS} method describing things supported by
the URL@.

@defun url-http-options url
Returns a property list describing options available for URL.  The
property list members are:

@table @code
@item methods
A list of symbols specifying what HTTP methods the resource
supports.

@item dav
@cindex DAV
A list of numbers specifying what DAV protocol/schema versions are
supported.

@item dasl
@cindex DASL
A list of supported DASL search types supported (string form).

@item ranges
A list of the units available for use in partial document fetches.

@item p3p
@cindex P3P
The @dfn{Platform For Privacy Protection} description for the resource.
Currently this is just the raw header contents.
@end table

@end defun

@node Dealing with HTTP documents
@subsection Dealing with HTTP documents

HTTP URLs are retrieved into a buffer containing the HTTP headers
followed by the body.  Since the headers are quasi-MIME, they may be
396 397 398
processed using the MIME library.  @xref{Top,, Emacs MIME,
emacs-mime, The Emacs MIME Manual}.  The URL package provides a
function to do this in general:
Stefan Monnier's avatar
Stefan Monnier committed
399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425

@defun url-decode-text-part handle &optional coding
This function decodes charset-encoded text in the current buffer.  In
Emacs, the buffer is expected to be unibyte initially and is set to
multibyte after decoding.
HANDLE is the MIME handle of the original part.  CODING is an explicit
coding to use, overriding what the MIME headers specify.
The coding system used for the decoding is returned.

Note that this function doesn't deal with @samp{http-equiv} charset
specifications in HTML @samp{<meta>} elements.
@end defun

@node file/ftp
@section file and ftp
@cindex files
@cindex FTP
@cindex File Transfer Protocol
@cindex compressed files
@findex dired

@example
ftp://@var{user}:@var{password}@@@var{host}:@var{port}/@var{file}
file://@var{user}:@var{password}@@@var{host}:@var{port}/@var{file}
@end example

These schemes are defined in RFC 1808.
Luc Teirlinck's avatar
Luc Teirlinck committed
426 427
@samp{ftp:} and @samp{file:} are synonymous in this library.  They
allow reading arbitrary files from hosts.  Either @samp{ange-ftp}
Stefan Monnier's avatar
Stefan Monnier committed
428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462
(Emacs) or @samp{efs} (XEmacs) is used to retrieve them from remote
hosts.  Local files are accessed directly.

Compressed files are handled, but support is hard-coded so that
@code{jka-compr-compression-info-list} and so on have no affect.
Suffixes recognized are @samp{.z}, @samp{.gz}, @samp{.Z} and
@samp{.bz2}.

@defopt url-directory-index-file
The filename to look for when indexing a directory, default
@samp{"index.html"}.  If this file exists, and is readable, then it
will be viewed instead of using @code{dired} to view the directory.
@end defopt

@node info
@section info
@cindex Info
@cindex Texinfo
@findex Info-goto-node

@example
info:@var{file}#@var{node}
@end example

Info URLs are not officially defined.  They invoke
@code{Info-goto-node} with argument @samp{(@var{file})@var{node}}.
@samp{#@var{node}} is optional, defaulting to @samp{Top}.

@node mailto
@section mailto

@cindex mailto
@cindex email
A mailto URL will send an email message to the address in the
URL, for example @samp{mailto:foo@@bar.com} would compose a
Luc Teirlinck's avatar
Luc Teirlinck committed
463
message to @samp{foo@@bar.com}.
Stefan Monnier's avatar
Stefan Monnier committed
464 465 466 467 468

@defopt url-mail-command
@vindex mail-user-agent
The function called whenever url needs to send mail.  This should
normally be left to default from @var{mail-user-agent}.  @xref{Mail
469
Methods, , Mail-Composition Methods, emacs, The GNU Emacs Manual}.
Stefan Monnier's avatar
Stefan Monnier committed
470 471 472 473 474 475 476 477 478 479
@end defopt

An @samp{X-Url-From} header field containing the URL of the document
that contained the mailto URL is added if that URL is known.

RFC 2368 extends the definition of mailto URLs in RFC 1738.
The form of a mailto URL is
@example
@samp{mailto:@var{mailbox}[?@var{header}=@var{contents}[&@var{header}=@var{contents}]]}
@end example
Luc Teirlinck's avatar
Luc Teirlinck committed
480
@noindent where an arbitrary number of @var{header}s can be added.  If the
Stefan Monnier's avatar
Stefan Monnier committed
481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504
@var{header} is @samp{body}, then @var{contents} is put in the body
otherwise a @var{header} header field is created with @var{contents}
as its contents.  Note that the URL library does not consider any
headers `dangerous' so you should check them before sending the
message.

@c Fixme: update
Email messages are defined in @sc{rfc}822.

@node news/nntp/snews
@section @code{news}, @code{nntp} and @code{snews}
@cindex news
@cindex network news
@cindex usenet
@cindex NNTP
@cindex snews

@c draft-gilman-news-url-01
The network news URL scheme take the following forms following RFC
1738 except that for compatibility with other clients, host and port
fields may be included in news URLs though they are properly only
allowed for nntp an snews.

@table @samp
Luc Teirlinck's avatar
Luc Teirlinck committed
505
@item news:@var{newsgroup}
Stefan Monnier's avatar
Stefan Monnier committed
506 507 508
Retrieves a list of messages in @var{newsgroup};
@item news:@var{message-id}
Retrieves the message with the given @var{message-id};
Luc Teirlinck's avatar
Luc Teirlinck committed
509
@item news:*
Stefan Monnier's avatar
Stefan Monnier committed
510 511 512 513 514 515 516 517 518 519 520 521
Retrieves a list of all available newsgroups;
@item nntp://@var{host}:@var{port}/@var{newsgroup}
@itemx nntp://@var{host}:@var{port}/@var{message-id}
@itemx nntp://@var{host}:@var{port}/*
Similar to the @samp{news} versions.
@end table

@samp{:@var{port}} is optional and defaults to :119.

@samp{snews} is the same as @samp{nntp} except that the default port
is :563.
@cindex SSL
Luc Teirlinck's avatar
Luc Teirlinck committed
522
(It is tunneled through SSL.)
Stefan Monnier's avatar
Stefan Monnier committed
523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561

An @samp{nntp} URL is the same as a news URL, except that the URL may
specify an article by its number.

@defopt url-news-server
This variable can be used to override the default news server.
Usually this will be set by the Gnus package, which is used to fetch
news.
@cindex environment variable
@vindex NNTPSERVER
It may be set from the conventional environment variable
@code{NNTPSERVER}.
@end defopt

@node rlogin/telnet/tn3270
@section rlogin, telnet and tn3270
@cindex rlogin
@cindex telnet
@cindex tn3270
@cindex terminal emulation
@findex terminal-emulator

These URL schemes from RFC 1738 for logon via a terminal emulator have
the form
@example
telnet://@var{user}:@var{password}@@@var{host}:@var{port}
@end example
but the @code{:@var{password}} component is ignored.

To handle rlogin, telnet and tn3270 URLs, a @code{rlogin},
@code{telnet} or @code{tn3270} (the program names and arguments are
hardcoded) session is run in a @code{terminal-emulator} buffer.
Well-known ports are used if the URL does not specify a port.

@node irc
@section irc
@cindex IRC
@cindex Internet Relay Chat
@cindex ZEN IRC
562 563
@cindex ERC
@cindex rcirc
Luc Teirlinck's avatar
Luc Teirlinck committed
564
@c Fixme: reference (was http://www.w3.org/Addressing/draft-mirashi-url-irc-01.txt)
Stefan Monnier's avatar
Stefan Monnier committed
565
@dfn{Internet Relay Chat} (IRC) is handled by handing off the @sc{irc}
Luc Teirlinck's avatar
Luc Teirlinck committed
566
session to a function named in @code{url-irc-function}.
Stefan Monnier's avatar
Stefan Monnier committed
567 568 569 570 571 572 573

@defopt url-irc-function
A function to actually open an IRC connection.
This function
must take five arguments, @var{host}, @var{port}, @var{channel},
@var{user} and @var{password}.  The @var{channel} argument specifies the
channel to join immediately, this can be @code{nil}.  By default this is
574
@code{url-irc-rcirc}.
Stefan Monnier's avatar
Stefan Monnier committed
575
@end defopt
576 577 578 579 580 581
@defun url-irc-rcirc host port channel user password
Processes the arguments and lets @code{rcirc} handle the session.
@end defun
@defun url-irc-erc host port channel user password
Processes the arguments and lets @code{ERC} handle the session.
@end defun
Stefan Monnier's avatar
Stefan Monnier committed
582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601
@defun url-irc-zenirc host port channel user password
Processes the arguments and lets @code{zenirc} handle the session.
@end defun

@node data
@section data
@cindex data URLs

@example
data:@r{[}@var{media-type}@r{]}@r{[};@var{base64}@r{]},@var{data}
@end example

Data URLs contain MIME data in the URL itself.  They are defined in
RFC 2397.

@var{media-type} is a MIME @samp{Content-Type} string, possibly
including parameters.  It defaults to
@samp{text/plain;charset=US-ASCII}.  The @samp{text/plain} can be
omitted but the charset parameter supplied.  If @samp{;base64} is
present, the @var{data} are base64-encoded.
Luc Teirlinck's avatar
Luc Teirlinck committed
602

Stefan Monnier's avatar
Stefan Monnier committed
603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677
@node nfs
@section nfs
@cindex NFS
@cindex Network File System
@cindex automounter

@example
nfs://@var{user}:@var{password}@@@var{host}:@var{port}/@var{file}
@end example

The @samp{nfs:} scheme is defined in RFC 2224.  It is similar to
@samp{ftp:} except that it points to a file on a remote host that is
handled by the automounter on the local host.

@defvar url-nfs-automounter-directory-spec
@end defvar
A string saying how to invoke the NFS automounter.  Certain @samp{%}
sequences are recognized:

@table @samp
@item %h
The hostname of the NFS server;
@item %n
The port number of the NFS server;
@item %u
The username to use to authenticate;
@item %p
The password to use to authenticate;
@item %f
The filename on the remote server;
@item %%
A literal @samp{%}.
@end table

Each can be used any number of times.

@node cid
@section cid
@cindex Content-ID

RFC 2111

@node about
@section about

@node ldap
@section ldap
@cindex LDAP
@cindex Lightweight Directory Access Protocol

The LDAP scheme is defined in RFC 2255.

@node imap
@section imap
@cindex IMAP

RFC 2192

@node man
@section man
@cindex @command{man}
@cindex Unix man pages
@findex man

@example
@samp{man:@var{page-spec}}
@end example

This is a non-standard scheme.  @var{page-spec} is passed directly to
the Lisp @code{man} function.

@node Defining New URLs
@chapter Defining New URLs

@menu
Luc Teirlinck's avatar
Luc Teirlinck committed
678 679 680 681 682
* Naming conventions::
* Required functions::
* Optional functions::
* Asynchronous fetching::
* Supporting file-name-handlers::
Stefan Monnier's avatar
Stefan Monnier committed
683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703
@end menu

@node Naming conventions
@section Naming conventions

@node Required functions
@section Required functions

@node Optional functions
@section Optional functions

@node Asynchronous fetching
@section Asynchronous fetching

@node Supporting file-name-handlers
@section Supporting file-name-handlers

@node General Facilities
@chapter General Facilities

@menu
Luc Teirlinck's avatar
Luc Teirlinck committed
704 705 706 707
* Disk Caching::
* Proxies::
* Gateways in general::
* History::
Stefan Monnier's avatar
Stefan Monnier committed
708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780
@end menu

@node Disk Caching
@section Disk Caching
@cindex Caching
@cindex Persistent Cache
@cindex Disk Cache

The disk cache stores retrieved documents locally, whence they can be
retrieved more quickly.  When requesting a URL that is in the cache,
the library checks to see if the page has changed since it was last
retrieved from the remote machine.  If not, the local copy is used,
saving the transmission over the network.
@cindex Cleaning the cache
@cindex Clearing the cache
@cindex Cache cleaning
Currently the cache isn't cleared automatically.
@c Running the @code{clean-cache} shell script
@c fist is recommended, to allow for future cleaning of the cache.  This
@c shell script will remove all files that have not been accessed since it
@c was last run.  To keep the cache pared down, it is recommended that this
@c script be run from @i{at} or @i{cron} (see the manual pages for
@c crontab(5) or at(1) for more information)

@defopt url-automatic-caching
Setting this variable non-@code{nil} causes documents to be cached
automatically.
@end defopt

@defopt url-cache-directory
This variable specifies the
directory to store the cache files.  It defaults to sub-directory
@file{cache} of @code{url-configuration-directory}.
@end defopt

@c Fixme: function v. option, but neither used.
@c @findex url-cache-expired
@c @defopt url-cache-expired
@c This is a function to decide whether or not a cache entry has expired.
@c It takes two times as it parameters and returns non-@code{nil} if the
@c second time is ``too old'' when compared with the first time.
@c @end defopt

@defopt url-cache-creation-function
The cache relies on a scheme for mapping URLs to files in the cache.
This variable names a function which sets the type of cache to use.
It takes a URL as argument and returns the absolute file name of the
corresponding cache file.  The two supplied possibilities are
@code{url-cache-create-filename-using-md5} and
@code{url-cache-create-filename-human-readable}.
@end defopt

@defun url-cache-create-filename-using-md5 url
Creates a cache file name from @var{url} using MD5 hashing.
@findex md5
This is creates entries with very few cache collisions and is fast if
you have the @code{md5} function as a primitive (Emacs 21 and XEmacs).
@smallexample
(url-cache-create-filename-using-md5 "http://www.example.com/foo/bar")
  @result{} "/home/fx/.url/cache/fx/http/com/example/www/b8a35774ad20db71c7c3409a5410e74f"
@end smallexample
@end defun

@defun url-cache-create-filename-human-readable url
Creates a cache file name from @var{url} more obviously connected to
@var{url} than for @code{url-cache-create-filename-using-md5}, but
more likely to conflict with other files.
@smallexample
(url-cache-create-filename-human-readable "http://www.example.com/foo/bar")
  @result{} "/home/fx/.url/cache/fx/http/com/example/www/foo/bar"
@end smallexample
@end defun

Luc Teirlinck's avatar
Luc Teirlinck committed
781
@c Fixme: never actually used currently?
Stefan Monnier's avatar
Stefan Monnier committed
782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802
@c @defopt url-standalone-mode
@c @cindex Relying on cache
@c @cindex Cache only mode
@c @cindex Standalone mode
@c If this variable is non-@code{nil}, the library relies solely on the
@c cache for fetching documents and avoids checking if they have changed
@c on remote servers.
@c @end defopt

@c With a large cache of documents on the local disk, it can be very handy
@c when traveling, or any other time the network connection is not active
@c (a laptop with a dial-on-demand PPP connection, etc).  Emacs/W3 can rely
@c solely on its cache, and avoid checking to see if the page has changed
@c on the remote server.  In the case of a dial-on-demand PPP connection,
@c this will keep the phone line free as long as possible, only bringing up
@c the PPP connection when asking for a page that is not located in the
@c cache.  This is very useful for demonstrations as well.

@node Proxies
@section Proxies and Gatewaying

Luc Teirlinck's avatar
Luc Teirlinck committed
803
@c fixme: check/document url-ns stuff
Stefan Monnier's avatar
Stefan Monnier committed
804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834
@cindex proxy servers
@cindex proxies
@cindex environment variables
@vindex HTTP_PROXY
Proxy servers are commonly used to provide gateways through firewalls
or as caches serving some more-or-less local network.  Each protocol
(HTTP, FTP, etc.)@: can have a different gateway server.  Proxying is
conventionally configured commonly amongst different programs through
environment variables of the form @code{@var{protocol}_proxy}, where
@var{protocol} is one of the supported network protocols (@code{http},
@code{ftp} etc.).  The library recognizes such variables in either
upper or lower case.  Their values are of one of the forms:
@itemize @bullet
@item @code{@var{host}:@var{port}}
@item A full URL;
@item Simply a host name.
@end itemize

@vindex NO_PROXY
The @code{NO_PROXY} environment variable specifies URLs that should be
excluded from proxying (on servers that should be contacted directly).
This should be a comma-separated list of hostnames, domain names, or a
mixture of both.  Asterisks can be used as wildcards, but other
clients may not support that.  Domain names may be indicated by a
leading dot.  For example:
@example
NO_PROXY="*.aventail.com,home.com,.seanet.com"
@end example
@noindent says to contact all machines in the @samp{aventail.com} and
@samp{seanet.com} domains directly, as well as the machine named
@samp{home.com}.  If @code{NO_PROXY} isn't defined, @code{no_PROXY}
Luc Teirlinck's avatar
Luc Teirlinck committed
835
and @code{no_proxy} are also tried, in that order.
Stefan Monnier's avatar
Stefan Monnier committed
836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862

Proxies may also be specified directly in Lisp.

@defopt url-proxy-services
This variable is an alist of URL schemes and proxy servers that
gateway them.  The items are of the form @w{@code{(@var{scheme}
. @var{host}:@var{portnumber})}}, says that the URL @var{scheme} is
gatewayed through @var{portnumber} on the specified @var{host}.  An
exception is the pseudo scheme @code{"no_proxy"}, which is paired with
a regexp matching host names not to be proxied.  This variable is
initialized from the environment as above.

@example
(setq url-proxy-services
      '(("http"     . "proxy.aventail.com:80")
        ("no_proxy" . "^.*\\(aventail\\|seanet\\)\\.com")))
@end example
@end defopt

@node Gateways in general
@section Gateways in General
@cindex gateways
@cindex firewalls

The library provides a general gateway layer through which all
networking passes.  It can both control access to the network and
provide access through gateways in firewalls.  This may make direct
Juanma Barranquero's avatar
Juanma Barranquero committed
863
connections in some cases and pass through some sort of gateway in
Stefan Monnier's avatar
Stefan Monnier committed
864 865
others.@footnote{Proxies (which only operate over HTTP) are
implemented using this.}  The library's basic function responsible for
Juanma Barranquero's avatar
Juanma Barranquero committed
866
making connections is @code{url-open-stream}.
Stefan Monnier's avatar
Stefan Monnier committed
867 868 869 870 871 872

@defun url-open-stream name buffer host service
@cindex opening a stream
@cindex stream, opening
Open a stream to @var{host}, possibly via a gateway.  The other
arguments are as for @code{open-network-stream}.  This will not make a
Juanma Barranquero's avatar
Juanma Barranquero committed
873
connection if @code{url-gateway-unplugged} is non-@code{nil}.
Stefan Monnier's avatar
Stefan Monnier committed
874 875 876 877
@end defun

@defvar url-gateway-local-host-regexp
This is a regular expression that matches local hosts that do not
Juanma Barranquero's avatar
Juanma Barranquero committed
878
require the use of a gateway.  If @code{nil}, all connections are made
Stefan Monnier's avatar
Stefan Monnier committed
879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944
through the gateway.
@end defvar

@defvar url-gateway-method
This variable controls which gateway method is used.  It may be useful
to bind it temporarily in some applications.  It has values taken from
a list of symbols.  Possible values are:

@table @code
@item telnet
@cindex @command{telnet}
Use this method if you must first telnet and log into a gateway host,
and then run telnet from that host to connect to outside machines.

@item rlogin
@cindex @command{rlogin}
This method is identical to @code{telnet}, but uses @command{rlogin}
to log into the remote machine without having to send the username and
password over the wire every time.

@item socks
@cindex @sc{socks}
Use if the firewall has a @sc{socks} gateway running on it.  The
@sc{socks} v5 protocol is defined in RFC 1928.

@c @item ssl
@c This probably shouldn't be documented
@c Fixme: why not? -- fx

@item native
This method uses Emacs's builtin networking directly.  This is the
default.  It can be used only if there is no firewall blocking access.
@end table
@end defvar

The following variables control the gateway methods.

@defopt url-gateway-telnet-host
The gateway host to telnet to.  Once logged in there, you then telnet
out to the hosts you want to connect to.
@end defopt
@defopt url-gateway-telnet-parameters
This should be a list of parameters to pass to the @command{telnet} program.
@end defopt
@defopt url-gateway-telnet-password-prompt
This is a regular expression that matches the password prompt when
logging in.
@end defopt
@defopt url-gateway-telnet-login-prompt
This is a regular expression that matches the username prompt when
logging in.
@end defopt
@defopt url-gateway-telnet-user-name
The username to log in with.
@end defopt
@defopt url-gateway-telnet-password
The password to send when logging in.
@end defopt
@defopt url-gateway-prompt-pattern
This is a regular expression that matches the shell prompt.
@end defopt

@defopt url-gateway-rlogin-host
Host to @samp{rlogin} to before telnetting out.
@end defopt
@defopt url-gateway-rlogin-parameters
945
Parameters to pass to @samp{rsh}.
Stefan Monnier's avatar
Stefan Monnier committed
946 947 948 949 950 951 952 953 954 955 956 957 958 959
@end defopt
@defopt url-gateway-rlogin-user-name
User name to use when logging in to the gateway.
@end defopt
@defopt url-gateway-prompt-pattern
This is a regular expression that matches the shell prompt.
@end defopt

@defopt socks-server
This specifies the default server, it takes the form
@w{@code{("Default server" @var{server} @var{port} @var{version})}}
where @var{version} can be either 4 or 5.
@end defopt
@defvar socks-password
Luc Teirlinck's avatar
Luc Teirlinck committed
960
If this is @code{nil} then you will be asked for the password,
Stefan Monnier's avatar
Stefan Monnier committed
961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999
otherwise it will be used as the password for authenticating you to
the @sc{socks} server.
@end defvar
@defvar socks-username
This is the username to use when authenticating yourself to the
@sc{socks} server.  By default this is your login name.
@end defvar
@defvar socks-timeout
This controls how long, in seconds, to wait for responses from the
@sc{socks} server; it is 5 by default.
@end defvar
@c fixme: these have been effectively commented-out in the code
@c @defopt socks-server-aliases
@c This a list of server aliases.  It is a list of aliases of the form
@c @var{(alias hostname port version)}.
@c @end defopt
@c @defopt socks-network-aliases
@c This a list of network aliases.  Each entry in the list takes the form
@c @var{(alias (network))} where @var{alias} is a string that names the
@c @var{network}.  The networks can contain a pair (not a dotted pair) of
@c @sc{ip} addresses which specify a range of @sc{ip} addresses, an @sc{ip}
@c address and a netmask, a domain name or a unique hostname or @sc{ip}
@c address.
@c @end defopt
@c @defopt socks-redirection-rules
@c This a list of redirection rules.  Each rule take the form
@c @var{(Destination network Connection type)} where @var{Destination
@c network} is a network alias from @code{socks-network-aliases} and
@c @var{Connection type} can be @code{nil} in which case a direct
@c connection is used, or it can be an alias from
@c @code{socks-server-aliases} in which case that server is used as a
@c proxy.
@c @end defopt
@defopt socks-nslookup-program
@cindex @command{nslookup}
This the @samp{nslookup} program.  It is @code{"nslookup"} by default.
@end defopt

@menu
Juanma Barranquero's avatar
Juanma Barranquero committed
1000
* Suppressing network connections::
Stefan Monnier's avatar
Stefan Monnier committed
1001
@end menu
Luc Teirlinck's avatar
Luc Teirlinck committed
1002
@c * Broken hostname resolution::
Stefan Monnier's avatar
Stefan Monnier committed
1003

Juanma Barranquero's avatar
Juanma Barranquero committed
1004 1005
@node Suppressing network connections
@subsection Suppressing Network Connections
Stefan Monnier's avatar
Stefan Monnier committed
1006

Juanma Barranquero's avatar
Juanma Barranquero committed
1007 1008
@cindex network connections, suppressing
@cindex suppressing network connections
Stefan Monnier's avatar
Stefan Monnier committed
1009 1010 1011
@cindex bugs, HTML
@cindex HTML `bugs'
In some circumstances it is desirable to suppress making network
Juanma Barranquero's avatar
Juanma Barranquero committed
1012
connections.  A typical case is when rendering HTML in a mail user
Stefan Monnier's avatar
Stefan Monnier committed
1013 1014 1015 1016 1017 1018
agent, when external URLs should not be activated, particularly to
avoid `bugs' which `call home' by fetch single-pixel images and the
like.  To arrange this, bind the following variable for the duration
of such processing.

@defvar url-gateway-unplugged
Juanma Barranquero's avatar
Juanma Barranquero committed
1019
If this variable is non-@code{nil} new network connections are never
Stefan Monnier's avatar
Stefan Monnier committed
1020 1021 1022 1023 1024 1025 1026 1027 1028 1029
opened by the URL library.
@end defvar

@c @node Broken hostname resolution
@c @subsection Broken Hostname Resolution

@c @cindex hostname resolver
@c @cindex resolver, hostname
@c Some C libraries do not include the hostname resolver routines in
@c their static libraries.  If Emacs was linked statically, and was not
Luc Teirlinck's avatar
Luc Teirlinck committed
1030
@c linked with the resolver libraries, it will not be able to get to any
Stefan Monnier's avatar
Stefan Monnier committed
1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071
@c machines off the local network.  This is characterized by being able
@c to reach someplace with a raw ip number, but not its hostname
@c (@url{http://129.79.254.191/} works, but
@c @url{http://www.cs.indiana.edu/} doesn't).  This used to happen on
@c SunOS4 and Ultrix, but is now probably now rare.  If Emacs can't be
@c rebuilt linked against the resolver library, it can use the external
@c @command{nslookup} program instead.

@c @defopt url-gateway-broken-resolution
@c @cindex @code{nslookup} program
@c @cindex program, @code{nslookup}
@c If non-@code{nil}, this variable says to use the program specified by
@c @code{url-gateway-nslookup-program} program to do hostname resolution.
@c @end defopt

@c @defopt url-gateway-nslookup-program
@c The name of the program to do hostname lookup if Emacs can't do it
@c directly.  This program should expect a single argument on the command
@c line---the hostname to resolve---and should produce output similar to
@c the standard Unix @command{nslookup} program:
@c @example
@c Name: www.cs.indiana.edu
@c Address: 129.79.254.191
@c @end example
@c @end defopt

@node History
@section History

The library can maintain a global history list tracking URLs accessed.
URL completion can be done from it.  The history mechanism is set up
@findex url-do-setup
automatically via @code{url-do-setup} when it is configured to be on.
Note that the size of the history list is currently not limited.

@vindex url-history-hash-table
The history `list' is actually a hash table,
@code{url-history-hash-table}.  It contains access times keyed by URL
strings.  The times are in the format returned by @code{current-time}.

@defun url-history-update-url url time
Luc Teirlinck's avatar
Luc Teirlinck committed
1072 1073
This function updates the history table with an entry for @var{url}
accessed at the given @var{time}.
Stefan Monnier's avatar
Stefan Monnier committed
1074 1075 1076 1077
@end defun

@defopt url-history-track
If non-@code{nil}, the library will keep track of all the URLs
1078
accessed.  If it is @code{t}, the list is saved to disk at the end of
Stefan Monnier's avatar
Stefan Monnier committed
1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163
each Emacs session.  The default is @code{nil}.
@end defopt

@defopt url-history-file
The file storing the history list between sessions.  It defaults to
@file{history} in @code{url-configuration-directory}.
@end defopt

@defopt url-history-save-interval
@findex url-history-setup-save-timer
The number of seconds between automatic saves of the history list.
Default is one hour.  Note that if you change this variable directly,
rather than using Custom, after @code{url-do-setup} has been run, you
need to run the function @code{url-history-setup-save-timer}.
@end defopt

@defun url-history-parse-history &optional fname
Parses the history file @var{fname} (default @code{url-history-file})
and sets up the history list.
@end defun

@defun url-history-save-history &optional fname
Saves the current history to file @var{fname} (default
@code{url-history-file}).
@end defun

@defun url-completion-function string predicate function
You can use this function to do completion of URLs from the history.
@end defun

@node Customization
@chapter Customization

@section Environment Variables

@cindex environment variables
The following environment variables affect the library's operation at
startup.

@table @code
@item TMPDIR
@vindex TMPDIR
@vindex url-temporary-directory
If this is defined, @var{url-temporary-directory} is initialized from
it.
@end table

@section General User Options

The following user options, settable with Customize, affect the
general operation of the package.

@defopt url-debug
@cindex debugging
Specifies the types of debug messages the library which are logged to
the @code{*URL-DEBUG*} buffer.
@code{t} means log all messages.
A number means log all messages and show them with @code{message}.
If may also be a list of the types of messages to be logged.
@end defopt
@defopt url-personal-mail-address
@end defopt
@defopt url-privacy-level
@end defopt
@defopt url-uncompressor-alist
@end defopt
@defopt url-passwd-entry-func
@end defopt
@defopt url-standalone-mode
@end defopt
@defopt url-bad-port-list
@end defopt
@defopt url-max-password-attempts
@end defopt
@defopt url-temporary-directory
@end defopt
@defopt url-show-status
@end defopt
@defopt url-confirmation-func
The function to use for asking yes or no functions.  This is normally
either @code{y-or-n-p} or @code{yes-or-no-p}, but could be another
function taking a single argument (the prompt) and returning @code{t}
only if an affirmative answer is given.
@end defopt
@defopt url-gateway-method
Luc Teirlinck's avatar
Luc Teirlinck committed
1164
@c fixme: describe gatewaying
Juanma Barranquero's avatar
Juanma Barranquero committed
1165
A symbol specifying the type of gateway support to use for connections
Stefan Monnier's avatar
Stefan Monnier committed
1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196
from the local machine.  The supported methods are:

@table @code
@item telnet
Run telnet in a subprocess to connect;
@item rlogin
Rlogin to another machine to connect;
@item socks
Connect through a socks server;
@item ssl
Connect with SSL;
@item native
Connect directly.
@end table
@end defopt

@node Function Index
@unnumbered Command and Function Index
@printindex fn

@node Variable Index
@unnumbered Variable Index
@printindex vr

@node Concept Index
@unnumbered Concept Index
@printindex cp

@setchapternewpage odd
@contents
@bye
1197 1198 1199 1200

@ignore
   arch-tag: c96be356-7e2d-4196-bcda-b13246c5c3f0
@end ignore