nonascii.texi 60.4 KB
Newer Older
Richard M. Stallman's avatar
Richard M. Stallman committed
1 2
@c -*-texinfo-*-
@c This is part of the GNU Emacs Lisp Reference Manual.
3
@c Copyright (C) 1998, 1999 Free Software Foundation, Inc.
Richard M. Stallman's avatar
Richard M. Stallman committed
4 5 6
@c See the file elisp.texi for copying conditions.
@setfilename ../info/characters
@node Non-ASCII Characters, Searching and Matching, Text, Top
7
@chapter Non-@acronym{ASCII} Characters
Richard M. Stallman's avatar
Richard M. Stallman committed
8
@cindex multibyte characters
9
@cindex non-@acronym{ASCII} characters
Richard M. Stallman's avatar
Richard M. Stallman committed
10

11
  This chapter covers the special issues relating to non-@acronym{ASCII}
Richard M. Stallman's avatar
Richard M. Stallman committed
12 13 14
characters and how they are stored in strings and buffers.

@menu
15 16 17 18 19 20 21 22 23 24 25 26 27
* Text Representations::    Unibyte and multibyte representations
* Converting Representations::  Converting unibyte to multibyte and vice versa.
* Selecting a Representation::  Treating a byte sequence as unibyte or multi.
* Character Codes::         How unibyte and multibyte relate to
                                codes of individual characters.
* Character Sets::          The space of possible characters codes
                                is divided into various character sets.
* Chars and Bytes::         More information about multibyte encodings.
* Splitting Characters::    Converting a character to its byte sequence.
* Scanning Charsets::       Which character sets are used in a buffer?
* Translation of Characters::   Translation tables are used for conversion.
* Coding Systems::          Coding systems are conversions for saving files.
* Input Methods::           Input methods allow users to enter various
Richard M. Stallman's avatar
Richard M. Stallman committed
28
                                non-ASCII characters without special keyboards.
29
* Locales::                 Interacting with the POSIX locale.
Richard M. Stallman's avatar
Richard M. Stallman committed
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
@end menu

@node Text Representations
@section Text Representations
@cindex text representations

  Emacs has two @dfn{text representations}---two ways to represent text
in a string or buffer.  These are called @dfn{unibyte} and
@dfn{multibyte}.  Each string, and each buffer, uses one of these two
representations.  For most purposes, you can ignore the issue of
representations, because Emacs converts text between them as
appropriate.  Occasionally in Lisp programming you will need to pay
attention to the difference.

@cindex unibyte text
  In unibyte representation, each character occupies one byte and
therefore the possible character codes range from 0 to 255.  Codes 0
47 48
through 127 are @acronym{ASCII} characters; the codes from 128 through 255
are used for one non-@acronym{ASCII} character set (you can choose which
49
character set by setting the variable @code{nonascii-insert-offset}).
Richard M. Stallman's avatar
Richard M. Stallman committed
50 51 52

@cindex leading code
@cindex multibyte text
53
@cindex trailing codes
Richard M. Stallman's avatar
Richard M. Stallman committed
54 55 56 57
  In multibyte representation, a character may occupy more than one
byte, and as a result, the full range of Emacs character codes can be
stored.  The first byte of a multibyte character is always in the range
128 through 159 (octal 0200 through 0237).  These values are called
58 59
@dfn{leading codes}.  The second and subsequent bytes of a multibyte
character are always in the range 160 through 255 (octal 0240 through
60
0377); these values are @dfn{trailing codes}.
Richard M. Stallman's avatar
Richard M. Stallman committed
61

Gerd Moellmann's avatar
Gerd Moellmann committed
62
  Some sequences of bytes are not valid in multibyte text: for example,
Gerd Moellmann's avatar
Gerd Moellmann committed
63 64 65 66
a single isolated byte in the range 128 through 159 is not allowed.  But
character codes 128 through 159 can appear in multibyte text,
represented as two-byte sequences.  All the character codes 128 through
255 are possible (though slightly abnormal) in multibyte text; they
Gerd Moellmann's avatar
Gerd Moellmann committed
67 68
appear in multibyte buffers and strings when you do explicit encoding
and decoding (@pxref{Explicit Encoding}).
69

Richard M. Stallman's avatar
Richard M. Stallman committed
70 71
  In a buffer, the buffer-local value of the variable
@code{enable-multibyte-characters} specifies the representation used.
Karl Heuer's avatar
Karl Heuer committed
72 73
The representation for a string is determined and recorded in the string
when the string is constructed.
Richard M. Stallman's avatar
Richard M. Stallman committed
74 75 76 77 78 79

@defvar enable-multibyte-characters
This variable specifies the current buffer's text representation.
If it is non-@code{nil}, the buffer contains multibyte text; otherwise,
it contains unibyte text.

80 81
You cannot set this variable directly; instead, use the function
@code{set-buffer-multibyte} to change a buffer's representation.
Richard M. Stallman's avatar
Richard M. Stallman committed
82 83 84
@end defvar

@defvar default-enable-multibyte-characters
85
This variable's value is entirely equivalent to @code{(default-value
Richard M. Stallman's avatar
Richard M. Stallman committed
86
'enable-multibyte-characters)}, and setting this variable changes that
87 88 89 90
default value.  Setting the local binding of
@code{enable-multibyte-characters} in a specific buffer is not allowed,
but changing the default value is supported, and it is a reasonable
thing to do, because it has no effect on existing buffers.
Richard M. Stallman's avatar
Richard M. Stallman committed
91 92 93 94 95

The @samp{--unibyte} command line option does its job by setting the
default value to @code{nil} early in startup.
@end defvar

96 97 98
@defun position-bytes position
@tindex position-bytes
Return the byte-position corresponding to buffer position @var{position}
99 100
in the current buffer.  If @var{position} is out of range, the value
is @code{nil}.
101 102 103 104 105
@end defun

@defun byte-to-position byte-position
@tindex byte-to-position
Return the buffer position corresponding to byte-position
106 107
@var{byte-position} in the current buffer.  If @var{byte-position} is
out of range, the value is @code{nil}.
108 109
@end defun

Richard M. Stallman's avatar
Richard M. Stallman committed
110
@defun multibyte-string-p string
111
Return @code{t} if @var{string} is a multibyte string.
Richard M. Stallman's avatar
Richard M. Stallman committed
112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136
@end defun

@node Converting Representations
@section Converting Text Representations

  Emacs can convert unibyte text to multibyte; it can also convert
multibyte text to unibyte, though this conversion loses information.  In
general these conversions happen when inserting text into a buffer, or
when putting text from several strings together in one string.  You can
also explicitly convert a string's contents to either representation.

  Emacs chooses the representation for a string based on the text that
it is constructed from.  The general rule is to convert unibyte text to
multibyte text when combining it with other multibyte text, because the
multibyte representation is more general and can hold whatever
characters the unibyte text has.

  When inserting text into a buffer, Emacs converts the text to the
buffer's representation, as specified by
@code{enable-multibyte-characters} in that buffer.  In particular, when
you insert multibyte text into a unibyte buffer, Emacs converts the text
to unibyte, even though this conversion cannot in general preserve all
the characters that might be in the multibyte text.  The other natural
alternative, to convert the buffer contents to multibyte, is not
acceptable because the buffer's representation is a choice made by the
137
user that cannot be overridden automatically.
Richard M. Stallman's avatar
Richard M. Stallman committed
138

139
  Converting unibyte text to multibyte text leaves @acronym{ASCII} characters
Gerd Moellmann's avatar
Gerd Moellmann committed
140
unchanged, and likewise character codes 128 through 159.  It converts
141
the non-@acronym{ASCII} codes 160 through 255 by adding the value
Gerd Moellmann's avatar
Gerd Moellmann committed
142 143 144 145
@code{nonascii-insert-offset} to each character code.  By setting this
variable, you specify which character set the unibyte characters
correspond to (@pxref{Character Sets}).  For example, if
@code{nonascii-insert-offset} is 2048, which is @code{(- (make-char
146
'latin-iso8859-1) 128)}, then the unibyte non-@acronym{ASCII} characters
Gerd Moellmann's avatar
Gerd Moellmann committed
147 148
correspond to Latin 1.  If it is 2688, which is @code{(- (make-char
'greek-iso8859-7) 128)}, then they correspond to Greek letters.
Richard M. Stallman's avatar
Richard M. Stallman committed
149

150 151 152 153 154 155
  Converting multibyte text to unibyte is simpler: it discards all but
the low 8 bits of each character code.  If @code{nonascii-insert-offset}
has a reasonable value, corresponding to the beginning of some character
set, this conversion is the inverse of the other: converting unibyte
text to multibyte and back to unibyte reproduces the original unibyte
text.
Richard M. Stallman's avatar
Richard M. Stallman committed
156 157

@defvar nonascii-insert-offset
158
This variable specifies the amount to add to a non-@acronym{ASCII} character
Richard M. Stallman's avatar
Richard M. Stallman committed
159
when converting unibyte text to multibyte.  It also applies when
160
@code{self-insert-command} inserts a character in the unibyte
161
non-@acronym{ASCII} range, 128 through 255.  However, the functions
Kenichi Handa's avatar
Kenichi Handa committed
162
@code{insert} and @code{insert-char} do not perform this conversion.
Richard M. Stallman's avatar
Richard M. Stallman committed
163 164

The right value to use to select character set @var{cs} is @code{(-
165
(make-char @var{cs}) 128)}.  If the value of
Richard M. Stallman's avatar
Richard M. Stallman committed
166 167 168 169
@code{nonascii-insert-offset} is zero, then conversion actually uses the
value for the Latin 1 character set, rather than zero.
@end defvar

170
@defvar nonascii-translation-table
Richard M. Stallman's avatar
Richard M. Stallman committed
171 172 173
This variable provides a more general alternative to
@code{nonascii-insert-offset}.  You can use it to specify independently
how to translate each code in the range of 128 through 255 into a
Kenichi Handa's avatar
Kenichi Handa committed
174
multibyte character.  The value should be a char-table, or @code{nil}.
175
If this is non-@code{nil}, it overrides @code{nonascii-insert-offset}.
Richard M. Stallman's avatar
Richard M. Stallman committed
176 177
@end defvar

178 179 180
The next three functions either return the argument @var{string}, or a
newly created string with no text properties.

Richard M. Stallman's avatar
Richard M. Stallman committed
181 182
@defun string-make-unibyte string
This function converts the text of @var{string} to unibyte
183
representation, if it isn't already, and returns the result.  If
184 185 186 187 188
@var{string} is a unibyte string, it is returned unchanged.  Multibyte
character codes are converted to unibyte according to
@code{nonascii-translation-table} or, if that is @code{nil}, using
@code{nonascii-insert-offset}.  If the lookup in the translation table
fails, this function takes just the low 8 bits of each character.
Richard M. Stallman's avatar
Richard M. Stallman committed
189 190 191 192
@end defun

@defun string-make-multibyte string
This function converts the text of @var{string} to multibyte
193
representation, if it isn't already, and returns the result.  If
194 195 196 197 198 199 200 201 202
@var{string} is a multibyte string or consists entirely of
@acronym{ASCII} characters, it is returned unchanged.  In particular,
if @var{string} is unibyte and entirely @acronym{ASCII}, the returned
string is unibyte.  (When the characters are all @acronym{ASCII},
Emacs primitives will treat the string the same way whether it is
unibyte or multibyte.)  If @var{string} is unibyte and contains
non-@acronym{ASCII} characters, the function
@code{unibyte-char-to-multibyte} is used to convert each unibyte
character to a multibyte character.
Richard M. Stallman's avatar
Richard M. Stallman committed
203 204
@end defun

205 206
@defun string-to-multibyte string
This function returns a multibyte string containing the same sequence
207 208 209 210
of character codes as @var{string}.  Unlike
@code{string-make-multibyte}, this function unconditionally returns a
multibyte string.  If @var{string} is a multibyte string, it is
returned unchanged.
211 212
@end defun

213 214 215 216 217 218 219 220 221 222 223 224
@defun multibyte-char-to-unibyte char
This convert the multibyte character @var{char} to a unibyte
character, based on @code{nonascii-translation-table} and
@code{nonascii-insert-offset}.
@end defun

@defun unibyte-char-to-multibyte char
This convert the unibyte character @var{char} to a multibyte
character, based on @code{nonascii-translation-table} and
@code{nonascii-insert-offset}.
@end defun

Richard M. Stallman's avatar
Richard M. Stallman committed
225 226 227 228 229 230 231 232 233 234 235 236 237 238 239
@node Selecting a Representation
@section Selecting a Representation

  Sometimes it is useful to examine an existing buffer or string as
multibyte when it was unibyte, or vice versa.

@defun set-buffer-multibyte multibyte
Set the representation type of the current buffer.  If @var{multibyte}
is non-@code{nil}, the buffer becomes multibyte.  If @var{multibyte}
is @code{nil}, the buffer becomes unibyte.

This function leaves the buffer contents unchanged when viewed as a
sequence of bytes.  As a consequence, it can change the contents viewed
as characters; a sequence of two bytes which is treated as one character
in multibyte representation will count as two characters in unibyte
Kenichi Handa's avatar
Kenichi Handa committed
240 241 242 243
representation.  Character codes 128 through 159 are an exception.  They
are represented by one byte in a unibyte buffer, but when the buffer is
set to multibyte, they are converted to two-byte sequences, and vice
versa.
Richard M. Stallman's avatar
Richard M. Stallman committed
244 245 246

This function sets @code{enable-multibyte-characters} to record which
representation is in use.  It also adjusts various data in the buffer
247 248
(including overlays, text properties and markers) so that they cover the
same text as they did before.
249 250 251 252

You cannot use @code{set-buffer-multibyte} on an indirect buffer,
because indirect buffers always inherit the representation of the
base buffer.
Richard M. Stallman's avatar
Richard M. Stallman committed
253 254 255 256 257 258 259
@end defun

@defun string-as-unibyte string
This function returns a string with the same bytes as @var{string} but
treating each byte as a character.  This means that the value may have
more characters than @var{string} has.

260
If @var{string} is already a unibyte string, then the value is
Dave Love's avatar
Dave Love committed
261 262
@var{string} itself.  Otherwise it is a newly created string, with no
text properties.  If @var{string} is multibyte, any characters it
263
contains of charset @code{eight-bit-control} or @code{eight-bit-graphic}
Dave Love's avatar
Dave Love committed
264
are converted to the corresponding single byte.
Richard M. Stallman's avatar
Richard M. Stallman committed
265 266 267 268 269 270 271
@end defun

@defun string-as-multibyte string
This function returns a string with the same bytes as @var{string} but
treating each multibyte sequence as one character.  This means that the
value may have fewer characters than @var{string} has.

272
If @var{string} is already a multibyte string, then the value is
Dave Love's avatar
Dave Love committed
273 274 275
@var{string} itself.  Otherwise it is a newly created string, with no
text properties.  If @var{string} is unibyte and contains any individual
8-bit bytes (i.e.@: not part of a multibyte form), they are converted to
276 277
the corresponding multibyte character of charset @code{eight-bit-control}
or @code{eight-bit-graphic}.
Richard M. Stallman's avatar
Richard M. Stallman committed
278 279 280 281 282 283 284 285 286 287
@end defun

@node Character Codes
@section Character Codes
@cindex character codes

  The unibyte and multibyte text representations use different character
codes.  The valid character codes for unibyte representation range from
0 to 255---the values that can fit in one byte.  The valid character
codes for multibyte representation range from 0 to 524287, but not all
Gerd Moellmann's avatar
Gerd Moellmann committed
288
values in that range are valid.  The values 128 through 255 are not
Gerd Moellmann's avatar
Gerd Moellmann committed
289
entirely proper in multibyte text, but they can occur if you do explicit
Gerd Moellmann's avatar
Gerd Moellmann committed
290
encoding and decoding (@pxref{Explicit Encoding}).  Some other character
291
codes cannot occur at all in multibyte text.  Only the @acronym{ASCII} codes
Gerd Moellmann's avatar
Gerd Moellmann committed
292
0 through 127 are completely legitimate in both representations.
Richard M. Stallman's avatar
Richard M. Stallman committed
293

Kenichi Handa's avatar
Kenichi Handa committed
294
@defun char-valid-p charcode &optional genericp
Richard M. Stallman's avatar
Richard M. Stallman committed
295 296 297 298 299 300 301 302 303 304 305
This returns @code{t} if @var{charcode} is valid for either one of the two
text representations.

@example
(char-valid-p 65)
     @result{} t
(char-valid-p 256)
     @result{} nil
(char-valid-p 2248)
     @result{} t
@end example
Kenichi Handa's avatar
Kenichi Handa committed
306

Richard M. Stallman's avatar
Richard M. Stallman committed
307
If the optional argument @var{genericp} is non-@code{nil}, this
308 309
function also returns @code{t} if @var{charcode} is a generic
character (@pxref{Splitting Characters}).
Richard M. Stallman's avatar
Richard M. Stallman committed
310 311 312 313 314 315 316 317 318 319 320 321 322
@end defun

@node Character Sets
@section Character Sets
@cindex character sets

  Emacs classifies characters into various @dfn{character sets}, each of
which has a name which is a symbol.  Each character belongs to one and
only one character set.

  In general, there is one character set for each distinct script.  For
example, @code{latin-iso8859-1} is one character set,
@code{greek-iso8859-7} is another, and @code{ascii} is another.  An
323 324
Emacs character set can hold at most 9025 characters; therefore, in some
cases, characters that would logically be grouped together are split
325 326 327
into several character sets.  For example, one set of Chinese
characters, generally known as Big 5, is divided into two Emacs
character sets, @code{chinese-big5-1} and @code{chinese-big5-2}.
Richard M. Stallman's avatar
Richard M. Stallman committed
328

329 330
  @acronym{ASCII} characters are in character set @code{ascii}.  The
non-@acronym{ASCII} characters 128 through 159 are in character set
Gerd Moellmann's avatar
Gerd Moellmann committed
331 332 333
@code{eight-bit-control}, and codes 160 through 255 are in character set
@code{eight-bit-graphic}.

Richard M. Stallman's avatar
Richard M. Stallman committed
334
@defun charsetp object
335
Returns @code{t} if @var{object} is a symbol that names a character set,
Richard M. Stallman's avatar
Richard M. Stallman committed
336 337 338
@code{nil} otherwise.
@end defun

339 340 341 342
@defvar charset-list
The value is a list of all defined character set names.
@end defvar

Richard M. Stallman's avatar
Richard M. Stallman committed
343
@defun charset-list
344 345
This function returns the value of @code{charset-list}.  It is only
provided for backward compatibility.
Richard M. Stallman's avatar
Richard M. Stallman committed
346 347 348
@end defun

@defun char-charset character
349
This function returns the name of the character set that @var{character}
350 351
belongs to, or the symbol @code{unknown} if @var{character} is not a
valid character.
Richard M. Stallman's avatar
Richard M. Stallman committed
352 353
@end defun

354 355 356 357 358
@defun charset-plist charset
@tindex charset-plist
This function returns the charset property list of the character set
@var{charset}.  Although @var{charset} is a symbol, this is not the same
as the property list of that symbol.  Charset properties are used for
359
special purposes within Emacs.
360 361
@end defun

Richard M. Stallman's avatar
Richard M. Stallman committed
362 363 364 365
@node Chars and Bytes
@section Characters and Bytes
@cindex bytes and characters

366 367
@cindex introduction sequence
@cindex dimension (of character set)
Richard M. Stallman's avatar
Richard M. Stallman committed
368
  In multibyte representation, each character occupies one or more
369
bytes.  Each character set has an @dfn{introduction sequence}, which is
370 371
normally one or two bytes long.  (Exception: the @code{ascii} character
set and the @code{eight-bit-graphic} character set have a zero-length
Kenichi Handa's avatar
Kenichi Handa committed
372 373 374 375 376 377
introduction sequence.)  The introduction sequence is the beginning of
the byte sequence for any character in the character set.  The rest of
the character's bytes distinguish it from the other characters in the
same character set.  Depending on the character set, there are either
one or two distinguishing bytes; the number of such bytes is called the
@dfn{dimension} of the character set.
378 379

@defun charset-dimension charset
380 381 382 383 384 385 386 387
This function returns the dimension of @var{charset}; at present, the
dimension is always 1 or 2.
@end defun

@defun charset-bytes charset
@tindex charset-bytes
This function returns the number of bytes used to represent a character
in character set @var{charset}.
388 389 390 391 392 393
@end defun

  This is the simplest way to determine the byte length of a character
set's introduction sequence:

@example
394
(- (charset-bytes @var{charset})
395 396 397 398 399 400 401 402 403
   (charset-dimension @var{charset}))
@end example

@node Splitting Characters
@section Splitting Characters

  The functions in this section convert between characters and the byte
values used to represent them.  For most purposes, there is no need to
be concerned with the sequence of bytes used to represent a character,
404
because Emacs translates automatically when necessary.
Richard M. Stallman's avatar
Richard M. Stallman committed
405 406 407

@defun split-char character
Return a list containing the name of the character set of
408 409 410
@var{character}, followed by one or two byte values (integers) which
identify @var{character} within that character set.  The number of byte
values is the character set's dimension.
Richard M. Stallman's avatar
Richard M. Stallman committed
411

412 413 414
If @var{character} is invalid as a character code, @code{split-char}
returns a list consisting of the symbol @code{unknown} and @var{character}.

Richard M. Stallman's avatar
Richard M. Stallman committed
415 416 417 418 419
@example
(split-char 2248)
     @result{} (latin-iso8859-1 72)
(split-char 65)
     @result{} (ascii 65)
Kenichi Handa's avatar
Kenichi Handa committed
420 421
(split-char 128)
     @result{} (eight-bit-control 128)
Richard M. Stallman's avatar
Richard M. Stallman committed
422 423 424
@end example
@end defun

Dave Love's avatar
Dave Love committed
425 426 427 428 429 430
@defun make-char charset &optional code1 code2
This function returns the character in character set @var{charset} whose
position codes are @var{code1} and @var{code2}.  This is roughly the
inverse of @code{split-char}.  Normally, you should specify either one
or both of @var{code1} and @var{code2} according to the dimension of
@var{charset}.  For example,
Richard M. Stallman's avatar
Richard M. Stallman committed
431 432 433 434 435

@example
(make-char 'latin-iso8859-1 72)
     @result{} 2248
@end example
436 437 438 439 440

Actually, the eighth bit of both @var{code1} and @var{code2} is zeroed
before they are used to index @var{charset}.  Thus you may use, for
instance, an ISO 8859 character code rather than subtracting 128, as
is necessary to index the corresponding Emacs charset.
Richard M. Stallman's avatar
Richard M. Stallman committed
441 442
@end defun

443 444 445 446 447 448 449 450 451 452 453 454 455 456
@cindex generic characters
  If you call @code{make-char} with no @var{byte-values}, the result is
a @dfn{generic character} which stands for @var{charset}.  A generic
character is an integer, but it is @emph{not} valid for insertion in the
buffer as a character.  It can be used in @code{char-table-range} to
refer to the whole character set (@pxref{Char-Tables}).
@code{char-valid-p} returns @code{nil} for generic characters.
For example:

@example
(make-char 'latin-iso8859-1)
     @result{} 2176
(char-valid-p 2176)
     @result{} nil
Kenichi Handa's avatar
Kenichi Handa committed
457 458
(char-valid-p 2176 t)
     @result{} t
459 460 461 462
(split-char 2176)
     @result{} (latin-iso8859-1 0)
@end example

463 464
The character sets @code{ascii}, @code{eight-bit-control}, and
@code{eight-bit-graphic} don't have corresponding generic characters.  If
Dave Love's avatar
Dave Love committed
465 466 467
@var{charset} is one of them and you don't supply @var{code1},
@code{make-char} returns the character code corresponding to the
smallest code in @var{charset}.
Kenichi Handa's avatar
Kenichi Handa committed
468

469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485
@node Scanning Charsets
@section Scanning for Character Sets

  Sometimes it is useful to find out which character sets appear in a
part of a buffer or a string.  One use for this is in determining which
coding systems (@pxref{Coding Systems}) are capable of representing all
of the text in question.

@defun find-charset-region beg end &optional translation
This function returns a list of the character sets that appear in the
current buffer between positions @var{beg} and @var{end}.

The optional argument @var{translation} specifies a translation table to
be used in scanning the text (@pxref{Translation of Characters}).  If it
is non-@code{nil}, then each character in the region is translated
through this table, and the value returned describes the translated
characters instead of the characters actually in the buffer.
Gerd Moellmann's avatar
Gerd Moellmann committed
486
@end defun
487 488

@defun find-charset-string string &optional translation
489 490 491 492
This function returns a list of the character sets that appear in the
string @var{string}.  It is just like @code{find-charset-region}, except
that it applies to the contents of @var{string} instead of part of the
current buffer.
493 494 495 496 497 498 499
@end defun

@node Translation of Characters
@section Translation of Characters
@cindex character translation tables
@cindex translation tables

500 501 502 503 504
  A @dfn{translation table} is a char-table that specifies a mapping
of characters into characters.  These tables are used in encoding and
decoding, and for other purposes.  Some coding systems specify their
own particular translation tables; there are also default translation
tables which apply to all other coding systems.
505

506 507
@defun make-translation-table &rest translations
This function returns a translation table based on the argument
Dave Love's avatar
Dave Love committed
508 509 510
@var{translations}.  Each element of @var{translations} should be a
list of elements of the form @code{(@var{from} . @var{to})}; this says
to translate the character @var{from} into @var{to}.
511

Dave Love's avatar
Dave Love committed
512 513 514 515 516
The arguments and the forms in each argument are processed in order,
and if a previous form already translates @var{to} to some other
character, say @var{to-alt}, @var{from} is also translated to
@var{to-alt}.

517 518 519
You can also map one whole character set into another character set with
the same dimension.  To do this, you specify a generic character (which
designates a character set) for @var{from} (@pxref{Splitting Characters}).
520 521 522 523 524 525 526
In this case, if @var{to} is also a generic character, its character
set should have the same dimension as @var{from}'s.  Then the
translation table translates each character of @var{from}'s character
set into the corresponding character of @var{to}'s character set.  If
@var{from} is a generic character and @var{to} is an ordinary
character, then the translation table translates every character of
@var{from}'s character set into @var{to}.
527 528 529 530
@end defun

  In decoding, the translation table's translations are applied to the
characters that result from ordinary decoding.  If a coding system has
531 532 533 534 535 536 537
property @code{translation-table-for-decode}, that specifies the
translation table to use.  (This is a property of the coding system,
as returned by @code{coding-system-get}, not a property of the symbol
that is the coding system's name. @xref{Coding System Basics,, Basic
Concepts of Coding Systems}.)  Otherwise, if
@code{standard-translation-table-for-decode} is non-@code{nil},
decoding uses that table.
538 539 540 541

  In encoding, the translation table's translations are applied to the
characters in the buffer, and the result of translation is actually
encoded.  If a coding system has property
542 543
@code{translation-table-for-encode}, that specifies the translation
table to use.  Otherwise the variable
544 545
@code{standard-translation-table-for-encode} specifies the translation
table.
546

547
@defvar standard-translation-table-for-decode
548 549 550 551
This is the default translation table for decoding, for
coding systems that don't specify any other translation table.
@end defvar

552
@defvar standard-translation-table-for-encode
553 554 555 556
This is the default translation table for encoding, for
coding systems that don't specify any other translation table.
@end defvar

557 558
@defvar translation-table-for-input
Self-inserting characters are translated through this translation
559 560
table before they are inserted.  This variable automatically becomes
buffer-local when set.
561 562
@end defvar

Richard M. Stallman's avatar
Richard M. Stallman committed
563 564 565 566 567 568 569 570 571
@node Coding Systems
@section Coding Systems

@cindex coding system
  When Emacs reads or writes a file, and when Emacs sends text to a
subprocess or receives text from a subprocess, it normally performs
character code conversion and end-of-line conversion as specified
by a particular @dfn{coding system}.

572 573
  How to define a coding system is an arcane matter, and is not
documented here.
574

575
@menu
576 577 578 579 580 581 582 583 584 585 586
* Coding System Basics::        Basic concepts.
* Encoding and I/O::            How file I/O functions handle coding systems.
* Lisp and Coding Systems::     Functions to operate on coding system names.
* User-Chosen Coding Systems::  Asking the user to choose a coding system.
* Default Coding Systems::      Controlling the default choices.
* Specifying Coding Systems::   Requesting a particular coding system
                                    for a single file operation.
* Explicit Encoding::           Encoding or decoding text without doing I/O.
* Terminal I/O Encoding::       Use of encoding for terminal I/O.
* MS-DOS File Types::           How DOS "text" and "binary" files
                                    relate to coding systems.
587 588 589 590 591
@end menu

@node Coding System Basics
@subsection Basic Concepts of Coding Systems

Richard M. Stallman's avatar
Richard M. Stallman committed
592 593 594 595 596 597 598 599 600 601 602
@cindex character code conversion
  @dfn{Character code conversion} involves conversion between the encoding
used inside Emacs and some other encoding.  Emacs supports many
different encodings, in that it can convert to and from them.  For
example, it can convert text to or from encodings such as Latin 1, Latin
2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022.  In some
cases, Emacs supports several alternative encodings for the same
characters; for example, there are three coding systems for the Cyrillic
(Russian) alphabet: ISO, Alternativnyj, and KOI8.

  Most coding systems specify a particular character code for
603 604
conversion, but some of them leave the choice unspecified---to be chosen
heuristically for each file, based on the data.
Richard M. Stallman's avatar
Richard M. Stallman committed
605

606 607 608 609
@cindex end of line conversion
  @dfn{End of line conversion} handles three different conventions used
on various systems for representing end of line in files.  The Unix
convention is to use the linefeed character (also called newline).  The
610 611
DOS convention is to use a carriage-return and a linefeed at the end of
a line.  The Mac convention is to use just carriage-return.
612

Richard M. Stallman's avatar
Richard M. Stallman committed
613 614 615 616 617 618
@cindex base coding system
@cindex variant coding system
  @dfn{Base coding systems} such as @code{latin-1} leave the end-of-line
conversion unspecified, to be chosen based on the data.  @dfn{Variant
coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and
@code{latin-1-mac} specify the end-of-line conversion explicitly as
619
well.  Most base coding systems have three corresponding variants whose
Richard M. Stallman's avatar
Richard M. Stallman committed
620 621
names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}.

622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657
  The coding system @code{raw-text} is special in that it prevents
character code conversion, and causes the buffer visited with that
coding system to be a unibyte buffer.  It does not specify the
end-of-line conversion, allowing that to be determined as usual by the
data, and has the usual three variants which specify the end-of-line
conversion.  @code{no-conversion} is equivalent to @code{raw-text-unix}:
it specifies no conversion of either character codes or end-of-line.

  The coding system @code{emacs-mule} specifies that the data is
represented in the internal Emacs encoding.  This is like
@code{raw-text} in that no code conversion happens, but different in
that the result is multibyte data.

@defun coding-system-get coding-system property
This function returns the specified property of the coding system
@var{coding-system}.  Most coding system properties exist for internal
purposes, but one that you might find useful is @code{mime-charset}.
That property's value is the name used in MIME for the character coding
which this coding system can read and write.  Examples:

@example
(coding-system-get 'iso-latin-1 'mime-charset)
     @result{} iso-8859-1
(coding-system-get 'iso-2022-cn 'mime-charset)
     @result{} iso-2022-cn
(coding-system-get 'cyrillic-koi8 'mime-charset)
     @result{} koi8-r
@end example

The value of the @code{mime-charset} property is also defined
as an alias for the coding system.
@end defun

@node Encoding and I/O
@subsection Encoding and I/O

658
  The principal purpose of coding systems is for use in reading and
659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674
writing files.  The function @code{insert-file-contents} uses
a coding system for decoding the file data, and @code{write-region}
uses one to encode the buffer contents.

  You can specify the coding system to use either explicitly
(@pxref{Specifying Coding Systems}), or implicitly using the defaulting
mechanism (@pxref{Default Coding Systems}).  But these methods may not
completely specify what to do.  For example, they may choose a coding
system such as @code{undefined} which leaves the character code
conversion to be determined from the data.  In these cases, the I/O
operation finishes the job of choosing a coding system.  Very often
you will want to find out afterwards which coding system was chosen.

@defvar buffer-file-coding-system
This variable records the coding system that was used for visiting the
current buffer.  It is used for saving the buffer, and for writing part
675 676 677 678 679 680 681 682 683
of the buffer with @code{write-region}.  If the text to be written
cannot be safely encoded using the coding system specified by this
variable, these operations select an alternative encoding by calling
the function @code{select-safe-coding-system} (@pxref{User-Chosen
Coding Systems}).  If selecting a different encoding requires to ask
the user to specify a coding system, @code{buffer-file-coding-system}
is updated to the newly selected coding system.

@code{buffer-file-coding-system} does @emph{not} affect sending text
684
to a subprocess.
685 686 687
@end defvar

@defvar save-buffer-coding-system
Kenichi Handa's avatar
Kenichi Handa committed
688 689 690
This variable specifies the coding system for saving the buffer (by
overriding @code{buffer-file-coding-system}).  Note that it is not used
for @code{write-region}.
691 692

When a command to save the buffer starts out to use
Kenichi Handa's avatar
Kenichi Handa committed
693 694
@code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),
and that coding system cannot handle
695
the actual text in the buffer, the command asks the user to choose
696 697 698 699
another coding system (by calling @code{select-safe-coding-system}).
After that happens, the command also updates
@code{buffer-file-coding-system} to represent the coding system that
the user specified.
700 701 702 703 704 705 706 707
@end defvar

@defvar last-coding-system-used
I/O operations for files and subprocesses set this variable to the
coding system name that was used.  The explicit encoding and decoding
functions (@pxref{Explicit Encoding}) set it too.

@strong{Warning:} Since receiving subprocess output sets this variable,
708 709
it can change whenever Emacs waits; therefore, you should copy the
value shortly after the function call that stores the value you are
710 711 712
interested in.
@end defvar

713 714 715
  The variable @code{selection-coding-system} specifies how to encode
selections for the window system.  @xref{Window System Selections}.

716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735
@defvar file-name-coding-system
The variable @code{file-name-coding-system} specifies the coding
system to use for encoding file names.  Emacs encodes file names using
that coding system for all file operations.  If
@code{file-name-coding-system} is @code{nil}, Emacs uses a default
coding system determined by the selected language environment.  In the
default language environment, any non-@acronym{ASCII} characters in
file names are not encoded specially; they appear in the file system
using the internal Emacs representation.
@end defvar

  @strong{Warning:} if you change @code{file-name-coding-system} (or
the language environment) in the middle of an Emacs session, problems
can result if you have already visited files whose names were encoded
using the earlier coding system and are handled differently under the
new coding system.  If you try to save one of these buffers under the
visited file name, saving may use the wrong file name, or it may get
an error.  If such a problem happens, use @kbd{C-x C-w} to specify a
new file name for that buffer.

736 737 738
@node Lisp and Coding Systems
@subsection Coding Systems in Lisp

739
  Here are the Lisp facilities for working with coding systems:
Richard M. Stallman's avatar
Richard M. Stallman committed
740 741 742 743

@defun coding-system-list &optional base-only
This function returns a list of all coding system names (symbols).  If
@var{base-only} is non-@code{nil}, the value includes only the
Kenichi Handa's avatar
Kenichi Handa committed
744 745
base coding systems.  Otherwise, it includes alias and variant coding
systems as well.
Richard M. Stallman's avatar
Richard M. Stallman committed
746 747 748 749
@end defun

@defun coding-system-p object
This function returns @code{t} if @var{object} is a coding system
750
name or @code{nil}.
Richard M. Stallman's avatar
Richard M. Stallman committed
751 752 753 754 755 756 757 758
@end defun

@defun check-coding-system coding-system
This function checks the validity of @var{coding-system}.
If that is valid, it returns @var{coding-system}.
Otherwise it signals an error with condition @code{coding-system-error}.
@end defun

759 760
@defun coding-system-change-eol-conversion coding-system eol-type
This function returns a coding system which is like @var{coding-system}
761
except for its eol conversion, which is specified by @code{eol-type}.
762 763 764
@var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or
@code{nil}.  If it is @code{nil}, the returned coding system determines
the end-of-line conversion from the data.
765 766

@var{eol-type} may also be 0, 1 or 2, standing for @code{unix},
Luc Teirlinck's avatar
Luc Teirlinck committed
767
@code{dos} and @code{mac}, respectively.
768
@end defun
769

770 771 772 773 774
@defun coding-system-change-text-conversion eol-coding text-coding
This function returns a coding system which uses the end-of-line
conversion of @var{eol-coding}, and the text conversion of
@var{text-coding}.  If @var{text-coding} is @code{nil}, it returns
@code{undecided}, or one of its variants according to @var{eol-coding}.
775 776
@end defun

777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800
@defun find-coding-systems-region from to
This function returns a list of coding systems that could be used to
encode a text between @var{from} and @var{to}.  All coding systems in
the list can safely encode any multibyte characters in that portion of
the text.

If the text contains no multibyte characters, the function returns the
list @code{(undecided)}.
@end defun

@defun find-coding-systems-string string
This function returns a list of coding systems that could be used to
encode the text of @var{string}.  All coding systems in the list can
safely encode any multibyte characters in @var{string}.  If the text
contains no multibyte characters, this returns the list
@code{(undecided)}.
@end defun

@defun find-coding-systems-for-charsets charsets
This function returns a list of coding systems that could be used to
encode all the character sets in the list @var{charsets}.
@end defun

@defun detect-coding-region start end &optional highest
Richard M. Stallman's avatar
Richard M. Stallman committed
801
This function chooses a plausible coding system for decoding the text
Gerd Moellmann's avatar
Gerd Moellmann committed
802
from @var{start} to @var{end}.  This text should be a byte sequence
803
(@pxref{Explicit Encoding}).
Richard M. Stallman's avatar
Richard M. Stallman committed
804

805
Normally this function returns a list of coding systems that could
Richard M. Stallman's avatar
Richard M. Stallman committed
806
handle decoding the text that was scanned.  They are listed in order of
807 808 809 810
decreasing priority.  But if @var{highest} is non-@code{nil}, then the
return value is just one coding system, the one that is highest in
priority.

811
If the region contains only @acronym{ASCII} characters, the value
812 813
is @code{undecided} or @code{(undecided)}, or a variant specifying
end-of-line conversion, if that can be deduced from the text.
Richard M. Stallman's avatar
Richard M. Stallman committed
814 815
@end defun

816
@defun detect-coding-string string &optional highest
Richard M. Stallman's avatar
Richard M. Stallman committed
817 818
This function is like @code{detect-coding-region} except that it
operates on the contents of @var{string} instead of bytes in the buffer.
819 820
@end defun

821 822 823 824
  @xref{Coding systems for a subprocess,, Process Information}, in
particular the description of the functions
@code{process-coding-system} and @code{set-process-coding-system}, for
how to examine or set the coding systems used for I/O to a subprocess.
825 826 827 828

@node User-Chosen Coding Systems
@subsection User-Chosen Coding Systems

829
@cindex select safe coding system
830
@defun select-safe-coding-system from to &optional default-coding-system accept-default-p file
831 832
This function selects a coding system for encoding specified text,
asking the user to choose if necessary.  Normally the specified text
833 834 835
is the text in the current buffer between @var{from} and @var{to}.  If
@var{from} is a string, the string specifies the text to encode, and
@var{to} is ignored.
836 837 838 839 840

If @var{default-coding-system} is non-@code{nil}, that is the first
coding system to try; if that can handle the text,
@code{select-safe-coding-system} returns that coding system.  It can
also be a list of coding systems; then the function tries each of them
841 842 843 844 845 846 847
one by one.  After trying all of them, it next tries the current
buffer's value of @code{buffer-file-coding-system} (if it is not
@code{undecided}), then the value of
@code{default-buffer-file-coding-system} and finally the user's most
preferred coding system, which the user can set using the command
@code{prefer-coding-system} (@pxref{Recognize Coding,, Recognizing
Coding Systems, emacs, The GNU Emacs Manual}).
848 849 850 851 852 853

If one of those coding systems can safely encode all the specified
text, @code{select-safe-coding-system} chooses it and returns it.
Otherwise, it asks the user to choose from a list of coding systems
which can encode all the text, and returns the user's choice.

854 855 856 857 858 859
@var{default-coding-system} can also be a list whose first element is
t and whose other elements are coding systems.  Then, if no coding
system in the list can handle the text, @code{select-safe-coding-system}
queries the user immediately, without trying any of the three
alternatives described above.

860
The optional argument @var{accept-default-p}, if non-@code{nil},
861 862 863 864 865 866 867
should be a function to determine whether a coding system selected
without user interaction is acceptable. @code{select-safe-coding-system}
calls this function with one argument, the base coding system of the
selected coding system.  If @var{accept-default-p} returns @code{nil},
@code{select-safe-coding-system} rejects the silently selected coding
system, and asks the user to select a coding system from a list of
possible candidates.
868 869 870 871 872

@vindex select-safe-coding-system-accept-default-p
If the variable @code{select-safe-coding-system-accept-default-p} is
non-@code{nil}, its value overrides the value of
@var{accept-default-p}.
873 874 875 876 877 878 879 880 881 882 883 884

As a final step, before returning the chosen coding system,
@code{select-safe-coding-system} checks whether that coding system is
consistent with what would be selected if the contents of the region
were read from a file.  (If not, this could lead to data corruption in
a file subsequently re-visited and edited.)  Normally,
@code{select-safe-coding-system} uses @code{buffer-file-name} as the
file for this purpose, but if @var{file} is non-@code{nil}, it uses
that file instead (this can be relevant for @code{write-region} and
similar functions).  If it detects an apparent inconsistency,
@code{select-safe-coding-system} queries the user before selecting the
coding system.
885 886 887 888 889
@end defun

  Here are two functions you can use to let the user specify a coding
system, with completion.  @xref{Completion}.

890
@defun read-coding-system prompt &optional default
891 892 893 894 895 896 897 898
This function reads a coding system using the minibuffer, prompting with
string @var{prompt}, and returns the coding system name as a symbol.  If
the user enters null input, @var{default} specifies which coding system
to return.  It should be a symbol or a string.
@end defun

@defun read-non-nil-coding-system prompt
This function reads a coding system using the minibuffer, prompting with
899
string @var{prompt}, and returns the coding system name as a symbol.  If
900 901
the user tries to enter null input, it asks the user to try again.
@xref{Coding Systems}.
Richard M. Stallman's avatar
Richard M. Stallman committed
902 903 904
@end defun

@node Default Coding Systems
905
@subsection Default Coding Systems
Richard M. Stallman's avatar
Richard M. Stallman committed
906

907 908
  This section describes variables that specify the default coding
system for certain files or when running certain subprograms, and the
909
function that I/O operations use to access them.
910 911 912 913 914 915 916

  The idea of these variables is that you set them once and for all to the
defaults you want, and then do not change them again.  To specify a
particular coding system for a particular operation in a Lisp program,
don't change these variables; instead, override them using
@code{coding-system-for-read} and @code{coding-system-for-write}
(@pxref{Specifying Coding Systems}).
Richard M. Stallman's avatar
Richard M. Stallman committed
917

918 919 920 921 922 923 924 925 926 927 928 929
@defvar auto-coding-regexp-alist
This variable is an alist of text patterns and corresponding coding
systems. Each element has the form @code{(@var{regexp}
. @var{coding-system})}; a file whose first few kilobytes match
@var{regexp} is decoded with @var{coding-system} when its contents are
read into a buffer.  The settings in this alist take priority over
@code{coding:} tags in the files and the contents of
@code{file-coding-system-alist} (see below).  The default value is set
so that Emacs automatically recognizes mail files in Babyl format and
reads them with no code conversions.
@end defvar

Richard M. Stallman's avatar
Richard M. Stallman committed
930 931 932 933 934 935 936
@defvar file-coding-system-alist
This variable is an alist that specifies the coding systems to use for
reading and writing particular files.  Each element has the form
@code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regular
expression that matches certain file names.  The element applies to file
names that match @var{pattern}.

937
The @sc{cdr} of the element, @var{coding}, should be either a coding
938 939 940
system, a cons cell containing two coding systems, or a function name (a
symbol with a function definition).  If @var{coding} is a coding system,
that coding system is used for both reading the file and writing it.  If
941 942
@var{coding} is a cons cell containing two coding systems, its @sc{car}
specifies the coding system for decoding, and its @sc{cdr} specifies the
943 944
coding system for encoding.

945 946 947 948 949
If @var{coding} is a function name, the function should take one
argument, a list of all arguments passed to
@code{find-operation-coding-system}.  It must return a coding system
or a cons cell containing two coding systems.  This value has the same
meaning as described above.
Richard M. Stallman's avatar
Richard M. Stallman committed
950 951 952 953 954 955 956 957 958 959 960 961
@end defvar

@defvar process-coding-system-alist
This variable is an alist specifying which coding systems to use for a
subprocess, depending on which program is running in the subprocess.  It
works like @code{file-coding-system-alist}, except that @var{pattern} is
matched against the program name used to start the subprocess.  The coding
system or systems specified in this alist are used to initialize the
coding systems used for I/O to the subprocess, but you can specify
other coding systems later using @code{set-process-coding-system}.
@end defvar

962 963
  @strong{Warning:} Coding systems such as @code{undecided}, which
determine the coding system from the data, do not work entirely reliably
964
with asynchronous subprocess output.  This is because Emacs handles
965 966 967 968 969 970 971 972 973 974
asynchronous subprocess output in batches, as it arrives.  If the coding
system leaves the character code conversion unspecified, or leaves the
end-of-line conversion unspecified, Emacs must try to detect the proper
conversion from one batch at a time, and this does not always work.

  Therefore, with an asynchronous subprocess, if at all possible, use a
coding system which determines both the character code conversion and
the end of line conversion---that is, one like @code{latin-1-unix},
rather than @code{undecided} or @code{latin-1}.

Richard M. Stallman's avatar
Richard M. Stallman committed
975 976 977
@defvar network-coding-system-alist
This variable is an alist that specifies the coding system to use for
network streams.  It works much like @code{file-coding-system-alist},
978
with the difference that the @var{pattern} in an element may be either a
Richard M. Stallman's avatar
Richard M. Stallman committed
979 980 981 982 983 984 985 986 987 988
port number or a regular expression.  If it is a regular expression, it
is matched against the network service name used to open the network
stream.
@end defvar

@defvar default-process-coding-system
This variable specifies the coding systems to use for subprocess (and
network stream) input and output, when nothing else specifies what to
do.

989 990 991
The value should be a cons cell of the form @code{(@var{input-coding}
. @var{output-coding})}.  Here @var{input-coding} applies to input from
the subprocess, and @var{output-coding} applies to output to it.
Richard M. Stallman's avatar
Richard M. Stallman committed
992 993
@end defvar

994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009
@defvar auto-coding-functions
This variable holds a list of functions that try to determine a
coding system for a file based on its undecoded contents.

Each function in this list should be written to look at text in the
current buffer, but should not modify it in any way.  The buffer will
contain undecoded text of parts of the file.  Each function should
take one argument, @var{size}, which tells it how many characters to
look at, starting from point.  If the function succeeds in determining
a coding system for the file, it should return that coding system.
Otherwise, it should return @code{nil}.

If a file has a @samp{coding:} tag, that takes precedence, so these
functions won't be called.
@end defvar

1010 1011 1012 1013 1014 1015
@defun find-operation-coding-system operation &rest arguments
This function returns the coding system to use (by default) for
performing @var{operation} with @var{arguments}.  The value has this
form:

@example
1016
(@var{decoding-system} . @var{encoding-system})
1017 1018 1019 1020 1021 1022 1023
@end example

The first element, @var{decoding-system}, is the coding system to use
for decoding (in case @var{operation} does decoding), and
@var{encoding-system} is the coding system for encoding (in case
@var{operation} does encoding).

1024
The argument @var{operation} should be a symbol, one of
1025 1026
@code{insert-file-contents}, @code{write-region}, @code{call-process},
@code{call-process-region}, @code{start-process}, or
1027 1028
@code{open-network-stream}.  These are the names of the Emacs I/O primitives
that can do coding system conversion.
1029 1030

The remaining arguments should be the same arguments that might be given
1031
to that I/O primitive.  Depending on the primitive, one of those
1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042
arguments is selected as the @dfn{target}.  For example, if
@var{operation} does file I/O, whichever argument specifies the file
name is the target.  For subprocess primitives, the process name is the
target.  For @code{open-network-stream}, the target is the service name
or port number.

This function looks up the target in @code{file-coding-system-alist},
@code{process-coding-system-alist}, or
@code{network-coding-system-alist}, depending on @var{operation}.
@end defun

Richard M. Stallman's avatar
Richard M. Stallman committed
1043
@node Specifying Coding Systems
1044
@subsection Specifying a Coding System for One Operation
Richard M. Stallman's avatar
Richard M. Stallman committed
1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066

  You can specify the coding system for a specific operation by binding
the variables @code{coding-system-for-read} and/or
@code{coding-system-for-write}.

@defvar coding-system-for-read
If this variable is non-@code{nil}, it specifies the coding system to
use for reading a file, or for input from a synchronous subprocess.

It also applies to any asynchronous subprocess or network stream, but in
a different way: the value of @code{coding-system-for-read} when you
start the subprocess or open the network stream specifies the input
decoding method for that subprocess or network stream.  It remains in
use for that subprocess or network stream unless and until overridden.

The right way to use this variable is to bind it with @code{let} for a
specific I/O operation.  Its global value is normally @code{nil}, and
you should not globally set it to any other value.  Here is an example
of the right way to use the variable:

@example
;; @r{Read the file with no character code conversion.}
1067
;; @r{Assume @acronym{crlf} represents end-of-line.}
Richard M. Stallman's avatar
Richard M. Stallman committed
1068 1069 1070 1071 1072
(let ((coding-system-for-write 'emacs-mule-dos))
  (insert-file-contents filename))
@end example

When its value is non-@code{nil}, @code{coding-system-for-read} takes
1073
precedence over all other methods of specifying a coding system to use for
Richard M. Stallman's avatar
Richard M. Stallman committed
1074 1075 1076 1077 1078 1079 1080 1081
input, including @code{file-coding-system-alist},
@code{process-coding-system-alist} and
@code{network-coding-system-alist}.
@end defvar

@defvar coding-system-for-write
This works much like @code{coding-system-for-read}, except that it
applies to output rather than input.  It affects writing to files,
1082
as well as sending output to subprocesses and net connections.
Richard M. Stallman's avatar
Richard M. Stallman committed
1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097

When a single operation does both input and output, as do
@code{call-process-region} and @code{start-process}, both
@code{coding-system-for-read} and @code{coding-system-for-write}
affect it.
@end defvar

@defvar inhibit-eol-conversion
When this variable is non-@code{nil}, no end-of-line conversion is done,
no matter which coding system is specified.  This applies to all the
Emacs I/O and subprocess primitives, and to the explicit encoding and
decoding functions (@pxref{Explicit Encoding}).
@end defvar

@node Explicit Encoding
1098
@subsection Explicit Encoding and Decoding
Richard M. Stallman's avatar
Richard M. Stallman committed
1099 1100 1101 1102 1103 1104 1105 1106 1107
@cindex encoding text
@cindex decoding text

  All the operations that transfer text in and out of Emacs have the
ability to use a coding system to encode or decode the text.
You can also explicitly encode and decode text using the functions
in this section.

  The result of encoding, and the input to decoding, are not ordinary
Gerd Moellmann's avatar
Gerd Moellmann committed
1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125
text.  They logically consist of a series of byte values; that is, a
series of characters whose codes are in the range 0 through 255.  In a
multibyte buffer or string, character codes 128 through 159 are
represented by multibyte sequences, but this is invisible to Lisp
programs.

  The usual way to read a file into a buffer as a sequence of bytes, so
you can decode the contents explicitly, is with
@code{insert-file-contents-literally} (@pxref{Reading from Files});
alternatively, specify a non-@code{nil} @var{rawfile} argument when
visiting a file with @code{find-file-noselect}.  These methods result in
a unibyte buffer.

  The usual way to use the byte sequence that results from explicitly
encoding text is to copy it to a file or process---for example, to write
it with @code{write-region} (@pxref{Writing to Files}), and suppress
encoding by binding @code{coding-system-for-write} to
@code{no-conversion}.
1126 1127

  Here are the functions to perform explicit encoding or decoding.  The
Gerd Moellmann's avatar
Gerd Moellmann committed
1128 1129 1130
decoding functions produce sequences of bytes; the encoding functions
are meant to operate on sequences of bytes.  All of these functions
discard text properties.
1131

1132 1133
@deffn Command encode-coding-region start end coding-system
This command encodes the text from @var{start} to @var{end} according
1134
to coding system @var{coding-system}.  The encoded text replaces the
Gerd Moellmann's avatar
Gerd Moellmann committed
1135 1136 1137
original text in the buffer.  The result of encoding is logically a
sequence of bytes, but the buffer remains multibyte if it was multibyte
before.
Richard M. Stallman's avatar
Richard M. Stallman committed
1138

1139 1140 1141 1142
This command returns the length of the encoded text.
@end deffn

@defun encode-coding-string string coding-system &optional nocopy
Richard M. Stallman's avatar
Richard M. Stallman committed
1143 1144
This function encodes the text in @var{string} according to coding
system @var{coding-system}.  It returns a new string containing the
1145 1146 1147
encoded text, except when @var{nocopy} is non-@code{nil}, in which
case the function may return @var{string} itself if the encoding
operation is trivial.  The result of encoding is a unibyte string.
Richard M. Stallman's avatar
Richard M. Stallman committed
1148 1149
@end defun

1150 1151
@deffn Command decode-coding-region start end coding-system
This command decodes the text from @var{start} to @var{end} according
Richard M. Stallman's avatar
Richard M. Stallman committed
1152 1153
to coding system @var{coding-system}.  The decoded text replaces the
original text in the buffer.  To make explicit decoding useful, the text
Gerd Moellmann's avatar
Gerd Moellmann committed
1154 1155
before decoding ought to be a sequence of byte values, but both
multibyte and unibyte buffers are acceptable.
Richard M. Stallman's avatar
Richard M. Stallman committed
1156

1157 1158 1159 1160
This command returns the length of the decoded text.
@end deffn

@defun decode-coding-string string coding-system &optional nocopy
Richard M. Stallman's avatar
Richard M. Stallman committed
1161 1162
This function decodes the text in @var{string} according to coding
system @var{coding-system}.  It returns a new string containing the
1163 1164 1165 1166
decoded text, except when @var{nocopy} is non-@code{nil}, in which
case the function may return @var{string} itself if the decoding
operation is trivial.  To make explicit decoding useful, the contents
of @var{string} ought to be a sequence of byte values, but a multibyte
Gerd Moellmann's avatar
Gerd Moellmann committed
1167
string is acceptable.
Richard M. Stallman's avatar
Richard M. Stallman committed
1168
@end defun
1169

1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180
@defun decode-coding-inserted-region from to filename &optional visit beg end replace
This function decodes the text from @var{from} to @var{to} as if
it were being read from file @var{filename} using @code{insert-file-contents}
using the rest of the arguments provided.

The normal way to use this function is after reading text from a file
without decoding, if you decide you would rather have decoded it.
Instead of deleting the text and reading it again, this time with
decoding, you can call this function.
@end defun

1181 1182 1183 1184
@node Terminal I/O Encoding
@subsection Terminal I/O Encoding

  Emacs can decode keyboard input using a coding system, and encode
1185 1186 1187 1188
terminal output.  This is useful for terminals that transmit or display
text using a particular encoding such as Latin-1.  Emacs does not set
@code{last-coding-system-used} for encoding or decoding for the
terminal.
1189 1190 1191 1192 1193 1194

@defun keyboard-coding-system
This function returns the coding system that is in use for decoding
keyboard input---or @code{nil} if no coding system is to be used.
@end defun

1195 1196
@deffn Command set-keyboard-coding-system coding-system
This command specifies @var{coding-system} as the coding system to
1197 1198
use for decoding keyboard input.  If @var{coding-system} is @code{nil},
that means do not decode keyboard input.
1199
@end deffn
1200 1201 1202 1203 1204 1205

@defun terminal-coding-system
This function returns the coding system that is in use for encoding
terminal output---or @code{nil} for no encoding.
@end defun

1206 1207
@deffn Command set-terminal-coding-system coding-system
This command specifies @var{coding-system} as the coding system to use
1208 1209
for encoding terminal output.  If @var{coding-system} is @code{nil},
that means do not encode terminal output.
1210
@end deffn
1211

1212
@node MS-DOS File Types
1213
@subsection MS-DOS File Types
1214 1215 1216 1217 1218 1219 1220
@cindex DOS file types
@cindex MS-DOS file types
@cindex Windows file types
@cindex file types on MS-DOS and Windows
@cindex text files and binary files
@cindex binary files and text files

1221 1222
  On MS-DOS and Microsoft Windows, Emacs guesses the appropriate
end-of-line conversion for a file by looking at the file's name.  This
Gerd Moellmann's avatar
Gerd Moellmann committed
1223
feature classifies files as @dfn{text files} and @dfn{binary files}.  By
1224 1225 1226 1227 1228 1229
``binary file'' we mean a file of literal byte values that are not
necessarily meant to be characters; Emacs does no end-of-line conversion
and no character code conversion for them.  On the other hand, the bytes
in a text file are intended to represent characters; when you create a
new file whose name implies that it is a text file, Emacs uses DOS
end-of-line conversion.
1230 1231 1232

@defvar buffer-file-type
This variable, automatically buffer-local in each buffer, records the
1233 1234 1235 1236 1237 1238 1239 1240 1241
file type of the buffer's visited file.  When a buffer does not specify
a coding system with @code{buffer-file-coding-system}, this variable is
used to determine which coding system to use when writing the contents
of the buffer.  It should be @code{nil} for text, @code{t} for binary.
If it is @code{t}, the coding system is @code{no-conversion}.
Otherwise, @code{undecided-dos} is used.

Normally this variable is set by visiting a file; it is set to
@code{nil} if the file was visited without any actual conversion.
1242 1243 1244 1245 1246 1247 1248 1249 1250 1251
@end defvar

@defopt file-name-buffer-file-type-alist
This variable holds an alist for recognizing text and binary files.
Each element has the form (@var{regexp} . @var{type}), where
@var{regexp} is matched against the file name, and @var{type} may be
@code{nil} for text, @code{t} for binary, or a function to call to
compute which.  If it is a function, then it is called with a single