unicode 9.27 KB
Newer Older
1
                                            -*-mode: text; coding: utf-8;-*-
Glenn Morris's avatar
Glenn Morris committed
2

3
Copyright (C) 2002-2014 Free Software Foundation, Inc.
Glenn Morris's avatar
Glenn Morris committed
4 5
See the end of the file for license conditions.

6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
Importing a new Unicode Standard version into Emacs
-------------------------------------------------------------

Emacs uses the following files from the Unicode Character Database
(a.k.a. "UCD):

  . UnicodeData.txt
  . BidiMirroring.txt
  . IVD_Sequences.txt

First, these files need to be copied into admin/unidata/, and then
Emacs should be rebuilt for them to take effect.  Rebuilding Emacs
updates several derived files elsewhere in the Emacs source tree,
mainly in lisp/international/.

When Emacs is rebuilt for the first time after importing the new
files, pay attention to any warning or error messages.  In particular,
admin/unidata/unidata-gen.el will complain if UnicodeData.txt defines
new bidirectional attributes of characters, because unidata-gen.el,
bidi.c and dispextern.h need to be updated in that case; failure to do
so will cause aborts in redisplay.

Next, review the changes in UnicodeData.txt vs the previous version
used by Emacs.  Any changes, be it introduction of new scripts or
addition of codepoints to existing scripts, need corresponding changes
in the data used for filling char-script-table, see characters.el
around line 1300.  Other databases and settings in characters.el, such
as the data for char-width-table, might also need changes.

Any new scripts added by UnicodeData.txt will also need updates to
script-representative-chars defined in fontset.el.  Other databases in
fontset.el might also need to be updated as needed.

Glenn Morris's avatar
Glenn Morris committed
39 40 41 42 43 44 45 46 47
Problems, fixmes and other unicode-related issues
-------------------------------------------------------------

Notes by fx to record various things of variable importance.  handa
needs to check them -- don't take too seriously, especially with
regard to completeness.

 * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
   undesirable effects.  E.g.:
48 49 50
   (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
   (multibyte-string-p (concat [?£])) => nil
   (text-char-description ?£) => "M-#"
Glenn Morris's avatar
Glenn Morris committed
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112

	These examples are all fixed by the change of 2002-10-14, but
	there still exist questionable SINGLE_BYTE_CHAR_P in the
	code (keymap.c and print.c).

 * Rationalize character syntax and its relationship to the Unicode
   database.  (Applies mainly to symbol an punctuation syntax.)

 * Fontset handling and customization needs work.  We want to relate
   fonts to scripts, probably based on the Unicode blocks.  The
   presence of small-repertoire 10646-encoded fonts in XFree 4 is a
   pain, not currently worked round.

	With the change on 2002-07-26, multiple fonts can be
	specified in a fontset for a specific range of characters.
	Each range can also be specified by script.  Before using
	ISO10646 fonts, Emacs checks their repertories to avoid such
	fonts that don't have a glyph for a specific character.

	fx has worked on fontset customization, but was stymied by
	basic problems with the way the default face is dealt with
	(and something else, I think).  This needs revisiting.

 * Work is also needed on charset and coding system priorities.

 * The relevant bits of latin1-disp.el need porting (and probably
   re-naming/updating).  See also cyril-util.el.

 * Quail files need more work now the encoding is largely irrelevant.

 * What to do with the old coding categories stuff?

 * The preferred-coding-system property of charsets should probably be
   junked unless it can be made more useful now.

 * find-multibyte-characters needs looking at.

 * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
   charsets.

 * Lazy-load tables for unify-charset somehow?

	Actually, Emacs clears out all charset maps and unify-map just
	before dumping, and they are loaded again on demand by the
	dumped emacs.  But, those maps (char tables) generated while
	temacs is running can't be removed from the dumped emacs.

 * iso-2022 charsets get unified on i/o.

	With the change on 2003-01-06, decoding routines put `charset'
	property to decoded text, and iso-2022 encoder pay attention
	to it.  Thus, for instance, reading and writing by
	iso-2022-7bit preserve the original designation sequences.
	The property name `preferred-charset' may be better?

	We may have to utilize this property to decide a font.

 * Revisit locale processing: look at treating the language and
   charset parts separately.  (Language should affect things like
   spelling and calendar, but that's not a Unicode issue.)

 * Handle Unicode combining characters usefully, e.g. diacritics, and
113
   handle more scripts specifically (à la Devanagari).  There are
Glenn Morris's avatar
Glenn Morris committed
114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133
   issues with canonicalization.

 * We need tabular input methods, e.g. for maths symbols.  (Not
   specific to Unicode.)

 * Need multibyte text in menus, e.g. for the above.  (Not specific to
   Unicode -- see Emacs etc/TODO, but now mostly works with gtk.)

 * There's currently no support for Unicode normalization.

 * Populate char-width-table correctly for Unicode characters and
   worry about what happens when double-width charsets covering
   non-CJK characters are unified.

 * There are type errors lurking, e.g. in
   Fcheck_coding_systems_region.  Define ENABLE_CHECKING to find them.

 * Old auto-save files, and similar files, such as Gnus drafts,
   containing non-ASCII characters probably won't be re-read correctly.

134 135 136 137 138 139

Source file encoding
--------------------

Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a
subset), but there are a few exceptions, listed below.  Perhaps
140
someday many of these files will be converted to UTF-8, for
141 142
convenience when using tools like 'grep -r', but this might need
nontrivial changes to the build process.
143 144 145

 * chinese-big5

146 147 148
     These are verbatim copies of files taken from external sources.
     They haven't been converted to UTF-8.

149 150 151 152 153 154 155 156 157 158 159 160 161
	leim/CXTERM-DIC/4Corner.tit
	leim/CXTERM-DIC/ARRAY30.tit
	leim/CXTERM-DIC/ECDICT.tit
	leim/CXTERM-DIC/ETZY.tit
	leim/CXTERM-DIC/PY-b5.tit
	leim/CXTERM-DIC/Punct-b5.tit
	leim/CXTERM-DIC/QJ-b5.tit
	leim/CXTERM-DIC/ZOZY.tit
	leim/MISC-DIC/CTLau-b5.html
	leim/MISC-DIC/cangjie-table.b5

 * chinese-iso-8bit

162 163 164
     These are verbatim copies of files taken from external sources.
     They haven't been converted to UTF-8.

165 166 167 168 169 170 171 172 173
	leim/CXTERM-DIC/CCDOSPY.tit
	leim/CXTERM-DIC/Punct.tit
	leim/CXTERM-DIC/QJ.tit
	leim/CXTERM-DIC/SW.tit
	leim/CXTERM-DIC/TONEPY.tit
	leim/MISC-DIC/pinyin.map
	leim/MISC-DIC/CTLau.html
	leim/MISC-DIC/ziranma.cin

174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189
 * cp850

     This file contains non-ASCII characters in unibyte strings.  When
     editing a keyboard layout it's more convenient to see 'é' than
     '\202', and the MS-DOS compiler requires the single byte if a
     backslash escape is not being used.

	src/msdos.c

 * iso-2022-cn-ext

     This file is externally generated from leim/MISC-DIC/cangjie-table.b5
     by Big5->CNS converter.  It hasn't been converted to UTF-8.

	leim/MISC-DIC/cangjie-table.cns

190 191
 * iso-latin-2

192 193 194 195 196
     These files are processed by csplain, a program that requires
     Latin-2 input.  In 2012 the csplain maintainers started
     recommending UTF-8, but these files haven't been converted yet.

	etc/refcards/cs-dired-ref.tex
197 198 199 200
	etc/refcards/cs-refcard.tex
	etc/refcards/cs-survival.tex
	etc/refcards/sk-dired-ref.tex
	etc/refcards/sk-refcard.tex
201
	etc/refcards/sk-survival.tex
202 203 204

 * japanese-iso-8bit

205
     SKK-JISYO.L is a verbatim copy of a file taken from an external source.
206
     It hasn't been converted to UTF-8.
207

208 209 210 211
	leim/SKK-DIC/SKK-JISYO.L

 * japanese-shift-jis

212 213 214
     This is a verbatim copy of a file taken from an external source.
     It hasn't been converted to UTF-8.

215 216
	admin/charsets/mapfiles/cns2ucsdkw.txt

217 218
 * iso-2022-7bit

Paul Eggert's avatar
Paul Eggert committed
219
     This file switches between CJK charsets, which is not encoded in UTF-8.
Paul Eggert's avatar
Paul Eggert committed
220 221 222

	etc/HELLO

Paul Eggert's avatar
Paul Eggert committed
223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243
     Each of these files contains just one CJK charset, but Emacs
     currently has no easy way to specify set-charset-priority on a
     per-file basis, so converting any of these files to UTF-8 might
     change the file's appearance when viewed by an Emacs that is
     operating in some other language environment.

	etc/tutorials/TUTORIAL.ja
	leim/quail/cyril-jis.el
	leim/quail/hanja-jis.el
	leim/quail/japanese.el
	leim/quail/py-punct.el
	leim/quail/pypunct-b5.el
	lisp/international/ja-dic-cnv.el
	lisp/international/ja-dic-utl.el
	lisp/international/kinsoku.el
	lisp/international/kkc.el
	lisp/international/titdic-cnv.el
	lisp/language/japan-util.el
	lisp/language/japanese.el
	lisp/term/x-win.el

244 245
 * utf-8-emacs

246 247 248 249 250 251 252 253 254
     These files contain characters that cannot be encoded in UTF-8.

	leim/quail/tibetan.el
	leim/quail/ethiopic.el
	lisp/international/titdic-cnv.el
	lisp/language/tibetan.el
	lisp/language/tibet-util.el
	lisp/language/ind-util.el

Glenn Morris's avatar
Glenn Morris committed
255 256 257

This file is part of GNU Emacs.

258
GNU Emacs is free software: you can redistribute it and/or modify
Glenn Morris's avatar
Glenn Morris committed
259
it under the terms of the GNU General Public License as published by
260 261
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
Glenn Morris's avatar
Glenn Morris committed
262 263 264 265 266 267 268

GNU Emacs is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
269
along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.