Commit ed7ebd93 authored by Eli Zaretskii's avatar Eli Zaretskii

Update the bidirectional reordering engine for Unicode 6.3 and 7.0.

 src/bidi.c (bidi_ignore_explicit_marks_for_paragraph_level): Remove
 variable.
 (bidi_get_type): Return the isolate initiators and terminator
 types.
 (bidi_isolate_fmt_char, bidi_paired_bracket_type)
 (bidi_fetch_char_skip_isolates, find_first_strong_char)
 (bidi_find_bracket_pairs, bidi_resolve_brackets): New functions.
 (bidi_set_sos_type): Renamed from bidi_set_sor_type and updated
 for the new features.
 (bidi_push_embedding_level, bidi_pop_embedding_level): Update to
 push and pop correctly for isolates.
 (bidi_remember_char): Modified to accept an additional argument
 and record the bidi type according to its value.
 (bidi_cache_iterator_state): Accept an additional argument to only
 update an existing state.  Handle the new members of struct bidi_it.
 (bidi_cache_find): Arguments changed: no lnger accepts a level,
 instead accepts a flag telling it whether it is okay to return
 unresolved neutrals.
 (bidi_initialize): Initiate and staticpro the bracket-type uniprop
 table.  Initialize new isolate-related members.
 (bidi_paragraph_init): Some code factored out into
 find_first_strong_char.
 (bidi_resolve_explicit_1): Function deleted, its code incorporated
 into bidi_resolve_explicit.
 (bidi_resolve_explicit): Support the isolate initiators and
 terminator.  Fix handling of embeddings and overrides according to
 new UBA requirements.  Record information about previously seen
 characters here (moved from bidi_level_of_next_char).
 (bidi_resolve_weak): Adapt to changes in struct members.
 (FLAG_EMBEDDING_INSIDE, FLAG_OPPOSITE_INSIDE, MAX_BPA_STACK)
 (STORE_BRACKET_CHARPOS, PUSH_BPA_STACK): New macros.
 (bidi_resolve_neutral): Call bidi_resolve_brackets to handle the
 paired bracket resolution.  Handle isolate initiators and
 terminator.
 (bidi_type_of_next_char): Remove unneeded code for BN limit.
 (bidi_level_of_next_char): Move the code that records information
 about previous characters to bidi_resolve_explicit.  Fix logic of
 resolving neutrals and make sure their cache entries are updated.
 Remove now unneeded special handling of PDF level.
 src/dispextern.h (struct glyph): Enlarge the width of resolved_level.
 (BIDI_MAXDEPTH): New macro, renamed from BIDI_MAXLEVEL and
 enlarged per Unicode 6.3.
 (enum bidi_bracket_type_t): New data type.
 (struct bidi_saved_info): Leave only 2 type members out of 4.
 Remove bytepos.
 (struct bidi_stack): Add members necessary to support isolating
 sequences.
 (struct bidi_it): Add new members necessary to support isolating
 sequences and bracket pair resolution.
 src/xdisp.c (Fbidi_resolved_levels): New function.
 (syms_of_xdisp): Defsubr it.
 (append_glyph, append_composite_glyph, produce_image_glyph)
 (append_stretch_glyph, append_glyphless_glyph): Convert aborts to
 assertions.
 (syms_of_xdisp) <inhibit-bidi-mirroring>: New variable.
 src/term.c (append_glyph, append_composite_glyph)
 (append_glyphless_glyph): Convert aborts to assertions.
 src/.gdbinit (pgx): Display the character codepoint, resolved level,
 and bidi type also for glyphless glyphs.

 lisp/simple.el (what-cursor-position): Update to support the new bidi
 characters.
 lisp/descr-text.el (describe-char): Update to support the new bidi
 characters.

 admin/unidata/unidata-gen.el (unidata-prop-alist): New properties
 'paired-bracket' and 'bracket-type', in support of the UBA 6.3.
 (unidata-gen-table): Support PROP-IDX being a function.
 (unidata-describe-bidi-bracket-type, unidata-gen-brackets-list)
 (unidata-gen-bracket-type-list): New functions.
 (unidata-check): Support checking the 'bracket-type' attribute.
 (unidata-gen-files): Don't create backups for uni-*.el files.
 admin/unidata/Makefile.in (${unidir}/charprop.el): Depend on
 BidiMirroring.txt and BidiBrackets.txt.
 admin/unidata/BidiBrackets.txt: New file, from Unicode.

 etc/NEWS: Mention the UBA implementation update.
 etc/HELLO: Remove now unneeded directional control characters.

 doc/lispref/nonascii.texi (Character Properties): Document the new
 properties 'bracket-type' and 'paired-bracket'.
 doc/lisprefdisplay.texi (Bidirectional Display): Update the version of the
 UBA to which we are conforming.

 test/BidiCharacterTest.txt: New file, from Unicode.
 test/biditest.el: New file.
parents 0b4d6d30 b8ec9776
......@@ -21,3 +21,4 @@ etc/refcards/*.aux
etc/refcards/*.log
info/dir
info/*.info
test/biditest.txt
2014-10-15 Eli Zaretskii <eliz@gnu.org>
* unidata/unidata-gen.el (unidata-prop-alist): New properties
'paired-bracket' and 'bracket-type', in support of the UBA 6.3.
(unidata-gen-table): Support PROP-IDX being a function.
(unidata-describe-bidi-bracket-type, unidata-gen-brackets-list)
(unidata-gen-bracket-type-list): New functions.
(unidata-check): Support checking the 'bracket-type' attribute.
(unidata-gen-files): Don't create backups for uni-*.el files.
* unidata/Makefile.in (${unidir}/charprop.el): Depend on
BidiMirroring.txt and BidiBrackets.txt.
* unidata/BidiBrackets.txt: New file, from Unicode.
2014-10-13 Glenn Morris <rgm@gnu.org>
* authors.el (authors-aliases, authors-fixed-case)
......
# BidiBrackets-7.0.0.txt
# Date: 2014-01-21, 02:30:00 GMT [AG, LI, KW]
#
# Bidi_Paired_Bracket and Bidi_Paired_Bracket_Type Properties
#
# This file is a normative contributory data file in the Unicode
# Character Database.
#
# Copyright (c) 1991-2014 Unicode, Inc.
# For terms of use, see http://www.unicode.org/terms_of_use.html
#
# Bidi_Paired_Bracket is a normative property of type Miscellaneous,
# which establishes a mapping between characters that are treated as
# bracket pairs by the Unicode Bidirectional Algorithm.
#
# Bidi_Paired_Bracket_Type is a normative property of type Enumeration,
# which classifies characters into opening and closing paired brackets
# for the purposes of the Unicode Bidirectional Algorithm.
#
# This file lists the set of code points with Bidi_Paired_Bracket_Type
# property values Open and Close. The set is derived from the character
# properties General_Category (gc), Bidi_Class (bc), Bidi_Mirrored (Bidi_M),
# and Bidi_Mirroring_Glyph (bmg), as follows: two characters, A and B,
# form a bracket pair if A has gc=Ps and B has gc=Pe, both have bc=ON and
# Bidi_M=Y, and bmg of A is B. Bidi_Paired_Bracket (bpb) maps A to B and
# vice versa, and their Bidi_Paired_Bracket_Type (bpt) property values are
# Open (o) and Close (c), respectively.
#
# For legacy reasons, the characters U+FD3E ORNATE LEFT PARENTHESIS and
# U+FD3F ORNATE RIGHT PARENTHESIS do not mirror in bidirectional display
# and therefore do not form a bracket pair.
#
# The Unicode property value stability policy guarantees that characters
# which have bpt=o or bpt=c also have bc=ON and Bidi_M=Y. As a result, an
# implementation can optimize the lookup of the Bidi_Paired_Bracket_Type
# property values Open and Close by restricting the processing to characters
# with bc=ON.
#
# The format of the file is three fields separated by a semicolon.
# Field 0: Unicode code point value, represented as a hexadecimal value
# Field 1: Bidi_Paired_Bracket property value, a code point value or <none>
# Field 2: Bidi_Paired_Bracket_Type property value, one of the following:
# o Open
# c Close
# n None
# The names of the characters in field 0 are given in comments at the end
# of each line.
#
# For information on bidirectional paired brackets, see UAX #9: Unicode
# Bidirectional Algorithm, at http://www.unicode.org/unicode/reports/tr9/
#
# This file was originally created by Andrew Glass and Laurentiu Iancu
# for Unicode 6.3.
0028; 0029; o # LEFT PARENTHESIS
0029; 0028; c # RIGHT PARENTHESIS
005B; 005D; o # LEFT SQUARE BRACKET
005D; 005B; c # RIGHT SQUARE BRACKET
007B; 007D; o # LEFT CURLY BRACKET
007D; 007B; c # RIGHT CURLY BRACKET
0F3A; 0F3B; o # TIBETAN MARK GUG RTAGS GYON
0F3B; 0F3A; c # TIBETAN MARK GUG RTAGS GYAS
0F3C; 0F3D; o # TIBETAN MARK ANG KHANG GYON
0F3D; 0F3C; c # TIBETAN MARK ANG KHANG GYAS
169B; 169C; o # OGHAM FEATHER MARK
169C; 169B; c # OGHAM REVERSED FEATHER MARK
2045; 2046; o # LEFT SQUARE BRACKET WITH QUILL
2046; 2045; c # RIGHT SQUARE BRACKET WITH QUILL
207D; 207E; o # SUPERSCRIPT LEFT PARENTHESIS
207E; 207D; c # SUPERSCRIPT RIGHT PARENTHESIS
208D; 208E; o # SUBSCRIPT LEFT PARENTHESIS
208E; 208D; c # SUBSCRIPT RIGHT PARENTHESIS
2308; 2309; o # LEFT CEILING
2309; 2308; c # RIGHT CEILING
230A; 230B; o # LEFT FLOOR
230B; 230A; c # RIGHT FLOOR
2329; 232A; o # LEFT-POINTING ANGLE BRACKET
232A; 2329; c # RIGHT-POINTING ANGLE BRACKET
2768; 2769; o # MEDIUM LEFT PARENTHESIS ORNAMENT
2769; 2768; c # MEDIUM RIGHT PARENTHESIS ORNAMENT
276A; 276B; o # MEDIUM FLATTENED LEFT PARENTHESIS ORNAMENT
276B; 276A; c # MEDIUM FLATTENED RIGHT PARENTHESIS ORNAMENT
276C; 276D; o # MEDIUM LEFT-POINTING ANGLE BRACKET ORNAMENT
276D; 276C; c # MEDIUM RIGHT-POINTING ANGLE BRACKET ORNAMENT
276E; 276F; o # HEAVY LEFT-POINTING ANGLE QUOTATION MARK ORNAMENT
276F; 276E; c # HEAVY RIGHT-POINTING ANGLE QUOTATION MARK ORNAMENT
2770; 2771; o # HEAVY LEFT-POINTING ANGLE BRACKET ORNAMENT
2771; 2770; c # HEAVY RIGHT-POINTING ANGLE BRACKET ORNAMENT
2772; 2773; o # LIGHT LEFT TORTOISE SHELL BRACKET ORNAMENT
2773; 2772; c # LIGHT RIGHT TORTOISE SHELL BRACKET ORNAMENT
2774; 2775; o # MEDIUM LEFT CURLY BRACKET ORNAMENT
2775; 2774; c # MEDIUM RIGHT CURLY BRACKET ORNAMENT
27C5; 27C6; o # LEFT S-SHAPED BAG DELIMITER
27C6; 27C5; c # RIGHT S-SHAPED BAG DELIMITER
27E6; 27E7; o # MATHEMATICAL LEFT WHITE SQUARE BRACKET
27E7; 27E6; c # MATHEMATICAL RIGHT WHITE SQUARE BRACKET
27E8; 27E9; o # MATHEMATICAL LEFT ANGLE BRACKET
27E9; 27E8; c # MATHEMATICAL RIGHT ANGLE BRACKET
27EA; 27EB; o # MATHEMATICAL LEFT DOUBLE ANGLE BRACKET
27EB; 27EA; c # MATHEMATICAL RIGHT DOUBLE ANGLE BRACKET
27EC; 27ED; o # MATHEMATICAL LEFT WHITE TORTOISE SHELL BRACKET
27ED; 27EC; c # MATHEMATICAL RIGHT WHITE TORTOISE SHELL BRACKET
27EE; 27EF; o # MATHEMATICAL LEFT FLATTENED PARENTHESIS
27EF; 27EE; c # MATHEMATICAL RIGHT FLATTENED PARENTHESIS
2983; 2984; o # LEFT WHITE CURLY BRACKET
2984; 2983; c # RIGHT WHITE CURLY BRACKET
2985; 2986; o # LEFT WHITE PARENTHESIS
2986; 2985; c # RIGHT WHITE PARENTHESIS
2987; 2988; o # Z NOTATION LEFT IMAGE BRACKET
2988; 2987; c # Z NOTATION RIGHT IMAGE BRACKET
2989; 298A; o # Z NOTATION LEFT BINDING BRACKET
298A; 2989; c # Z NOTATION RIGHT BINDING BRACKET
298B; 298C; o # LEFT SQUARE BRACKET WITH UNDERBAR
298C; 298B; c # RIGHT SQUARE BRACKET WITH UNDERBAR
298D; 2990; o # LEFT SQUARE BRACKET WITH TICK IN TOP CORNER
298E; 298F; c # RIGHT SQUARE BRACKET WITH TICK IN BOTTOM CORNER
298F; 298E; o # LEFT SQUARE BRACKET WITH TICK IN BOTTOM CORNER
2990; 298D; c # RIGHT SQUARE BRACKET WITH TICK IN TOP CORNER
2991; 2992; o # LEFT ANGLE BRACKET WITH DOT
2992; 2991; c # RIGHT ANGLE BRACKET WITH DOT
2993; 2994; o # LEFT ARC LESS-THAN BRACKET
2994; 2993; c # RIGHT ARC GREATER-THAN BRACKET
2995; 2996; o # DOUBLE LEFT ARC GREATER-THAN BRACKET
2996; 2995; c # DOUBLE RIGHT ARC LESS-THAN BRACKET
2997; 2998; o # LEFT BLACK TORTOISE SHELL BRACKET
2998; 2997; c # RIGHT BLACK TORTOISE SHELL BRACKET
29D8; 29D9; o # LEFT WIGGLY FENCE
29D9; 29D8; c # RIGHT WIGGLY FENCE
29DA; 29DB; o # LEFT DOUBLE WIGGLY FENCE
29DB; 29DA; c # RIGHT DOUBLE WIGGLY FENCE
29FC; 29FD; o # LEFT-POINTING CURVED ANGLE BRACKET
29FD; 29FC; c # RIGHT-POINTING CURVED ANGLE BRACKET
2E22; 2E23; o # TOP LEFT HALF BRACKET
2E23; 2E22; c # TOP RIGHT HALF BRACKET
2E24; 2E25; o # BOTTOM LEFT HALF BRACKET
2E25; 2E24; c # BOTTOM RIGHT HALF BRACKET
2E26; 2E27; o # LEFT SIDEWAYS U BRACKET
2E27; 2E26; c # RIGHT SIDEWAYS U BRACKET
2E28; 2E29; o # LEFT DOUBLE PARENTHESIS
2E29; 2E28; c # RIGHT DOUBLE PARENTHESIS
3008; 3009; o # LEFT ANGLE BRACKET
3009; 3008; c # RIGHT ANGLE BRACKET
300A; 300B; o # LEFT DOUBLE ANGLE BRACKET
300B; 300A; c # RIGHT DOUBLE ANGLE BRACKET
300C; 300D; o # LEFT CORNER BRACKET
300D; 300C; c # RIGHT CORNER BRACKET
300E; 300F; o # LEFT WHITE CORNER BRACKET
300F; 300E; c # RIGHT WHITE CORNER BRACKET
3010; 3011; o # LEFT BLACK LENTICULAR BRACKET
3011; 3010; c # RIGHT BLACK LENTICULAR BRACKET
3014; 3015; o # LEFT TORTOISE SHELL BRACKET
3015; 3014; c # RIGHT TORTOISE SHELL BRACKET
3016; 3017; o # LEFT WHITE LENTICULAR BRACKET
3017; 3016; c # RIGHT WHITE LENTICULAR BRACKET
3018; 3019; o # LEFT WHITE TORTOISE SHELL BRACKET
3019; 3018; c # RIGHT WHITE TORTOISE SHELL BRACKET
301A; 301B; o # LEFT WHITE SQUARE BRACKET
301B; 301A; c # RIGHT WHITE SQUARE BRACKET
FE59; FE5A; o # SMALL LEFT PARENTHESIS
FE5A; FE59; c # SMALL RIGHT PARENTHESIS
FE5B; FE5C; o # SMALL LEFT CURLY BRACKET
FE5C; FE5B; c # SMALL RIGHT CURLY BRACKET
FE5D; FE5E; o # SMALL LEFT TORTOISE SHELL BRACKET
FE5E; FE5D; c # SMALL RIGHT TORTOISE SHELL BRACKET
FF08; FF09; o # FULLWIDTH LEFT PARENTHESIS
FF09; FF08; c # FULLWIDTH RIGHT PARENTHESIS
FF3B; FF3D; o # FULLWIDTH LEFT SQUARE BRACKET
FF3D; FF3B; c # FULLWIDTH RIGHT SQUARE BRACKET
FF5B; FF5D; o # FULLWIDTH LEFT CURLY BRACKET
FF5D; FF5B; c # FULLWIDTH RIGHT CURLY BRACKET
FF5F; FF60; o # FULLWIDTH LEFT WHITE PARENTHESIS
FF60; FF5F; c # FULLWIDTH RIGHT WHITE PARENTHESIS
FF62; FF63; o # HALFWIDTH LEFT CORNER BRACKET
FF63; FF62; c # HALFWIDTH RIGHT CORNER BRACKET
# EOF
......@@ -54,7 +54,9 @@ FORCE =
FORCE:
.PHONY: FORCE
${unidir}/charprop.el: ${FORCE} ${srcdir}/unidata-gen.el ${srcdir}/UnicodeData.txt | \
${unidir}/charprop.el: ${FORCE} ${srcdir}/unidata-gen.el \
${srcdir}/UnicodeData.txt ${srcdir}/BidiMirroring.txt \
${srcdir}/BidiBrackets.txt | \
${srcdir}/unidata-gen.elc unidata.txt
-if [ -f "$@" ]; then \
cd ${unidir} && chmod +w charprop.el `sed -n 's/^;; FILE: //p' < charprop.el`; \
......
......@@ -154,7 +154,8 @@
;; PROP: character property
;; INDEX: index to each element of unidata-list for PROP.
;; It may be a function that generates an alist of character codes
;; vs. the corresponding property values.
;; vs. the corresponding property values. Currently, only character
;; codepoints or symbol values are supported in this case.
;; GENERATOR: function to generate a char-table
;; FILENAME: filename to store the char-table
;; DOCSTRING: docstring for the property
......@@ -273,7 +274,23 @@ is the character itself."
"Unicode bidi-mirroring characters.
Property value is a character that has the corresponding mirroring image or nil.
The value nil means that the actual property value of a character
is the character itself.")))
is the character itself.")
(paired-bracket
unidata-gen-brackets-list unidata-gen-table-character "uni-brackets.el"
"Unicode bidi paired-bracket characters.
Property value is the paired bracket character, or nil.
The value nil means that the character is neither an opening nor
a closing paired bracket."
string)
(bracket-type
unidata-gen-bracket-type-list unidata-gen-table-symbol "uni-brackets.el"
"Unicode bidi paired-bracket type.
Property value is a symbol `o' (Open), `c' (Close), or `n' (None)."
unidata-describe-bidi-bracket-type
n
;; The order of elements must be in sync with bidi_bracket_type_t
;; in src/dispextern.h.
(n o c))))
;; Functions to access the above data.
(defsubst unidata-prop-index (prop) (nth 1 (assq prop unidata-prop-alist)))
......@@ -451,7 +468,10 @@ is the character itself.")))
(unidata-encode-val val-list (nth 2 elm)))
(set-char-table-range table (cons (car elm) (nth 1 elm)) (nth 2 elm)))
(setq tail unidata-list)
(if (functionp prop-idx)
(setq tail (funcall prop-idx)
prop-idx 1)
(setq tail unidata-list))
(while tail
(setq elt (car tail) tail (cdr tail))
(setq range (car elt)
......@@ -1157,6 +1177,12 @@ is the character itself.")))
(string ?'))))
val " "))
(defun unidata-describe-bidi-bracket-type (val)
(cdr (assq val
'((n . "Not a paired bracket character.")
(o . "Opening paired bracket character.")
(c . "Closing paired bracket character.")))))
(defun unidata-gen-mirroring-list ()
(let ((head (list nil))
tail)
......@@ -1170,6 +1196,36 @@ is the character itself.")))
(setq tail (setcdr tail (list (list char mirror)))))))
(cdr head)))
(defun unidata-gen-brackets-list ()
(let ((head (list nil))
tail)
(with-temp-buffer
(insert-file-contents (expand-file-name "BidiBrackets.txt" unidata-dir))
(goto-char (point-min))
(setq tail head)
(while (re-search-forward
"^\\([0-9A-F]+\\);\\s +\\([0-9A-F]+\\);\\s +\\([oc]\\)"
nil t)
(let ((char (string-to-number (match-string 1) 16))
(paired (match-string 2)))
(setq tail (setcdr tail (list (list char paired)))))))
(cdr head)))
(defun unidata-gen-bracket-type-list ()
(let ((head (list nil))
tail)
(with-temp-buffer
(insert-file-contents (expand-file-name "BidiBrackets.txt" unidata-dir))
(goto-char (point-min))
(setq tail head)
(while (re-search-forward
"^\\([0-9A-F]+\\);\\s +\\([0-9A-F]+\\);\\s +\\([oc]\\)"
nil t)
(let ((char (string-to-number (match-string 1) 16))
(type (match-string 3)))
(setq tail (setcdr tail (list (list char type)))))))
(cdr head)))
;; Verify if we can retrieve correct values from the generated
;; char-tables.
;;
......@@ -1218,7 +1274,9 @@ is the character itself.")))
((eq generator 'unidata-gen-table-decomposition)
(setq val1 (unidata-split-decomposition val1))))
(cond ((eq prop 'decomposition)
(setq val1 (list char)))))
(setq val1 (list char)))
((eq prop 'bracket-type)
(setq val1 'n))))
(when (>= char check)
(message "%S %04X" prop check)
(setq check (+ check #x400)))
......@@ -1261,6 +1319,9 @@ is the character itself.")))
(describer (unidata-prop-describer prop))
(default-value (unidata-prop-default prop))
(val-list (unidata-prop-val-list prop))
;; Avoid creating backup files for those uni-*.el files
;; that hold more than one table.
(backup-inhibited t)
table)
;; Filename in this comment line is extracted by sed in
;; Makefile.
......
2014-10-15 Eli Zaretskii <eliz@gnu.org>
* nonascii.texi (Character Properties): Document the new
properties 'bracket-type' and 'paired-bracket'.
* display.texi (Bidirectional Display): Update the version of the
UBA to which we are conforming.
2014-10-13 Glenn Morris <rgm@gnu.org>
* Makefile.in (dist): Update for new output variables.
......
......@@ -6613,10 +6613,9 @@ positions do not increase monotonically with string or buffer
position. In performing this @dfn{bidirectional reordering}, Emacs
follows the Unicode Bidirectional Algorithm (a.k.a.@: @acronym{UBA}),
which is described in Annex #9 of the Unicode standard
(@url{http://www.unicode.org/reports/tr9/}). Emacs currently provides
a ``Non-isolate Bidirectionality'' class implementation of the
@acronym{UBA}: it does not yet support the isolate directional
formatting characters introduced with Unicode Standard v6.3.0.
(@url{http://www.unicode.org/reports/tr9/}). Emacs provides a ``Full
Bidirectionality'' class implementation of the @acronym{UBA},
consistent with the requirements of the Unicode Standard v7.0.
@defvar bidi-display-reordering
If the value of this buffer-local variable is non-@code{nil} (the
......
......@@ -520,6 +520,24 @@ property to display mirror images of characters when appropriate
(@pxref{Bidirectional Display}). For unassigned codepoints, the value
is @code{nil}.
@item paired-bracket
Corresponds to the Unicode @code{Bidi_Paired_Bracket} property. The
value of this property is the codepoint of a character's @dfn{paired
bracket}, or @code{nil} if the character is not a bracket character.
This establishes a mapping between characters that are treated as
bracket pairs by the Unicode Bidirectional Algorithm; Emacs uses this
property when it decides how to reorder for display parentheses,
braces, and other similar characters (@pxref{Bidirectional Display}).
@item bracket-type
Corresponds to the Unicode @code{Bidi_Paired_Bracket_Type} property.
For characters whose @code{paired-bracket} property is non-@code{nil},
the value of this property is a symbol, either @code{o} (for opening
bracket characters) or @code{c} (for closing bracket characters). For
characters whose @code{paired-bracket} property is @code{nil}, the
value is the symbol @code{n} (None). Like @code{paired-bracket}, this
property is used for bidirectional display.
@item old-name
Corresponds to the Unicode @code{Unicode_1_Name} property. The value
is a string. Unassigned codepoints, and characters that have no value
......@@ -574,6 +592,14 @@ This function returns the value of @var{char}'s @var{propname} property.
(get-char-code-property ?\u2163 'numeric-value)
@result{} 4
@end group
@group
(get-char-code-property ?\( 'paired-bracket)
@result{} 41 ;; closing parenthesis
@end group
@group
(get-char-code-property ?\) 'bracket-type)
@result{} c
@end group
@end example
@end defun
......
2014-10-15 Eli Zaretskii <eliz@gnu.org>
* NEWS: Mention the UBA implementation update.
* HELLO: Remove now unneeded directional control characters.
2014-10-13 Jan Djärv <jan.h.d@swipnet.se>
* NEWS: Move and clarify OSX >= 10.6.
......
......@@ -18,7 +18,7 @@ Non-ASCII examples:
LANGUAGE (NATIVE NAME) HELLO
---------------------- -----
Amharic ($,1O M[MmN{(B) $,1M`MKM](B
Arabic $,1ro(B($,1-g.$-y-q-h.*.1-i(B) $,1-g.$-s.1.$-g.%(B $,1-y.$.*.#.%(B
Arabic ($,1-g.$-y-q-h.*.1-i(B) $,1-g.$-s.1.$-g.%(B $,1-y.$.*.#.%(B
Bengali ($,17,7>6b727>(B) $,17(7.787M6u7>70(B
Braille $,2(3(1('('(5(B
Burmese ($,1H9H\H4HZH9HL(B) $,1H9H$HZHYH"H<HLH5HK(B
......@@ -37,7 +37,7 @@ German (Deutsch) Guten Tag / Gr,A|_(B Gott
Greek (,Fekkgmij\(B) ,FCei\(B ,Fsar(B
Greek, ancient ($,1p1,Fkkgmij^(B) ,FO$,1pv,Fk](B ,Fte(B ,Fja$,1q6(B ,Fl]ca(B ,Fwa$,1r6,Fqe(B
Gujarati ($,19W:!9\9p9~9d: (B) $,19h9n9x:-9d:'(B
Hebrew $,1ro(B($,1-",q-(,y-*(B) ,Hylem(B
Hebrew ($,1-",q-(,y-*(B) ,Hylem(B
Hungarian (magyar) Sz,Bi(Bp j,Bs(B napot!
Hindi ($,15y55B5f6 (B) $,15h5n5x6-5d6'(B / $,15h5n5x6-5U5~5p(B $,16D(B
Italian (italiano) Ciao / Buon giorno
......
......@@ -110,6 +110,14 @@ character in the pasted text as actual user input. This results in a
paste experience similar to that under a window system, and significant
performance improvements when pasting large amounts of text.
** Emacs now supports the latest version of the UBA.
The Emacs implementation of the Unicode Bidirectional Algorithm (UBA)
was updated to support all the latest additions and changes introduced
in Unicode Standard versions 6.3 and 7.0, and a few changes suggested
for Unicode 8.0. This includes full support for directional isolates
and the Bidirectional Parentheses Algorithm (BPA) specified by these
Unicode standards.
* Changes in Specialized Modes and Packages in Emacs 25.1
......
2014-10-15 Eli Zaretskii <eliz@gnu.org>
* simple.el (what-cursor-position): Update to support the new bidi
characters.
* descr-text.el (describe-char): Update to support the new bidi
characters.
* emacs-lisp/tabulated-list.el (tabulated-list-mode): Force
bidi-paragraph-direction to 'left-to-right'. This fixes
buffer-menu display when the first buffer happens to start with
......
......@@ -434,13 +434,26 @@ relevant to POS."
code (encode-char char charset)))
(setq code char))
(cond
;; Append a PDF character to directional embeddings and
;; overrides, to prevent potential messup of the following
;; text.
((memq char '(?\x202a ?\x202b ?\x202d ?\x202e))
;; Append a PDF character to left-to-right directional
;; embeddings and overrides, to prevent potential messup of the
;; following text.
((memq char '(?\x202a ?\x202d))
(setq char-description
(concat char-description
(propertize (string ?\x202c) 'invisible t))))
;; Append a PDF character followed by LRM to right-to-left
;; directional embeddings and overrides, to prevent potential
;; messup of the following numerical text.
((memq char '(?\x202b ?\x202e))
(setq char-description
(concat char-description
(propertize (string ?\x202c ?\x200e) 'invisible t))))
;; Append a PDI character to directional isolate initiators, to
;; prevent potential messup of the following numerical text
((memq char '(?\x2066 ?\x2067 ?\x2068))
(setq char-description
(concat char-description
(propertize (string ?\x2069) 'invisible t))))
;; Append a LRM character to any strong character to avoid
;; messing up the numerical codepoint.
((memq (get-char-code-property char 'bidi-class) '(R AL))
......
......@@ -1223,15 +1223,21 @@ in *Help* buffer. See also the command `describe-char'."
(interactive "P")
(let* ((char (following-char))
(bidi-fixer
(cond ((memq char '(?\x202a ?\x202b ?\x202d ?\x202e))
;; If the character is one of LRE, LRO, RLE, RLO, it
;; will start a directional embedding, which could
;; completely disrupt the rest of the line (e.g., RLO
;; will display the rest of the line right-to-left).
;; So we put an invisible PDF character after these
;; characters, to end the embedding, which eliminates
;; any effects on the rest of the line.
;; If the character is one of LRE, LRO, RLE, RLO, it will
;; start a directional embedding, which could completely
;; disrupt the rest of the line (e.g., RLO will display the
;; rest of the line right-to-left). So we put an invisible
;; PDF character after these characters, to end the
;; embedding, which eliminates any effects on the rest of
;; the line. For RLE and RLO we also append an invisible
;; LRM, to avoid reordering the following numerical
;; characters. For LRI/RLI/FSI we append a PDI.
(cond ((memq char '(?\x202a ?\x202d))
(propertize (string ?\x202c) 'invisible t))
((memq char '(?\x202b ?\x202e))
(propertize (string ?\x202c ?\x200e) 'invisible t))
((memq char '(?\x2066 ?\x2067 ?\x2068))
(propertize (string ?\x2069) 'invisible t))
;; Strong right-to-left characters cause reordering of
;; the following numerical characters which show the
;; codepoint, so append LRM to countermand that.
......
......@@ -468,18 +468,18 @@ define pgx
end
# GLYPHLESS_GLYPH
if ($g.type == 2)
printf "GLYPHLESS["
printf "G-LESS["
if ($g.u.glyphless.method == 0)
printf "THIN]"
printf "THIN;0x%x]", $g.u.glyphless.ch
end
if ($g.u.glyphless.method == 1)
printf "EMPTY]"
printf "EMPTY;0x%x]", $g.u.glyphless.ch
end
if ($g.u.glyphless.method == 2)
printf "ACRO]"
printf "ACRO;0x%x]", $g.u.glyphless.ch
end
if ($g.u.glyphless.method == 3)
printf "HEX]"
printf "HEX;0x%x]", $g.u.glyphless.ch
end
end
# IMAGE_GLYPH
......@@ -498,7 +498,7 @@ define pgx
printf " pos=%d", $g.charpos
end
# For characters, print their resolved level and bidi type
if ($g.type == 0)
if ($g.type == 0 || $g.type == 2)
printf " blev=%d,btyp=", $g.resolved_level
pbiditype $g.bidi_type
end
......
2014-10-15 Eli Zaretskii <eliz@gnu.org>
Update the bidirectional reordering engine for Unicode 6.3 and 7.0.
* bidi.c (bidi_ignore_explicit_marks_for_paragraph_level): Remove
variable.
(bidi_get_type): Return the isolate initiators and terminator
types.
(bidi_isolate_fmt_char, bidi_paired_bracket_type)
(bidi_fetch_char_skip_isolates, find_first_strong_char)
(bidi_find_bracket_pairs, bidi_resolve_brackets): New functions.
(bidi_set_sos_type): Renamed from bidi_set_sor_type and updated
for the new features.
(bidi_push_embedding_level, bidi_pop_embedding_level): Update to
push and pop correctly for isolates.
(bidi_remember_char): Modified to accept an additional argument
and record the bidi type according to its value.
(bidi_cache_iterator_state): Accept an additional argument to only
update an existing state. Handle the new members of struct bidi_it.
(bidi_cache_find): Arguments changed: no lnger accepts a level,
instead accepts a flag telling it whether it is okay to return
unresolved neutrals.
(bidi_initialize): Initiate and staticpro the bracket-type uniprop
table. Initialize new isolate-related members.
(bidi_paragraph_init): Some code factored out into
find_first_strong_char.
(bidi_resolve_explicit_1): Function deleted, its code incorporated
into bidi_resolve_explicit.
(bidi_resolve_explicit): Support the isolate initiators and
terminator. Fix handling of embeddings and overrides according to
new UBA requirements. Record information about previously seen
characters here (moved from bidi_level_of_next_char).
(bidi_resolve_weak): Adapt to changes in struct members.
(FLAG_EMBEDDING_INSIDE, FLAG_OPPOSITE_INSIDE, MAX_BPA_STACK)
(STORE_BRACKET_CHARPOS, PUSH_BPA_STACK): New macros.
(bidi_resolve_neutral): Call bidi_resolve_brackets to handle the
paired bracket resolution. Handle isolate initiators and
terminator.
(bidi_type_of_next_char): Remove unneeded code for BN limit.
(bidi_level_of_next_char): Move the code that records information
about previous characters to bidi_resolve_explicit. Fix logic of
resolving neutrals and make sure their cache entries are updated.
Remove now unneeded special handling of PDF level.
* dispextern.h (struct glyph): Enlarge the width of resolved_level.
(BIDI_MAXDEPTH): New macro, renamed from BIDI_MAXLEVEL and
enlarged per Unicode 6.3.
(enum bidi_bracket_type_t): New data type.
(struct bidi_saved_info): Leave only 2 type members out of 4.
Remove bytepos.
(struct bidi_stack): Add members necessary to support isolating
sequences.
(struct bidi_it): Add new members necessary to support isolating
sequences and bracket pair resolution.
* xdisp.c (Fbidi_resolved_levels): New function.
(syms_of_xdisp): Defsubr it.
(append_glyph, append_composite_glyph, produce_image_glyph)
(append_stretch_glyph, append_glyphless_glyph): Convert aborts to
assertions.
(syms_of_xdisp) <inhibit-bidi-mirroring>: New variable.
* term.c (append_glyph, append_composite_glyph)
(append_glyphless_glyph): Convert aborts to assertions.
* .gdbinit (pgx): Display the character codepoint, resolved level,
and bidi type also for glyphless glyphs.
2014-10-15 Dmitry Antipov <dmantipov@yandex.ru>
Avoid unwanted point motion in Fline_beginning_position.
......
This diff is collapsed.
......@@ -445,8 +445,8 @@ struct glyph
/* True means don't display cursor here. */
bool_bf avoid_cursor_p : 1;
/* Resolved bidirectional level of this character [0..63]. */
unsigned resolved_level : 5;
/* Resolved bidirectional level of this character [0..127]. */
unsigned resolved_level : 7;
/* Resolved bidirectional type of this character, see enum
bidi_type_t below. Note that according to UAX#9, only some
......@@ -1857,7 +1857,9 @@ GLYPH_CODE_P (Lisp_Object gc)
extern int face_change_count;
/* For reordering of bidirectional text. */
#define BIDI_MAXLEVEL 64
/* UAX#9's max_depth value. */
#define BIDI_MAXDEPTH 125
/* Data type for describing the bidirectional character types. The
first 7 must be at the beginning, because they are the only values
......@@ -1894,23 +1896,39 @@ typedef enum {
NEUTRAL_ON /* other neutrals */
} bidi_type_t;
/* Data type for describing the Bidi Paired Bracket Type of a character.
The order of members must be in sync with the 8th element of the
member of unidata-prop-alist (in admin/unidata/unidata-gen.el) for
Unicode character property `bracket-type'. */
typedef enum {
BIDI_BRACKET_NONE = 1,
BIDI_BRACKET_OPEN,
BIDI_BRACKET_CLOSE
} bidi_bracket_type_t;
/* The basic directionality data type. */
typedef enum { NEUTRAL_DIR, L2R, R2L } bidi_dir_t;
/* Data type for storing information about characters we need to
remember. */
struct bidi_saved_info {
ptrdiff_t bytepos, charpos; /* character's buffer position */