GitLab

* lisp/progmodes/python.el (python-shell-prompt-block-regexp): Match
also IPython's multiline prompt.  It allows to correctly disable
non-native completions during multiline statement in
inferior-python-mode that runs IPython interpreter (Bug#34582).

Copyright-paperwork-exempt: yes

* lisp/emacs-lisp/smie.el (smie-indent-fixindent):
* lisp/cedet/semantic/doc.el (semantic-doc-snarf-comment-for-tag):
* lisp/progmodes/fortran.el (fortran-previous-statement)
(fortran-next-statement)
(fortran-fill-statement):
* lisp/progmodes/vhdl-mode.el (vhdl-beginning-of-statement): Bracket
comment-start-skip and comment-end-skip to avoid unexpected regexp
operator precedence.

(python-shell-prompt-detect): Use unwind-protect to try and not leave
file behind in case of error.

* lisp/progmodes/project.el (project--files-in-directory):
Sort the files alphabetically
(https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23179#296).

* lisp/progmodes/xref.el (xref-file-header, xref-line-number)
(xref-match):  New faces.
(xref--insert-xrefs, xref--collect-matches-1): Use them.

* lisp/progmodes/cc-langs.el (c-class-id-suffix-ws-ids-kwds)
(c-class-id-suffix-ws-ids-key): New lang const/var.

* lisp/progmodes/cc-engine.el (c-guess-basic-syntax CASE 5D.4): Check for and
skip over any matches for c-class-id-suffix-ws-ids-key (i.e. "final") before
":".

Problem reported by Mattias Engdegård in:
https://lists.gnu.org/r/emacs-devel/2019-04/msg00527.html
* lisp/align.el (align-rules-list):
* lisp/cedet/srecode/srt-mode.el (srecode-font-lock-keywords):
* lisp/emacs-lisp/copyright.el (copyright-regexp):
* lisp/erc/erc-backend.el (JOIN):
* lisp/erc/erc-goodies.el (erc-unmorse):
* lisp/mail/mail-extr.el (mail-extr-telephone-extension-pattern):
* lisp/net/tramp-adb.el (tramp-adb-prompt):
* lisp/org/org-table.el (org-table-range-regexp):
* lisp/progmodes/idlwave.el (idlwave-where):
* lisp/progmodes/verilog-mode.el (verilog-declaration-re-2-no-macro)
(verilog-declaration-re-2-macro, verilog-delete-auto-buffer)
(verilog-auto-inst-port):
* lisp/url/url-misc.el (url-data):
Omit or rewrite useless repetitions that risk being very slow in
the backtracking regexp engine in Emacs.

Also firm up detection of beginning of brace list in
c-looking-at-or-maybe-in-bracelist.

* lisp/progmodes/cc-engine.el (c-looking-at-or-maybe-in-bracelist): On
detection of such a ref-qualifier, set braceassignp to nil.  When this
variable has a nil value, return nil as the value of the function.  On
encountering a } when scanning backwards, recognise this as the end of a
previous construct and stop the scan.

* lisp/progmodes/grep.el (grep-read-files): Allow major modes to
define file name to use for default search pattern.
Add non-directory file at point as default search pattern candidate.

* lisp/dired.el (dired-grep-read-files): Use non-directory file at
point for grep file name pattern.  (Bug#34621)

Copyright-paperwork-exempt: yes

* lisp/progmodes/js.el (js-jsx--put-syntax-table): New function for
consistently ensuring smooth js2-mode integration.  js2-mode sets
syntax-table temporarily while parsing buffers—seemingly to recover
from parsing interruptions—and then it later clears syntax-table
blindly.  When integrating with js-mode, this means that unterminated
string quotes are re-broken in JSX (i.e., they become strings again,
often stringifying large regions of the buffer which should not be
strings).  We try to treat quotes in JSXText as non-strings by setting
syntax-table to a non-“string quote” syntax class, but that stops
working if we lose the property.  On the js2-mode end, by scanning for
this second js-jsx-syntax-table property, we can recover the
syntax-table property there.
(js-jsx--text-range, js-jsx--syntax-propertize-tag): Use
js-jsx--put-syntax-table for above reason.
(js-jsx--text-properties): Clear the js-jsx-syntax-table property too.

* lisp/progmodes/js.el (js-jsx-align->-with-<): New variable for users
to control one opinionated aspect of JSX indentation.  It defaults to
the style seen in the React docs, which many users expected as the
“correct” indentation.  Still, the old SGML-style of indentation could
be desirable too, especially since it was the old default.  This
ensures users have a way of getting back the old behavior.
(js-jsx--contextual-indentation): Respect js-jsx-align->-with-<.

* test/manual/indent/jsx-align-gt-with-lt.jsx: New test for
js-jsx-align->-with-<.

* lisp/progmodes/js.el (js-jsx-indent-level): New variable for users
to set JSX indentation differently than JS, like before.
(js-jsx--contextual-indentation): Respect js-jsx-indent-level when
it’s set.

* test/manual/indent/jsx-indent-level.jsx: New test for
js-jsx-indent-level.

* .dir-locals.el: Set bug-reference-url-format in all modes, not just
changelog mode. Use (eval . (bug-reference-mode)) as described
in (info "(emacs) Specifying File Variables")
* lisp/progmodes/bug-reference.el: Use lexical binding.
(bug-reference-unfontify):
(bug-reference-fontify): Mention args in docstring.

Bug#35123

* lisp/progmodes/python.el (python-syntax-propertize-function):
Only mark triple-quoted strings, let the normal syntax-table handle
the rest.
(python-syntax-stringify): Adjust accordingly.

* lisp/progmodes/js.el: Throughout the code, provide explanations for
why JSX support was implemented in the way that it was; in particular,
address the overlap between syntax-propertize-function, font-lock, and
indentation (as requested by Stefan).

* lisp/progmodes/js.el (js-jsx--enclosing-curly-pos)
(js-jsx--goto-outermost-enclosing-curly): As the code evolved, these
functions’ definitions ended up being far away from the only places
where they were used.  Move them there.

* lisp/progmodes/js.el (js-jsx--syntax-propertize-tag): Like in
sgml-mode, treat “<” and “>” like open/close parenthesis, making the
text more navigable via forward-sexp, etc.

* lisp/progmodes/js.el (js--name-start-re): Generally allow
identifiers to begin with non-ASCII letters.  This is of particular
importance to JSX parsing.

* test/manual/indent/jsx-unclosed-2.jsx: Add test to ensure non-ASCII
characters are parsed properly.

* lisp/progmodes/js.el (js-jsx--syntax-propertize-tag): Derived modes
like js2-mode may use font-lock-syntactic-face-function to apply faces
to JSX strings (and only JSX strings).  Apply the js-jsx-string text
property to such strings so they can be distinctly identified.
(js-jsx--text-properties): Ensure the js-jsx-string text property gets
cleaned up, too.

Use mode-line-format constructs to properly set mode-name, rather than
use the very hacky solution that was filling-in for my lack of
knowledge of this feature.

* lisp/progmodes/js.el (js--update-mode-name)
(js--idly-update-mode-name): Remove.

(js--syntactic-mode-name-part): New helper function for mode-name.
(js-use-syntactic-mode-name): Helper to set up the dynamic mode-name.

(js-jsx-enable): Don’t need to call any extra functions now.
(js-mode): Use the new setup function rather than the old ones.

(js-jsx-mode): Use the same initial mode name as js-mode so the final
one is identical for both modes.

* lisp/progmodes/js.el (js--name-start-chars): Remove, adding these
chars back to js--name-start-re.
(js--name-start-re): Add chars back from js--name-start-chars.

(js-jsx--tag-start-re): Improve regexp to capture the tag name (so it
can be disambiguated from a unary keyword), to match newlines (which
are common in this spot), and to require at least one whitespace
character before the attribute name.

(js-jsx--matched-tag-type): Ensure the “tag name” isn’t possibly a
unary keyword.

(js-jsx--self-closing-re, js-jsx--matching-close-tag-pos): Allow
whitespace around “<” and “>”.

* test/manual/indent/jsx-unclosed-2.jsx: Add tests for unary keyword
and whitespace parsing.

* lisp/progmodes/js.el (js-jsx--enclosing-tag-pos): Update docstring
to be more precise.  Also, remember close tag positions after they’ve
been calculated once to avoid many redundant calls to
js-jsx--matching-close-tag-pos.
(js-jsx--text-properties): Ensure js-jsx-close-tag-pos text properties
get cleaned up, too.

This function’s performance was having a noticeable impact when
editing large JSX structures.  Improve its performance
slightly (elapsed time will be cut in half according to ELP).

* lisp/progmodes/js.el (js-jsx--tag-re): Remove.
(js-jsx--matched-tag-type): Simplify implementation with respect to
the new implementation of js-jsx--matching-close-tag-pos.
(js-jsx--self-closing-re): Simplify regexp slightly in sync with a
generally simpler matching algorithm.
(js-jsx--matching-close-tag-pos): Optimize matching algorithm by using
multiple simple regexp searches, rather than one big complex search.

* test/manual/indent/jsx-unclosed-2.jsx: Use the term “inequality” and
add a test for a possible parsing foible.

* lisp/progmodes/js.el (js-jsx--contextual-indentation)
(js-jsx--expr-attribute-pos, js-jsx--expr-indentation): Extract logic
from js-jsx--indentation, and improve the logic’s documentation.
(js-jsx--indentation): Simplify by splitting into several
functions (see above) and improve the logic’s documentation.

* lisp/progmodes/js.el (js-jsx--syntax-propertize-tag): Refer to the
beginning of a JSXExpressionContainer’s associated JSXAttribute (so
line numbers can be calculated later).
(js-jsx--text-properties): Also clear the new text property
js-jsx-expr-attribute.

(js-jsx--indenting): Remove.
(js-jsx--indent-col, js-jsx--indent-attribute-line): New variables.
(js-jsx--indentation): Instead of alternating between two separate
column calculations, neither necessarily correct, bind the JSX column
such that the second call to js--proper-indentation can use it as a
base column.
(js--proper-indentation): Use JSX as the base column for some indents
while indenting JSX.

* test/manual/indent/jsx.jsx: Add more tests for expression indents.

* lisp/progmodes/js.el (js-jsx--matching-close-tag-pos): Fix bug where
self-closing JSXOpeningElements might be missed if one was nested
within another.

* test/manual/indent/jsx-self-closing.jsx: Add test for bug concerning
self-closing JSXOpeningElement counting.

* lisp/progmodes/js.el (js--line-terminating-arrow-re): Revise regexp
for use with re-search-backward.
(js--looking-at-broken-arrow-function-p): Remove.
(js--broken-arrow-terminates-line-p): Replacement for
js--looking-at-broken-arrow-function-p.  Don’t consider whether an
arrow appears at point (in an arglist); instead, just look for an
arrow that terminates the line.
(js--proper-indentation): Use js--broken-arrow-terminates-line-p.

* test/manual/indent/js.js: Add test for a broken arrow as an N+1th
arg.

* lisp/progmodes/js.el (js-jsx--attribute-name-re): New variable.
(js-jsx--syntax-propertize-tag): Allow “-” in JSXAttribute names.  Fix
“out of range” error when typing at the end of a buffer.  Fix/improve
future propertization of unfinished JSXBoundaryElements.

* test/manual/indent/js-jsx-unclosed-2.js: Add tests for allowed
characters in JSX.

* lisp/files.el (auto-mode-alist): Simply enable
javascript-mode (js-mode) when opening “.jsx” files, since the “.jsx”
file extension will be used as an indicator of JSX syntax by js-mode,
and more code is likely to work in js-mode than js-jsx-mode, and we
probably want to guide users to use js-mode (with js-jsx-syntax)
instead.  Code that used to work exclusively in js-jsx-mode (if anyone
ever wrote any) ought to be updated to work in js-mode too when
js-jsx-syntax is set to t.

* lisp/progmodes/js.el (js-jsx-detect-syntax, js-jsx-regexps)
(js-jsx--detect-and-enable, js-jsx--detect-after-change): New
variables and functions for detecting and enabling JSX.

(js-jsx-syntax): Update docstring with respect to the widened scope of
the effects and use of this variable.

(js-syntactic-mode-name, js--update-mode-name)
(js--idly-update-mode-name, js-jsx-enable): New variable and functions
for indicating when JSX is enabled.

(js-mode): Detect and enable JSX.  Print all enabled syntaxes after
the mode name whenever Emacs goes idle; this ensures lately-enabled
syntaxes are evident.

(js-jsx-mode): Update mode name for consistency with the state in
which JSX is enabled in js-mode.  Update docstring to suggest
alternative means of using JSX without this mode.  Going forward, it
may be best to gently guide users away from js-jsx-mode, since a “one
mode per syntax extension” model would not scale well if more syntax
extensions were to be simultaneously supported (e.g. Facebook’s
“Flow”).

This removes the last dependency on sgml-mode for JSX-related logic.

* lisp/progmodes/js.el (js-jsx--start-tag-re)
(js-jsx--end-tag-re): Remove.
(js-jsx--looking-at-start-tag-p)
(js-jsx--looking-back-at-end-tag-p): Reimplement using text
properties, using syntax information which ought to be slightly more
accurate than regexps since it was found by complete parsing.

Fixes the following issues (and re-fixes indentation issues initially
fixed but later re-broken by previous commits in the process of adding
comprehensive JSX support):

- https://github.com/mooz/js2-mode/issues/389#issuecomment-390766873
- https://github.com/mooz/js2-mode/issues/482
- Bug#32158
- https://github.com/mooz/js2-mode/issues/462

Previously, we delegated to sgml-mode functions for JSX indentation.
However, there were some problems with this approach:

- sgml-mode does not anticipate tags inside attributes when indenting,
  which compromises JSX indentation inside JSXExpressionContainers
  inside JSXAttributes.

- In previous iterations to provide comprehensive JSX support, it
  proved tedious to disambiguate “<” and “>” as JS inequality
  operators and arrow functions from opening and closing angle
  brackets as part of SGML tags.  That code evolved into a more
  complete JSX parsing implementation for syntax-propertize rules for
  font-locking, discarding the superfluous “<”/“>” disambiguation in
  anticipation of using the improved JSX analysis for indentation.

- Using sgml-mode functions, we controlled JSX indentation using SGML
  variables.  However, JSX is a different thing than SGML; referencing
  SGML in JS was a leaky abstraction.

To resolve these issues, use the text properties added by the JSX
syntax-propertize code to determine the boundaries of various aspects
of JSX syntax, and reimplement the sgml-mode indentation code in
js-mode with better respect to JSX indentation conventions.

* lisp/progmodes/js.el (js-jsx-attribute-offset): New variable to
provide a way for users to still control JSX attribute offsets as they
could with sgml-attribute-offset before.  The value of this feature is
dubious IMO, but it’s trivial to keep it, so let’s do it just in case.

(js-jsx--goto-outermost-enclosing-curly): New function.

(js-jsx--enclosing-tag-pos): Refactor to be unbounded by curlies, so
this function can be used to find JSXExpressionContainers within JSX.
Fix bug where an enclosing JSXElement couldn’t be found when point was
at the start of its JSXClosingElement.  Return the JSXClosingElement’s
position as well, so the JSXClosingElement can be indentified when
indenting and be indented like the matching JSXOpeningElement.

(js-jsx--at-enclosing-tag-child-p): js-jsx--enclosing-tag-pos now
returns a list rather than a cons, so retrieve the JSXOpeningElement’s
end position from a list.

(js-jsx--context, js-jsx--indenting): New function and variable.
(js-jsx--indentation): New function replacing the prior
js-jsx--indent* functions and js-jsx-indent-line’s implementation.
Use the JSX parsing performed in a JS context to more accurately
calculate JSX indentation than by delegating to sgml-mode functions.
(js--proper-indentation): Use js-jsx--indentation as yet another type
of indentation.
(js-jsx--as-sgml, js-jsx--outermost-enclosing-tag-pos)
(js-jsx--indentation-type, js-jsx--indent-line-in-expression)
(js-jsx--indent-n+1th-line): Remove obsolete functions.

(js-jsx-indent-line): Refactor nearly-obsolete function to behave the
same as it usually would before these changes, without respect to the
binding of js-jsx-syntax.

(js-jsx-mode): Remove obsolete documentation about the use of SGML
variables to control indentation, and don’t bind indent-line-function
any more, because it is no longer necessary given the new
implementation of js-jsx-indent-line.

This completes highlighting support for JSX, as requested in:

- https://github.com/mooz/js2-mode/issues/140
- https://github.com/mooz/js2-mode/issues/330
- https://github.com/mooz/js2-mode/issues/409

* lisp/progmodes/js.el (js--name-start-chars): Extract part of
js--name-start-re so it can be reused in another regexp.
(js--name-start-re): Use js--name-start-chars.

(js-jsx--font-lock-keywords): Use new matchers.
(js-jsx--match-text, js-jsx--match-expr): New matchers to remove
typical JS font-locking and extend the font-locked region,
respectively.

(js-jsx--tag-re, js-jsx--self-closing-re): New regexps matching JSX.
(js-jsx--matched-tag-type, js-jsx--matching-close-tag-pos)
(js-jsx--enclosing-curly-pos, js-jsx--enclosing-tag-pos)
(js-jsx--at-enclosing-tag-child-p): New functions for parsing and
analyzing JSX.

(js-jsx--text-range, js-jsx--syntax-propertize-tag-text): New
functions for propertizing JSXText.
(js-jsx--syntax-propertize-tag): Propertize JSXText children of tags.
(js-jsx--text-properties): Remove JSXText-related text properties when
repropertizing.
(js-mode): Extend the syntax-propertize region with
syntax-propertize-multiline; we are now adding the syntax-multiline
text property to buffer ranges that are JSXText to ensure the whole
multiline JSX construct is reidentified.

* lisp/progmodes/js.el (js-jsx--font-lock-keywords):
Call tag beginning and end matchers.
(js-jsx--match-tag-beg, js-jsx--match-tag-end): New functions.
(js-jsx--syntax-propertize-tag): Record buffer positions of JSXElement
beginning and end for font-locking.

(js--syntax-propertize-extend-region)
(js-jsx--syntax-propertize-extend-region): New functions for extending
the syntax-propertize region backwards to the start of a JSXElement so
its JSXAttribute children on its n+1th lines can be parsed as such
while editing those lines.
(js-mode): Add js--syntax-propertize-extend-region to
syntax-propertize-extend-region-functions.

Font-lock JSX from the beginning of the buffer to the end.  Tends to
break temporarily when editing lines, because the parser doesn’t yet
look backwards to determine if the end of a tag in the current range
starts before the range.

This also re-breaks some tests fixed by previous commits, as we begin
to take a different direction in our parsing code, looking for JSX,
rather than for non-JSX.  The parsing code will eventually provide
information for indentation again.

* lisp/progmodes/js.el (js--dotted-captured-name-re)
(js-jsx--disambiguate-beginning-of-tag)
(js-jsx--disambiguate-end-of-tag, js-jsx--disambiguate-syntax):
Remove.
(js-jsx--font-lock-keywords): New variable.
(js--font-lock-keywords-3): Add JSX matchers.
(js-jsx--match-tag-name, js-jsx--match-attribute-name): New functions.
(js-jsx--syntax-propertize-tag): New function to aid in JSX
font-locking and eventually indentation.
(js-jsx--text-properties): New variable.
(js-syntax-propertize): Propertize JSX properly using
syntax-propertize-rules.

* lisp/progmodes/js.el (js--disambiguate-beginning-of-jsx-tag): Rename
to js-jsx--disambiguate-beginning-of-tag.
(js--disambiguate-end-of-jsx-tag): Rename to
js-jsx--disambiguate-end-of-tag.
(js--disambiguate-js-from-jsx): Rename to js-jsx--disambiguate-syntax.
(js--jsx-start-tag-re): Rename to js-jsx--start-tag-re.
(js--looking-at-jsx-start-tag-p): Rename to
js-jsx--looking-at-start-tag-p.
(js--jsx-end-tag-re): Rename to js-jsx--end-tag-re.
(js--looking-back-at-jsx-end-tag-p): Rename to
js-jsx--looking-back-at-end-tag-p.
(js--as-sgml): Rename to js-jsx--as-sgml.
(js--outermost-enclosing-jsx-tag-pos): Rename to
js-jsx--outermost-enclosing-tag-pos.
(js--jsx-indentation): Rename to js-jsx--indentation-type.
(js--indent-line-in-jsx-expression): Rename to
js-jsx--indent-line-in-expression.
(js--indent-n+1th-jsx-line): Rename to js-jsx--indent-n+1th-line.

Fix some JSX indentation bugs:

- Bug#24896 / https://github.com/mooz/js2-mode/issues/389
- Bug#30225
- https://github.com/mooz/js2-mode/issues/459

* lisp/progmodes/js.el (js--dotted-captured-name-re)
(js--unary-keyword-re, js--unary-keyword-p)
(js--disambiguate-beginning-of-jsx-tag)
(js--disambiguate-end-of-jsx-tag)
(js--disambiguate-js-from-jsx): New variables and functions.

(js-syntax-propertize): Additionally clarify when syntax is JS so that
‘(with-syntax-table sgml-mode-syntax-table …)’ does not mistake some
JS punctuation syntax for SGML parenthesis syntax, namely ‘<’ and ‘>’.

* test/manual/indent/js-jsx-unclosed-2.js: Add additional test for
unary operator parsing.

Fix a number of bugs reported for JSX indentation (caused by poor JSX
detection):

- https://github.com/mooz/js2-mode/issues/140#issuecomment-166250016
- https://github.com/mooz/js2-mode/issues/490
- Bug#24896 / https://github.com/mooz/js2-mode/issues/389 (with
respect to comments)
- Bug#26001 /
https://github.com/mooz/js2-mode/issues/389#issuecomment-271869380
- https://github.com/mooz/js2-mode/issues/411 / Bug#27000 /
https://github.com/mooz/js2-mode/issues/451

Potentially manifest some new bugs (due to false positives with ‘<’
and ‘>’ and SGML detection).  Slow down indentation a fair bit.

* list/progmodes/js.el (js-jsx-syntax, js--jsx-start-tag-re)
(js--looking-at-jsx-start-tag-p, js--looking-back-at-jsx-end-tag-p):
New variables and functions.
(js--jsx-find-before-tag, js--jsx-after-tag-re): Deleted.

(js--looking-at-operator-p): Don’t mistake a JSXOpeningElement for the
‘<’ operator.
(js--continued-expression-p): Don’t mistake a JSXClosingElement as a
fragment of a continued expression including the ‘>’ operator.

(js--as-sgml): Simplify.  Probably needn’t bind forward-sexp-function
to nil (sgml-mode already does) and probably shouldn’t bind
parse-sexp-lookup-properties to nil either (see Bug#24896).

(js--outermost-enclosing-jsx-tag-pos): Find enclosing JSX more
accurately than js--jsx-find-before-tag.  Use sgml-mode’s parsing
logic, rather than unreliable heuristics like paren-wrapping.  This
implementation is much slower; the previous implementation was fast,
but at the expense of accuracy.  To make up for all the grief we’ve
caused users, we will prefer accuracy over speed from now on.  That
said, this can still probably be optimized a lot.

(js--jsx-indented-element-p): Rename to js--jsx-indentation, since it
doesn’t just return a boolean.
(js--jsx-indentation): Refactor js--jsx-indented-element-p to simplify
the implementation as the improved accuracy of other code allows (and
to repent for some awful stylistic choices I made earlier).

(js--expression-in-sgml-indent-line): Rename to
js--indent-line-in-jsx-expression, since it’s a private function and
we can give it a name that reads more like English.
(js--indent-line-in-jsx-expression): Restructure point adjustment
logic more like js-indent-line.

(js--indent-n+1th-jsx-line): New function to complement
js--indent-line-in-jsx-expression.

(js-jsx-indent-line): Refactor.  Don’t bind js--continued-expression-p
to ignore any more; instead, rely on the improved accuracy of
js--continued-expression-p.

(js-jsx-mode): Set js-jsx-syntax to t.  For now, this will be the flag
we use to determine whether ‘JSX is enabled.’  (Maybe later, we will
refactor the code to use this variable instead of requiring
js-jsx-mode to be enabled, thus rendering the mode obsolete.)

* lisp/progmodes/verilog-mode.el (verilog-sk-define-signal):
Rename sig-re to sig-chars, to make it clear that it isn't a regexp.