GitLab

We used to store module runtime and environment pointers in the static
lists Vmodule_runtimes and Vmodule_environments.  However, this is
incorrect because these objects have to be kept per-thread.  With this
naive approach, interleaving module function calls in separate threads
leads to environments being removed in the wrong order, which in turn
can cause local module values to be incorrectly garbage-collected.
The fix isn't completely trivial: specbinding the lists wouldn't work
either, because then the garbage collector wouldn't find the
environments in other threads than the current ones, again leading to
objects being garbage-collected incorrectly.  While introducing custom
pseudovector types would fix this, it's simpler to put the runtime and
environment pointers into the specbinding list as new specbinding
kinds.  This works since we need to unwind them anyway, and we only
ever treat the lists as a stack.  The thread switching machinery
ensures that the specbinding lists are thread-local, and that all
elements of the specbinding lists in all threads are marked during
garbage collection.

Module assertions now have to walk the specbinding list for the
current thread, which is more correct since they now only find
environments for the current thread.  As a result, we can now remove
the faulty Vmodule_runtimes and Vmodule_environments variables
entirely.

Also add a unit test that exemplifies the problem.  It interleaves two
module calls in two threads so that the first call ends while the
second one is still active.  Without this change, this test triggers
an assertion failure.

* src/lisp.h (enum specbind_tag): Add new tags for module runtimes and
environments.

* src/eval.c (record_unwind_protect_module): New function to record a
module object in the specpdl list.
(do_one_unbind): Unwind module objects.
(backtrace_eval_unrewind, default_toplevel_binding, lexbound_p)
(Fbacktrace__locals): Deal with new specbinding types.
(mark_specpdl): Mark module environments as needed.

* src/alloc.c (garbage_collect): Remove call to 'mark-modules'.
Garbage collection of module values is now handled as part of marking
the specpdl of each thread.

* src/emacs-module.c (Fmodule_load, funcall_module): Use specpdl to
record module runtimes and environments.
(module_assert_runtime, module_assert_env, value_to_lisp): Walk
through specpdl list instead of list variables.
(mark_module_environment): Rename from 'mark_modules'.  Don't attempt
to walk though current thread's environments only, since that would
miss other threads.
(initialize_environment, finalize_environment): Don't change
Vmodule_environments variable; environments are now in the specpdl
list.
(finalize_environment_unwind, finalize_runtime_unwind): Make 'extern'
since do_one_unbind now calls them.
(finalize_runtime_unwind): Don't change Vmodule_runtimes variable;
runtimes are now in the specpdl list.
(syms_of_module): Remove Vmodule_runtimes and Vmodule_environments.

* test/data/emacs-module/mod-test.c (Fmod_test_funcall): New test
function.
(emacs_module_init): Bind it.

* test/src/emacs-module-tests.el (emacs-module-tests--variable): New
helper type to guard access to state in a thread-safe way.
(emacs-module-tests--wait-for-variable)
(emacs-module-tests--change-variable): New helper functions.
(emacs-module-tests/interleaved-threads): New unit test.

* test/data/emacs-module/mod-test.c: Move from here...
* test/src/emacs-module-resources/mod-test.c: ...to here.
* test/src/emacs-module-tests.el (ert-x): Require.
(mod-test-file, module/describe-function-1):
* test/Makefile.in (test_module_dir): Adjust for move.

* test/data/files-bug18141.el.gz: Move from here...
* test/lisp/files-resources/files-bug18141.el.gz: ... to here.
* test/lisp/files-tests.el (ert-x): Require.
(files-test-bug-18141-file): Use ert-resource-file.

* test/data/mailcap/mime.types: Move from here...
* test/lisp/net/mailcap-resources/mime.types: ...to here.
* test/lisp/net/mailcap-tests.el (ert-x): Require.
(mailcap-tests-path): Use ert-resource-file.

* test/data/somelib.el:
* test/data/somelib2.el: Move from here...
* test/src/lread-resources/somelib.el:
* test/src/lread-resources/somelib2.el: ...to here.
* test/src/lread-tests.el (ert, ert-x): Require.
(lread-test-bug26837): Use ert-resource-directory.

* test/data/syntax-comments.txt: Move from here....
* test/src/syntax-resources/syntax-comments.txt: ...to here.
* test/src/syntax-tests.el (ert-x): Require.
(syntax-comments, syntax-br-comments, syntax-pps-comments):
Use ert-resource-file.

* test/data/xref/file1.txt:
* test/data/xref/file2.txt: Move from here...
* test/lisp/progmodes/xref-resources/file1.txt:
* test/lisp/progmodes/xref-resources/file2.txt: ...to here.
* test/lisp/progmodes/xref-tests.el (ert, ert-x): Require.
(xref-tests-data-dir): Use ert-resource-directory.

* doc/lispref/internals.texi (Module Values): Document
make_unibyte_string (bug#34873).

* src/emacs-module.c (module_make_unibyte_string): New function.
(initialize_environment): Export it.

* src/module-env-25.h: Define it.

* test/data/emacs-module/mod-test.c (Fmod_test_return_unibyte):
Test it.

* test/src/emacs-module-tests.el (module/unibyte): Test it.

Problem reported by Koki Fukuda in:
https://lists.gnu.org/r/emacs-devel/2020-07/msg00169.html
* test/Makefile.in (MODULE_CFLAGS):
Include from the same directories included from in ../src.
* test/src/emacs-module-tests.el (module/describe-function-1):
Strip path to source directory.

(cherry picked from commit c86f3fe0)

* src/module-env-28.h: Add field for 'make_interactive' function.

* src/emacs-module.c (Lisp_Module_Function): Add new field holding the
interactive form.
(allocate_module_function): Adapt to structure layout change.
(module_make_interactive, module_function_interactive_form): New
functions.
(initialize_environment): Use them.

* src/eval.c (Fcommandp):
* src/data.c (Finteractive_form): Also handle interactive module
functions.

* test/data/emacs-module/mod-test.c (Fmod_test_identity): New test
function.
(emacs_module_init): Create two interactive module test functions.

* test/src/emacs-module-tests.el (module/interactive/return-t)
(module/interactive/return-t-int, module/interactive/identity):
New unit tests.

* doc/lispref/internals.texi (Module Functions): Document new
function.  Rework paragraph about wrapping module functions, as the
example no longer applies.

* etc/NEWS: Document new facility.

* test/lisp/subr-tests.el (subr-test-version-parsing):
* test/lisp/emacs-lisp/gv-tests.el (gv-dont-define-expander-other-file):
* test/src/callint-tests.el (call-interactively/incomplete-multibyte-sequence):
* test/src/emacs-module-tests.el (module/describe-function-1):
Don't fail if curly quotes are in use, as they can be if LC_ALL != C.

This library already used section headings but it used just two
instead of three semicolons, making them indistinguishable from
plain comments.

* test/src/emacs-module-tests.el (mod-test-globref-reordered): New
unit test.

* test/data/emacs-module/mod-test.c (Fmod_test_globref_reordered): New
test module function.
(emacs_module_init): Export it.

This backports commit 9f01ce63 from master.  Since the bug isn’t
present on emacs-27, just backport the new test case.

* test/data/emacs-module/mod-test.c (Fmod_test_globref_invalid_free):
New test module function.
(emacs_module_init): Export it.

* test/src/emacs-module-tests.el
(module--test-assertions--globref-invalid-free): New unit test.

This backports commit 6355a3ec from master.  Since the bug isn’t
present in emacs-27, just backport the test case.

* test/data/emacs-module/mod-test.c
(Fmod_test_invalid_store_copy): New test module function.
(emacs_module_init): Export it.

* test/src/emacs-module-tests.el
(module--test-assertions--load-non-live-object-with-global-copy):
New unit test.

We can't just use a hash lookup because a global and a local reference
might refer to the same Lisp object.

* src/emacs-module.c (module_free_global_ref): More precise check for
global liveness.

* test/data/emacs-module/mod-test.c (Fmod_test_globref_invalid_free):
New test module function.
(emacs_module_init): Export it.

* test/src/emacs-module-tests.el
(module--test-assertions--globref-invalid-free): New unit test.

We can't simply look up the Lisp object in the global reference table
because an invalid local and a valid global reference might refer to
the same object.  Instead, we have to test the address of the global
reference against the stored references.

* src/emacs-module.c (module_global_reference_p): New helper function.
(value_to_lisp): Use it.

* test/data/emacs-module/mod-test.c
(Fmod_test_invalid_store_copy): New test module function.
(emacs_module_init): Export it.

* test/src/emacs-module-tests.el
(module--test-assertions--load-non-live-object-with-global-copy):
New unit test.

Problem reported by Koki Fukuda in:
https://lists.gnu.org/r/emacs-devel/2020-07/msg00169.html
* test/Makefile.in (MODULE_CFLAGS):
Include from the same directories included from in ../src.
* test/src/emacs-module-tests.el (module/describe-function-1):
Strip path to source directory.

These changes let the code compile and produce a valid DLL, but the
test hangs.  It looks like the hang is in Fdelete_process, when it
closes one of the descriptors of the pipe process.
In addition, this use of the pipe process cannot currently work
on MS-Windows, since make-pipe-process doesn't set up the reader
thread to read from the Emacs's side of the pipe, so the select
emulation doesn't know there's stuff to read from that pipe.
* test/data/emacs-module/mod-test.c [WINDOWSNT]: Include
windows.h.
(ALIGN_STACK) [!__x86_64__]: Define for 32-bit builds.
(sleep_for_half_second): New function.
(write_to_pipe): Declare return type differently for WINDOWSNT.
Call sleep_for_half_second.
(Fmod_test_async_pipe) [WINDOWSNT]: Use _beginthread as substitute
for pthread_create.
(invalid_finalizer): Replace non_ASCII character in a comment.

* test/src/emacs-module-tests.el (module/async-pipe): Skip on
MS-Windows, as the test fails and then hangs.

A common complaint about the module API is that modules can't
communicate asynchronously with Emacs.  While it isn't possible to
call arbitrary Emacs functions asynchronously, writing to a pipe
should always be fine and is a pretty low-hanging fruit.

This patch implements a function that adapts an existing pipe
process.  That way, users can use familiar tools like process filters
or 'accept-process-output'.

* src/module-env-28.h: Add 'open_channel' module function.

* src/emacs-module.c (module_open_channel): Provide definition for
'open_channel'.
(initialize_environment): Use it.

* src/process.c (open_channel_for_module): New helper function.
(syms_of_process): Define necessary symbol.

* test/src/emacs-module-tests.el (module/async-pipe): New unit test.

* test/data/emacs-module/mod-test.c (signal_system_error): New helper
function.
(signal_errno): Use it.
(write_to_pipe): New function running in the background.
(Fmod_test_async_pipe): New test module function.
(emacs_module_init): Export it.

* doc/lispref/internals.texi (Module Misc): Document new module
function.

* doc/lispref/processes.texi (Asynchronous Processes): New anchor
for pipe processes.

* etc/NEWS: Document 'open_channel' function.

This is especially useful in cases where modules only use a single
entry point and use the data to dispatch to the actual function.  Such
a design is common for languages such as Go and C++.

* src/emacs-module.c (module_function_data): New function.

* src/print.c (print_vectorlike): Use it to print module function data
if not NULL.
(print_object): Adapt size of buffer.

* test/data/emacs-module/mod-test.c (emacs_module_init): Pass some
non-NULL data to ‘mod-test-sum’.
(Fmod_test_sum): Check that correct data is passed through.

* test/src/emacs-module-tests.el (mod-test-sum-test)
(module-function-object): Adapt unit tests.

* test/src/emacs-module-tests.el (module/function-finalizer): Create
100 leaked functions to increase the probability that at least one
gets garbage-collected.

* src/module-env-28.h: Add new module environment functions to
module environment for Emacs 28.

* src/emacs-module.h.in: Document that 'emacs_finalizer' also works
for function finalizers.

* src/emacs-module.c (CHECK_MODULE_FUNCTION): New function.
(struct Lisp_Module_Function): Add finalizer data member.
(module_make_function): Initialize finalizer.
(module_get_function_finalizer)
(module_set_function_finalizer): New module environment functions.
(module_finalize_function): New function.
(initialize_environment): Initialize new environment functions.

* src/alloc.c (cleanup_vector): Call potential module function
finalizer during garbage collection.

* test/data/emacs-module/mod-test.c (signal_error): New helper
function.
(memory_full): Use it.
(finalizer): New example function finalizer.
(Fmod_test_make_function_with_finalizer)
(Fmod_test_function_finalizer_calls): New test module functions.
(emacs_module_init): Define them.

* test/src/emacs-module-tests.el (module/function-finalizer): New unit
test.

* doc/lispref/internals.texi (Module Functions): Document new
functionality.
(Module Misc): Move description of 'emacs_finalizer' type to 'Module
Functions' node, and add a reference to it.

* etc/NEWS: Mention new functionality.

Run "TZ=UTC0 admin/update-copyright $(git ls-files)".

On macOS, shared libraries typically have the suffix .dylib.  This
commit switches the module suffix to .dylib on Darwin to account for
that.  To also support the .so suffix, introduce the concept of a
secondary module suffix.

* configure.ac: Switch MODULES_SUFFIX to .dylib for Darwin, introduce
MODULES_SECONDARY_SUFFIX.

* src/lread.c (Fload, syms_of_lread): Also use
MODULES_SECONDARY_SUFFIX if defined.

* test/src/emacs-module-tests.el (module-darwin-secondary-suffix): New
unit test.

* test/data/emacs-module/mod-test.c (extract_big_integer): Fix
calculation of size of 'magnitude' when 'emacs_limb_t' is wider
than 'unsigned long'.

* test/src/emacs-module-tests.el
(module--test-assertions--load-non-live-object)
(module--test-assertions--call-emacs-from-gc): On MS-Windows,
check also mod-test-emacs with ".exe" appended, before
skipping the tests.

Without this fix, time arithmetic yielded results that were not
mathematically accurate, even though the exact results were
representable; for example, (time-add 0 1e-13) yielded a timestamp
equal to 0 instead of to 1e-13.
* lisp/timezone.el (timezone-time-from-absolute):
Let time-add do its thing rather than using floating point
internally, which has rounding errors.  We now have bignums and so
don’t need floating point to avoid overflow issues.
* src/timefns.c (timeform_sub_ps_p): New function.
(time_arith): If either argument is a float, represent the
result exactly instead of discarding sub-ps info.
* test/lisp/timezone-tests.el (timezone-tests-time-from-absolute):
Don’t assume (HI LO US PS) timestamp format.
* test/src/emacs-module-tests.el (mod-test-add-nanosecond/valid):
Don’t assume that time-add discards sub-ns info.
* test/src/timefns-tests.el (time-rounding-tests):
Add regression test to detect time-add rounding bug.

* test/src/emacs-module-tests.el (mod-test-nanoseconds):
Don’t assume -1000000000 is a fixnum.

* admin/charsets/mapconv:
* build-aux/gitlog-to-changelog (parse_amend_file, git_dir_option):
* lisp/progmodes/verilog-mode.el (verilog-delay-re):
(verilog-type-font-keywords, verilog-read-always-signals-recurse):
(verilog-is-number):
* lisp/progmodes/vera-mode.el (vera-font-lock-keywords):
* test/src/emacs-module-tests.el (mod-test-sum-test):
* lisp/xml.el: (xml--entity-replacement-text):
* lisp/version.el (emacs-repository-version-git):
* lisp/textmodes/sgml-mode.el (sgml-quote):
* lisp/textmodes/css-mode.el (css-escapes-re)
(css--colors-regexp):
* lisp/progmodes/prolog.el (prolog-syntax-propertize-function):
* lisp/progmodes/hideif.el (hif-token-regexp, hif-tokenize):
* lisp/progmodes/ebnf-dtd.el: (ebnf-dtd-attlistdecl)
(ebnf-dtd-entitydecl, ebnf-dtd-lex):
* lisp/progmodes/ebnf-ebx.el (ebnf-ebx-hex-character):
* lisp/progmodes/ebnf-abn.el (ebnf-abn-character):
* lisp/progmodes/cperl-mode.el (cperl-highlight-charclass)
(cperl-find-pods-heres):
* lisp/progmodes/cc-mode.el (c-maybe-quoted-number-head)
(c-maybe-quoted-number, c-parse-quotes-before-change)
(c-parse-quotes-after-change, c-quoted-number-head-before-point)
(c-quoted-number-straddling-point):
* lisp/progmodes/ada-mode.el (featurep, ada-in-numeric-literal-p)
(ada-font-lock-keywords):
* lisp/org/org-mobile.el (org-mobile-copy-agenda-files)
* lisp/org/org-table.el (org-table-number-regexp):
(org-mobile-update-checksum-for-capture-file):
* lisp/nxml/xsd-regexp.el (xsdre-gen-categories):
* lisp/nxml/xmltok.el (let*):
* lisp/nxml/rng-xsd.el (rng-xsd-convert-hex-binary)
(rng-xsd-convert-any-uri):
* lisp/nxml/rng-uri.el (rng-uri-file-name-1)
(rng-uri-unescape-multibyte, rng-uri-unescape-unibyte)
(rng-uri-unescape-unibyte-match)
(rng-uri-unescape-unibyte-replace):
* lisp/nxml/rng-cmpct.el (rng-c-process-escapes):
* lisp/nxml/nxml-maint.el (nxml-insert-target-repertoire-glyph-set):
* lisp/net/shr-color.el (shr-color->hexadecimal):
* lisp/mail/rfc2231.el (rfc2231-decode-encoded-string):
* lisp/international/mule-cmds.el (read-char-by-name):
* lisp/htmlfontify.el (hfy-hex-regex):
* lisp/gnus/nneething.el (nneething-decode-file-name):
* lisp/gnus/mml-sec.el (mml-secure-find-usable-keys):
* lisp/gnus/gnus-art.el (gnus-button-mid-or-mail-heuristic-alist):
* lisp/faces.el (read-color):
* lisp/epg.el (epg--status-ERRSIG, epg--status-VALIDSIG)
(epg--status-SIG_CREATED, epg--decode-percent-escape)
(epg--decode-hexstring, epg--decode-quotedstring)
(epg-dn-from-string):
* lisp/emulation/cua-rect.el (cua-incr-rectangle):
* lisp/dnd.el (dnd-unescape-uri):
* lisp/cedet/semantic/lex.el (semantic-lex-number-expression):
* lisp/cedet/semantic/java.el (semantic-java-number-regexp):
* lisp/calc/calc-lang.el (pascal):
* lisp/calc/calc-ext.el (math-read-number-fancy):
* lisp/calc/calc-aent.el (math-read-token):
Replace various combinations of [0-9a-fA-F] with [[:xdigit:]].
(Bug#36167)

* test/data/emacs-module/mod-test.c (Fmod_test_sleep_until): Use
extract_time for time argument.
(signal_wrong_type_argument): Remove unused function.

* test/src/emacs-module-tests.el (mod-test-sleep-until): Remove
unnecessary ‘float-time’.

* test/src/emacs-module-tests.el (multiply-string): Remove unused
variable ‘i’.

* test/src/emacs-module-tests.el: Add package name and standard
sections.
(multiply-string): Add a docstring.

* doc/lispref/internals.texi (Module Values): Clarify that the
truncation is towards negative infinity.

* test/data/emacs-module/mod-test.c (Fmod_test_nanoseconds): Add test
function.
(emacs_module_init): Define it.

* test/src/emacs-module-tests.el (mod-test-nanoseconds): New unit test.

make_time is documented to return a (TICKS . HZ) pair, so we can’t use
make_lisp_time.  Introduce a new conversion function instead.

* src/emacs-module.c (module_make_time): Use timespec_to_lisp to
correct return type.

* src/timefns.c (timespec_to_lisp): New function.
(make_lisp_time): Use it.

* test/src/emacs-module-tests.el (mod-test-add-nanosecond/valid):
Check return type.

* src/module-env-27.h: Add new module functions to convert big
integers.

* src/emacs-module.h.in (emacs_mpz): Define if GMP is available.

* src/emacs-module.c (module_extract_big_integer)
(module_make_big_integer): New functions.
(initialize_environment): Use them.

* test/data/emacs-module/mod-test.c (Fmod_test_double): New test
function.
(emacs_module_init): Define it.

* test/src/emacs-module-tests.el (mod-test-double): New unit test.

* doc/lispref/internals.texi (Module Values): Document new functions.

Time values are a fundamental data type, and such conversions are hard
to implement within modules because of the various forms of time
values in Emacs Lisp.  Adding dedicated conversion functions can
significantly simplify module code dealing with times.

This approach uses nanosecond precision.  While Emacs in theory has
support for higher-precision time values, in practice most languages
and standards, such as POSIX, C, Java, and Go, have settled on
nanosecond-precision integers to represent time.

* src/emacs-module.h.in: Add header for struct timespec.

* src/module-env-27.h: Add module functions for time conversion.

* src/emacs-module.c (module_extract_time, module_make_time): New
functions.
(initialize_environment): Use them.

* test/data/emacs-module/mod-test.c (Fmod_test_add_nanosecond): New
test function.
(emacs_module_init): Define it.

* test/src/emacs-module-tests.el (mod-test-add-nanosecond/valid)
(mod-test-add-nanosecond/nil, mod-test-add-nanosecond/invalid): New
unit tests.

* doc/lispref/internals.texi (Module Values): Document time
conversion functions.

This reverts commit 093d3e78,
which reverted ee7ad83f,
which reverted 3eb93c07.

This reverts commit ee7ad83f.

There was no consensus on reverting
3eb93c07, so doing that will have to
wait until the discussion ends.

This reverts commit 3eb93c07.

There was no consensus for that commit, see
https://lists.gnu.org/archive/html/emacs-devel/2016-01/msg00150.html.
Also, reverting this commit should fix Bug#31238.

pending_signals is often set if no quit is pending.  This results in
bugs in module code if the module returns but no quit is actually
pending.

As a better alternative, add a new process_input environment function
for Emacs 27.  That function processes signals (like maybe_quit).

* configure.ac: Add module snippet for Emacs 27.

* src/module-env-27.h: New file.

* src/emacs-module.h.in: Add process_input function to environment
interface.

* src/emacs-module.c (module_should_quit): Use QUITP macro to check
whether the caller should quit.
(module_process_input): New function.
(initialize_environment): Use it.

* src/eval.c: Remove obsolete comment.

* test/data/emacs-module/mod-test.c (signal_wrong_type_argument)
(signal_errno): New helper functions.
(Fmod_test_sleep_until): New test module function.

* test/src/emacs-module-tests.el (mod-test-sleep-until): New unit
test.

* doc/lispref/internals.texi (Module Misc): Document process_input.

Run 'TZ=UTC0 admin/update-copyright $(git ls-files)'.

* src/emacs-module.c (module_free_global_ref): Compare a value to be
freed with all entries of the list.

* test/data/emacs-module/mod-test.c (Fmod_test_globref_free): New
function.
(emacs_module_init): Make it accessible from Lisp.
* test/src/emacs-module-tests.el (mod-test-globref-free-test): New
test which uses it.

* test/src/emacs-module-tests.el (mod-test-sum-test): Update
for bignums.

* doc/lispref/control.texi (Iteration):
* doc/misc/cl.texi (Iteration): Document deprecation of its use.
* doc/lispintro/emacs-lisp-intro.texi (dotimes):
* test/src/emacs-module-tests.el (multiply-string):
* test/lisp/filenotify-tests.el (file-notify-test07-many-events):
Place RESULT field after the form.