external/cde
1
0
Fork 0
mirror of git://git.code.sf.net/p/cdesktopenv/code synced 2025-02-13 03:32:24 +00:00
Code Issues Releases Wiki Activity
cde/NEWS

1709 lines
69 KiB
Text
Raw Normal View History

Expose --histreedit and --histverify options (re: 921bbcae) This adds two long-form shell options that modify history expansion (-H/--histexpand). If --histreedit is on and a history expansion fails, the command line is reloaded into the next prompt's edit buffer, allowing corrections. If --histverify is on, the results of a history expansion are not immediately executed but instead loaded into the next prompt's edit buffer, allowing further changes. SH_HISTREEDIT and SH_HISTVERIFY were already handled all along in slowread() in io.c and via 'reedit' arguments to functions called there, but could not be turned on as they were only ever exposed as shopt options in the removed bash compatibility mode (in theory only, as it failed to compile). I had overlooked them until now. So, since the code is there, let's expose these options through the normal long options interface. They're working fine, and activating them actually makes history expansion tolerable to use. src/cmd/ksh93/data/options.c: - Make these options available as "histreedit" and "histverify". src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/sh.1: - Document the "new" options. src/cmd/ksh93/include/defs.h: - Remove unused SH_HISTAPPEND and SH_HISTORY2 macros which were part of the removed bash compatibility code. Note that ksh does not need a histappend option as it never overwrites the history file (in the bash mode, this shopt option was a no-op).
2022-01-12 14:55:02 +00:00
This documents significant changes in the 1.0 branch of ksh 93u+m.
For full details, see the git log at: https://github.com/ksh93/ksh/tree/1.0
Fix BUG_CASELIT: pattern matching as literal string in 'case' This fixes an undocumented 'case' pattern matching misbehaviour (labelled BUG_CASELIT in modernish) that goes back to the original Bourne shell, but wasn't discovered until 2018. If a pattern doesn't match as a pattern, it's tried again as a literal string. This breaks common validation use cases, such as: n='[0-9]' case $n in ( [0-9] ) echo "$n is a number" ;; esac would output "[0-9] is a number" as the literal string fallback matches the pattern. As this misbehaviour was never documented anywhere (not for Bourne, ksh88, or ksh93), and it was never replicated in other shells (not even in ksh88 clones pdksh and mksh), it is unlikely any scripts rely on it. Of course, a literal string fallback, should it be needed, is trivial to implement correctly without this breakage: case $n in ( [0-9] | "[0-9]") echo "$n is a number or the number pattern" ;; esac src/cmd/ksh93/sh/xec.c: - Remove trim_eq() function responsible for implementing the misbehaviour described above. NEWS: - Added. Document this bugfix. Ref.: - The problem: thread starting at https://www.mail-archive.com/austin-group-l@opengroup.org/msg02127.html - The solution, thanks to George Koehler: comments/commits in https://github.com/att/ast/issues/476 - Modernish BUG_CASELIT bug test & documentation: https://github.com/modernish/modernish/commit/b2024ae3 (cherry picked from commit 8d6c8ce69884767a160c1e20049e77bdd849c248 with some extra edits to NEWS to upate the info for this reboot)
2020-06-11 15:14:31 +00:00
Any uppercase BUG_* names are modernish shell bug IDs.
Fix a crash when 'read -u' is given an invalid fd (#53) File descriptors that are too far out of range will cause the read builtin to crash. The following example will generate two crashes: $ ksh -c 'read -u 2000000' || ksh -c 'read -u-2000000' The fix is to error out when the given file descriptor is out of range. This bugfix is from Tomas Klacko, although it was modified to use 'sh_iovalidfd' and reject numbers greater than 'INT_MAX': https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01912.html The question about 'shp->fdstatus[-1]' only applies to ksh93v- (ksh93u+ doesn't have any references to 'shp->fdstatus[-1]'). src/cmd/ksh93/bltins/read.c: - File descriptors that are out of range should be rejected with an error message (like invalid file descriptors that are in range). The seemingly redundant check for negative numbers is there because out of range negative numbers also cause memory faults despite the later 'fd<0' check. src/cmd/ksh93/tests/io.sh: - Add three tests for attempting 'read -u' on various invalid file descriptor numbers.
2020-07-01 17:14:10 +00:00
Cygwin: workaround for ksh to execute #!-less scripts with itself On Cygwin, ksh does not execute scripts without a #! path in a fork of the ksh process as it does on other systems. Reproducer (run from ksh): $ cat test.sh echo "${BASH_VERSION:-not bash}" echo "${.sh.version}" $ chmod +x test.sh $ ./test.sh 4.4.12(3)-release ./test.sh: line 2: ${.sh.version}: bad substitution The script was executed in bash instead of ksh. After this fix, the output on Cygwin is like ksh on other systems: not bash Version AJM 93u+m/1.1.0-alpha+dev 2022-01-26 This also fixes a number of regression test failures, as quite a few tests create and execute temp scripts without a hashbang path. Analysis: On Cygwin, execve(2) happily executes shell scripts without a #! path with /bin/sh (which is bash --posix). However, ksh relies on execve(2) executing binaries or #! only, as it uses an ENOEXEC failure to decide whether to fork and execute a #!-less shell script with a reinitialized copy of itself via exscript(). src/cmd/ksh93/sh/path.c: path_spawn(): - Look at the magic first two bytes of the file; if it is "MZ" (Mark Zbikowski, originator of the .exe format) or "#!", continue as normal, otherwise simulate an ENOEXEC failure from execve(2) which will cause ksh to fall back on #!-less script execution.
2022-01-26 04:20:56 +00:00
2022-01-26:
- On Cygwin, ksh now executes scripts that do not have a #! path itself,
like it does on other systems, instead of with /bin/sh.
Fix a crashing bug in history expansion Reproducer: $ set -o histexpand $ echo foo !#^:h !#^:& /usr/local/bin/ksh: :&: no previous substitution ksh(80822,0x10bc2a5c0) malloc: *** error for object 0x10a13bae3: pointer being freed was not allocated ksh(80822,0x10bc2a5c0) malloc: *** set a breakpoint in malloc_error_break to debug Abort Analysis: In hist_expand(), the 'cc' variable has two functions: it holds a pointer to a malloc'ed copy of the current line, and is also used as a temporary pointer with functions like strchr(). After that temporary use, it is set to NULL again, because the 'done:' routine checks if it non-NULL to decide whether to free the pointer. But if an error occurs, the function may jump straight to 'done' without first setting cc to NULL if it had been used as a temporary pointer. It then tries to free an unallocated pointer. src/cmd/ksh93/edit/hexpand.c: hist_expand(): - Eliminate this bad practice by using a separate variable for temporary pointer purposes. (I was unable to reproduce the crash in a pty regression test, though it is consistently reproducible in a real interactive session. So I haven't added that test.)
2022-01-24 08:22:39 +00:00
2022-01-24:
- Fixed a crashing bug in history expansion that could occur when using the "&"
modifier to repeat a substitution while there was no previous substitution.
Document history expansion and fix a few loose ends src/cmd/ksh93/sh.1: - Add a new section on history expansion mostly adapted from the "History substitution" section from the tcsh(1) man page. This has the standard BSD license which is already in the COPYRIGHT file. Inapplicable stuff was removed, some new stuff added. src/cmd/ksh93/edit/hexpand.c, src/cmd/ksh93/sh/io.c: - Set '#' as the default history comment character as on bash; no longer disable it by default. - Add the 'a' modifier as a synonym for 'g', as on bash. - Remove pointless static keyword from np variable; since the value from previous calls is never used it can just be local. - Use NV_NOADD flag with nv_open() to avoid pointlessly creating the node if the variable doesn't exist yet. - Fix a bug in history expansion where the 'p' modifier had no effect if the 'histverify' option is on. Reproducer: $ set -H -o histv $ true a b c $ !!:p $ true a b c▁ <= instead of printed, the line is re-edited Expected: $ set -H -o histv $ true a b c $ !!:p true a b c $ ▁ This is fixed by making 'p' remove the HIST_EVENT bit from the returned flag bits in hist_expand(), leaving only the HIST_PRINT flag, then adding special handling for this case to slowread() in io.c (print the line, then instead of executing it, continue and read the next line).
2022-01-25 03:16:53 +00:00
- History expansion is now documented in the ksh(1) manual page.
- In history expansion, the history comment character '#' is now enabled by
default, as it is on bash.
- In history expansion, the 'a' modifier is now a synonym for 'g', as on bash.
- Fixed a bug in history expansion where the 'p' modifier had no effect if
the 'histverify' option is on.
Fix history expansion buffer overflow (#434) History expansion currently crashes under ASan due to a buffer overflow. Reproducer: $ set -H $ !!:s/old/new/ Explanation from <https://github.com/att/ast/issues/1369>: > The problem is the code assumes the buffer allocated for a string > stream is zero initialized. But the SFIO code uses malloc() to > allocate the buffer and does not explicitly initialize it with > memset(). That it works at all, even without ASAN enabled, is > purely accidental. It will fail if that malloc() returns a block > that had been previously allocated, used, and freed. Under ASAN > the buffer is initialized (at least on my system) to a sequence > of 0xBE bytes. So the strdup() happily tries to duplicate a > string that is the size of that buffer and fails when it reads > past the end of the buffer looking for the terminating zero byte. src/cmd/ksh93/edit/hexpand.c: - Backport ksh2020 bugfix that avoids assuming the string stream has been initialized to zeros: https://github.com/att/ast/commit/cf16bcca (minus the incorrect change to the static wm variable).
2022-01-20 22:30:02 +00:00
2022-01-20:
history expansion: add missing bounds check So far all ksh versions accept event numbers referring to nonexistent history events in history expansion (-H/-o histexpand), e.g. !9999 is accepted even if the history file has no item 9999. These expansions seem to show random content from the history file, sometimes including binary data. Of course an "event not found" error should have been thrown instead. hist_expand() in hexpand.c calls hist_seek() (from history.c) without any bounds checking except verifying the history event number is greater than zero. This commit adds a bounds check to hist_seek() itself as it's called from three other places in history.c, so perhaps this fixes a few other bugs as well. src/cmd/ksh93/edit/history.c: hist_seek(): - Use the hist_min() and hist_max() macros provided in history.h to check bounds. Note that hist_max() yields the number of the command line currently being entered, so the maximum for seeking purposes is actually its result minus 1.
2022-01-21 01:18:34 +00:00
- Disallow out-of-range event numbers in history expansion (-H/-o histexpand).
Previously these were accepted, resulting in corrupted expansions.
Fix history expansion buffer overflow (#434) History expansion currently crashes under ASan due to a buffer overflow. Reproducer: $ set -H $ !!:s/old/new/ Explanation from <https://github.com/att/ast/issues/1369>: > The problem is the code assumes the buffer allocated for a string > stream is zero initialized. But the SFIO code uses malloc() to > allocate the buffer and does not explicitly initialize it with > memset(). That it works at all, even without ASAN enabled, is > purely accidental. It will fail if that malloc() returns a block > that had been previously allocated, used, and freed. Under ASAN > the buffer is initialized (at least on my system) to a sequence > of 0xBE bytes. So the strdup() happily tries to duplicate a > string that is the size of that buffer and fails when it reads > past the end of the buffer looking for the terminating zero byte. src/cmd/ksh93/edit/hexpand.c: - Backport ksh2020 bugfix that avoids assuming the string stream has been initialized to zeros: https://github.com/att/ast/commit/cf16bcca (minus the incorrect change to the static wm variable).
2022-01-20 22:30:02 +00:00
- Fixed a potential crash in history expansion due to a buffer overflow.
Expose --histreedit and --histverify options (re: 921bbcae) This adds two long-form shell options that modify history expansion (-H/--histexpand). If --histreedit is on and a history expansion fails, the command line is reloaded into the next prompt's edit buffer, allowing corrections. If --histverify is on, the results of a history expansion are not immediately executed but instead loaded into the next prompt's edit buffer, allowing further changes. SH_HISTREEDIT and SH_HISTVERIFY were already handled all along in slowread() in io.c and via 'reedit' arguments to functions called there, but could not be turned on as they were only ever exposed as shopt options in the removed bash compatibility mode (in theory only, as it failed to compile). I had overlooked them until now. So, since the code is there, let's expose these options through the normal long options interface. They're working fine, and activating them actually makes history expansion tolerable to use. src/cmd/ksh93/data/options.c: - Make these options available as "histreedit" and "histverify". src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/sh.1: - Document the "new" options. src/cmd/ksh93/include/defs.h: - Remove unused SH_HISTAPPEND and SH_HISTORY2 macros which were part of the removed bash compatibility code. Note that ksh does not need a histappend option as it never overwrites the history file (in the bash mode, this shopt option was a no-op).
2022-01-12 14:55:02 +00:00
2022-01-12:
- Added bash-inspired --histreedit and --histverify options that modify history
expansion (--histexpand), allowing editing on failure or before execution.
Fix vi mode tab completion with spaces (#413) Attempting to complete file names in vi mode using tab completion can fail if the last character on the command line is a space. Reproducer (note that this bug doesn't occur in emacs mode): $ set -o vi $ mkdir '/tmp/foo bar' $ test -d /tmp/foo\ <Tab> src/cmd/ksh93/edit/vi.c: - Don't disable tab completion or reset the tab count just because the last character on the command line is a space. This bugfix was backported from ksh93v- 2014-06-06. src/cmd/ksh93/tests/pty.sh: - Add a regression test for the tab completion bug.
2022-01-07 15:44:05 +00:00
2022-01-04:
- Fixed an issue in vi mode that caused tab completion to fail if the
last character on the command line was a space.
Allow ksh to compile on Haiku; implement SIGKILLTHR support (#408) This commit implements the build fixes required to get ksh running on Haiku. Note that while ksh does compile, it has a ton of regression test failures on Haiku. src/cmd/ksh93/data/signals.c, src/lib/libast/features/signal.c: - Add support for the SIGKILLTHR signal, which is supported by BeOS and Haiku. - SIGINFO was missing an entry in the libast feature test, so add one (re: 658bba74). src/cmd/ksh93/RELEASE: - Add an entry noting that ksh now compiles on Haiku, albeit with many regression test failures. src/cmd/ksh93/{include/terminal.h,sh/path.c}: - Silence compiler warnings on Haiku. src/lib/libast/features/mmap: - The mmap feature test freezes on Haiku, so modify the test to fail immediately on that OS. src/lib/libast/misc/signal.c: - Avoid redefining the signal definition on Haiku to fix a compiler error. src/lib/libast/features/nl_types: - For some reason the nl_item typedef on Haiku doesn't work correctly. Work around that by creating the nl_item type in the libast nl_types feature test.
2022-01-06 20:35:44 +00:00
2021-12-29:
- Fixed numerous build errors that prevented ksh from running on Haiku OS.
- Added support for the SIGKILLTHR signal on operating systems that
implement it.
Add three options to 'ulimit' (#406) This patch adds a few extra options to the ulimit command (if the OS supports them). These options are also present in Bash, although in ksh additional long forms of each option are available: ulimit -k/--kqueues This is the maximum number of kqueues. ulimit -P/--npts This is the maximum number of pseudo-terminals. ulimit -R/--rttime This is the time a real-time process can run before blocking, in microseconds. When the limit is exceeded, the process is sent SIGXCPU. Other changes: - bltins/ulimit.c: Change the formatting from sfprintf and increase the size of the tmp buffer to prevent text from being cut off in ulimit -a (this was required to add ulimit -R). - data/limits.c: Add support for using microseconds as a unit.
2021-12-28 21:55:56 +00:00
2021-12-28:
Actually deactivate CDPATH when unsetting it After 'unset CDPATH', CDPATH continued to work as if nothing happened. Unsetting it should be a valid way to deactivate it. This bug is in every ksh93 version. src/cmd/ksh93/bltins/cd_pwd.c: b_cd(): - Fix a manifest logic error: first check if CDPATH (CDPNOD) is unset before assigning to 'cdpath', not the other way around. Setting the 'cdpath' pointer is what activates the CDPATH search.
2021-12-28 23:53:08 +00:00
- Fixed a bug that caused CDPATH to continue working after unsetting it.
Add three options to 'ulimit' (#406) This patch adds a few extra options to the ulimit command (if the OS supports them). These options are also present in Bash, although in ksh additional long forms of each option are available: ulimit -k/--kqueues This is the maximum number of kqueues. ulimit -P/--npts This is the maximum number of pseudo-terminals. ulimit -R/--rttime This is the time a real-time process can run before blocking, in microseconds. When the limit is exceeded, the process is sent SIGXCPU. Other changes: - bltins/ulimit.c: Change the formatting from sfprintf and increase the size of the tmp buffer to prevent text from being cut off in ulimit -a (this was required to add ulimit -R). - data/limits.c: Add support for using microseconds as a unit.
2021-12-28 21:55:56 +00:00
- Added three options to the ulimit builtin with the same names and
functionality as in Bash:
- 'ulimit -k' sets the maximum number of kqueues.
- 'ulimit -P' sets the maximum number of pseudo-terminals.
- 'ulimit -R' sets the maximum time in microseconds a real-time process
can run before blocking.
Note that to use these options the operating system must support the
corresponding resource limit.
Port cksum builtin performance improvements from illumos (#391) This commit ports performance optimizations from illumos for the libsum code (used by the cksum and sum builtins): https://github.com/illumos/ast/commit/98bea71f0d345ee7a661e1d9873e8d9b0cd78d37 The new codepath in libsum uses prefetching and loop unrolling to improve performance (prefetching is done with __builtin_prefetch() or sun_prefetch_read_many() if either is available). Script for testing (note that cksum must be enabled in src/cmd/ksh93/data/builtins.c): #!/bin/ksh builtin cksum || exit 1 for ((i=0; i!=50000; i++)) do cksum -x att /etc/hosts done >/dev/null Results on Linux x86_64 (using CCFLAGS=-O2): $ echo 'UNPATCHED:'; time arch/linux.i386-64/bin/ksh /tmp/foo; echo 'PATCHED'; time /tmp/ksh /tmp/foo UNPATCHED: real 0m09.989s user 0m07.582s sys 0m02.406s PATCHED: real 0m06.536s user 0m04.331s sys 0m02.204s src/lib/libsum/{sum-att.c,sum-crc.c,Mamfile}: - Port the performance optimizations from illumos to 93u+m libsum. To prevent problems with older versions of GCC, avoid the new codepath if GCC is older than the 3.1 release series. Additionally, the ast.h header must be included to handle tcc defining __GNUC__ on FreeBSD. - Apply some build fixes to allow the new codepath to build with Clang 3.6 and newer (my own testing indicates an even better performance improvement with Clang than with GCC).
2021-12-28 22:22:38 +00:00
- Ported performance optimizations from illumos to improve the performance
of the cksum builtin. (Note that the cksum builtin is not enabled by
default.)
Fix crash/freeze upon interrupting command substitution with pipe On some systems (at least Linux and macOS): 1. Run on a command line: t=$(sleep 10|while :; do :; done) 2. Press Ctrl+C in the first 10 seconds. 3. Execute any other command substitution. The shell crashes. Analysis: Something in the job_wait() call in the sh_subshell() restore routine may be interrupted by a signal such as SIGINT on Linux and macOS. Exactly what that interruptible thing is remains to be determined. In any case, since job_wait() was invoked after sh_popcontext(), interrupting it caused the sh_subshell() restore routine to be aborted, resulting in an inconsistent state of the shell. The fix is to sh_popcontext() at a later stage instead. src/cmd/ksh93/sh/subshell.c: sh_subshell(): - Rename struct checkpt buff to checkpoint because it's clearer. - Move the sh_popcontext() call to near the end, just after decreasing the subshell level counters and restoring the global subshell data struct to its parent. This seems like a logical place for it and could allow other things to be interrupted, too. - Get rid of the if(shp->subshell) because it is known that the value is > 0 at this point. - The short exit routine run if the subshell forked now needs a new sh_popcontext() call, because this is handled before restoring the virtual subshell state. - While we're here, do a little more detransitioning from all those pointless shp pointers. Fixes: https://github.com/ksh93/ksh/issues/397
2021-12-27 03:00:15 +00:00
2021-12-27:
Backport -P and -t flags for 'type'/'whence' from ksh93v- (#392) This commit backports the whence '-t' option from ksh93v-. The '-t' option is useful when one needs to identify the type of a command. The '-t' flag was added by ksh93v- for compatibility with Bash. It should be noted the ksh93v- patch had one bug, which this commit fixes. Path-bound builtins from /opt/ast/bin were classified as files if loaded from /opt/ast/bin in the PATH. Reproducer: $ PATH=/opt/ast/bin whence -t cat file src/cmd/ksh93/bltins/whence.c: - Simplify the bitmask values for the command and whence builtin flags. - Add the -t flag to the whence and type builtins. To prevent bugs, -t will always override -v if both of those flags were passed. src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/sh.1: - Add documentation for the new -t option.
2021-12-27 06:37:01 +00:00
- Two bash-like flags for 'whence' were backported from ksh 93v-:
- 'whence -P/type -P' is an alias to the existing -p flag.
- 'whence -t/type -t' will print only the type of a command in a simple
format that is designed to be easy to use for scripts. Example:
$ type -t typeset; whence -t sh
builtin
file
Fix crash/freeze upon interrupting command substitution with pipe On some systems (at least Linux and macOS): 1. Run on a command line: t=$(sleep 10|while :; do :; done) 2. Press Ctrl+C in the first 10 seconds. 3. Execute any other command substitution. The shell crashes. Analysis: Something in the job_wait() call in the sh_subshell() restore routine may be interrupted by a signal such as SIGINT on Linux and macOS. Exactly what that interruptible thing is remains to be determined. In any case, since job_wait() was invoked after sh_popcontext(), interrupting it caused the sh_subshell() restore routine to be aborted, resulting in an inconsistent state of the shell. The fix is to sh_popcontext() at a later stage instead. src/cmd/ksh93/sh/subshell.c: sh_subshell(): - Rename struct checkpt buff to checkpoint because it's clearer. - Move the sh_popcontext() call to near the end, just after decreasing the subshell level counters and restoring the global subshell data struct to its parent. This seems like a logical place for it and could allow other things to be interrupted, too. - Get rid of the if(shp->subshell) because it is known that the value is > 0 at this point. - The short exit routine run if the subshell forked now needs a new sh_popcontext() call, because this is handled before restoring the virtual subshell state. - While we're here, do a little more detransitioning from all those pointless shp pointers. Fixes: https://github.com/ksh93/ksh/issues/397
2021-12-27 03:00:15 +00:00
- Fixed a crash or freeze that would occur on Linux when using Ctrl+C to
interrupt a command substitution containing a pipe in an interactive shell.
Fix crash in xtrace while processing here-document (re: d7cada7b) Depending on the OS, the heredoc.sh regression tests, and possibly others, still crashed with the -x option (xtrace) on. Analysis: The lexer crashes in lex_advance(). Something has caused an inconsistent lexer state, and it happened earlier on, so the backtrace is useless for figuring out where that happened. But I think I've found it. It's the sh_mactry() call here: src/cmd/ksh93/sh/xec.c, lines 2800 to 2807 in f7213f03 2800: if(!(cp=nv_getval(sh_scoped(shp,PS4NOD)))) 2801: cp = "+ "; 2802: else 2803: { 2804: sh_offoption(SH_XTRACE); 2805: cp = sh_mactry(shp,cp); 2806: sh_onoption(SH_XTRACE); 2807: } sh_mactry() needs to parse the contents of $PS4 to perform expansions and command substitutions in it, which involves the lexer. If that happens in a here-document, the lexer is in the C function call stack, in the middle of parsing the here-document. Result: inconsistent lexer state. Solution: save and restore lexer state in sh_mactry(). After this commit, all regression tests should pass with the '-x'/'--xtrace' option in use, with no errors or crashes. Note for backporters: this fix depends both on on d7cada7b and on the consistency fix for the Lex_t type's size applied in a7ed5d9f. src/cmd/ksh93/include/shlex.h: - Cosmetic fix: remove a copied & pasted backslash. (re: a7ed5d9f) src/cmd/ksh93/sh/macro.c: sh_mactry(): - Save and restore the lexer state before letting sh_mactrim() indirectly parse and execute code. src/cmd/ksh93/tests/*.sh: - Turn off xtrace in various command substitutions that contain 2>&1 redirections, so that the xtrace output is not caught by the command substitutions, causing tests to fail incorrectly. - Turn off xtrace for a few code blocks with 2>&1 redirections, stopping xtrace output from being written to standard output. Resolves: https://github.com/ksh93/ksh/issues/306 (again)
2021-12-27 03:39:32 +00:00
- Fixed a crash that could occur while processing a here-document while
xtrace (set -x) is on and the $PS4 prompt contains parameter expansions or
command substitutions.
Fix `mkservice` compile errors and add `SHOPT_MKSERVICE` (#401) The unused mkservice and eloop builtins are currently not built, and if an attempt to compile them is made the build ends in failure. This commit backports a few build fixes from ksh93v- 2012-08-24 that allow mkservice and eloop to build (plus an additional compiler warning fix not in ksh93v-). I've also added a new SHOPT_MKSERVICE setting (turned off by default) so that mkservice and eloop can be built if the user chooses to include them in their build of ksh.
2021-12-28 16:53:30 +00:00
- The mkservice and eloop builtins can now be built by enabling the
new SHOPT_MKSERVICE setting in src/cmd/ksh93/SHOPT.sh.
Fix multiple bugs when using 'alias -p' to print aliases (#398) This commit was originally intended to fix just one bug with shcomp's handling of 'alias -p', but while fixing that I found a large number of related issues in the alias command's -p, -t and -x options. The current patch provides bugfixes for all of the bugs listed below: 1) Listing aliases in a script with 'alias -p' or 'alias' broke shcomp's bytecode output: https://github.com/ksh93/ksh/issues/87#issuecomment-813819122 2) Listing individual aliases with the -p option doesn't work: $ alias foo=bar bar=foo $ alias foo foo=bar $ alias -p foo # No output 3) Listing specific tracked aliases with -pt does not display them in a reusable format, but rather adds another tracked alias: $ hash -r cat vi $ alias -pt vi # No output $ alias -pt rm $ alias -t cat=/usr/bin/cat rm=/usr/bin/rm vi=/usr/bin/vi 4) Listing all tracked aliases with -pt does not output them in a reusable format (the resulting command printed only creates a normal alias, which is different from a tracked alias): $ hash -r cat $ alias -pt alias cat=/usr/bin/cat # Expected 'alias -t cat' 5) Listing a non-existent alias with -p doesn't cause an error: $ unalias -a $ alias -p notanalias # No output $ echo $? 0 $ alias notanalias notanalias: alias not found $ echo $? 1 $ hash -r $ alias -pt notacommand # No output $ echo $? 0 6) Attempting to list 256 non-existent aliases results in exit status zero: $ unalias -a $ alias $(awk -v ORS= 'BEGIN { for(i=0;i<256;i++) print "x "; }') x: alias not found --cut error message-- $ echo $? 0 Changes: - typeset.c: Avoid printing anything while shcomp is compiling a script. This is needed because the alias command is run by shcomp to prevent parsing issues. - b_alias(): To avoid adding tracked aliases with -pt, set tdata.aflag to '+' so that setall() and other related functions only list tracked aliases. - b_alias(): Set tdata.pflag to 1 so that setall() and other functions recognize -p was passed. - print_value(): Add support for listing specific aliases with 'alias -p'. - setall(): To avoid any issues with zombie tracked aliases (see also the regression tests) ignore tracked alias nodes marked with the NV_NOALIAS attribute. This bit is set for tracked alias nodes by the nv_rehash() function. - setall(): For backward compatibility, continue incrementing the exit status for each invalid alias and tracked alias passed. This was already how alias behaved when listing aliases without -p, so using -p shouldn't cause a change in behavior: $ unalias -a $ alias foo bar foo: alias not found bar: alias not found $ echo $? 2 To fix bug 6, the exit status is set to one if an enforced 8-bit exit status would be zero. - print_namval(): Set the prefix to 'alias -t' so that listing tracked aliases with 'alias -pt' works correctly. - data/msg.c and include/name.h: Add an error message for when 'alias -pt' doesn't find a tracked alias. - tests/alias.sh: Add a ton of regression tests for the bugs fixed in this commit.
2021-12-27 00:25:05 +00:00
2021-12-26:
- Listing aliases or tracked aliases in a script no longer corrupts
shcomp's generated bytecode.
- Listing specific aliases with 'alias -p' and specific tracked aliases
with 'alias -pt' now works as documented. This means that the following
string of commands now works as you would expect:
$ hash -r; unalias -a
$ alias foo=bar; hash cat
$ alias -p foo; alias -pt cat
alias foo=bar
alias -t cat
- As a result of the above fix, listing all tracked aliases with 'alias -pt'
now prints commands that can be reused to recreate the tracked aliases.
- Attempting to list a non-existent alias or tracked alias with the -p option
now causes an error and sets the exit status to the number of non-existent
aliases passed.
- Attempting to list 256 non-existent aliases now errors out with the exit
status set to one.
Fix process substitutions printing PIDs in profile scripts (#395) - sh/args.c: A process substitution run in a profile script may print its PID as if it was a command spawned with '&'. Reproducer: $ cat /tmp/env true >(false) $ ENV=/tmp/env ksh [1] 730227 $ This bug is fixed by turning off the SH_PROFILE state while running a process substitution. - sh/subshell.c: The SH_INTERACTIVE fix in 3525535e renders the extra check for SH_PROFILE redundant, so it has been removed. - tests/io.sh: Update the procsub PIDs test to also check the result after using process substitution in a profile script.
2021-12-22 13:25:26 +00:00
2021-12-22:
- Process substitutions run in a profile script no longer print their
process ID when run.
sh_parse(): don't turn on interactive state (re: 48ba6964) Reproducer: $ (sleep 1& echo done) done $ (eval "echo hi"; sleep 1& echo done) hi [1] 30587 done No job control output should be printed for a background process invoked from a subshell, not even after 'eval'. The cause: sh_parse() turns on the shell's interactive state bit (sh_state(SH_INTERACTIVE)) if the interactive shell option is on. This is incorrect. The parser should have no involvement with shell interactivity in principle because that's not its domain. Not only that, the parser may need to run in a subshell, e.g. when executing traps or 'eval' commands (as above). By definition, a subshell can never be interactive. We already fixed many bugs related to job control and the shell's interactive state. Even if these two lines previously papered over some breakage, I can't find any now after simply removing them. If any is found later, then it'll need to be fixed properly instead. Related: https://github.com/ksh93/ksh/issues/390
2021-12-22 03:10:11 +00:00
2021-12-21:
- Fixed a bug that caused subshells (such as code blocks in parentheses) to
partially behave like the interactive main shell after running anything
that invokes the parser. For example:
$ (eval :; sleep 1 & echo done)
[1] 30909 <--- incorrect job control output from subshell
done
Interactive: Avoid losing the job after suspending a subshell Reproducer: run vi in a subshell: $ (vi) vi opens; now press Ctrl+Z to suspend. The output is as expected: [2] + Stopped (vi) …but the exit status is 18 (SIGTSTP's signal number) instead of 0. Now do: $ fg (vi) $ The exit status is 18 again, vi is not resumed, and the job is lost. You have to find vi's pid manually using ps and kill it. Forking all non-command substitution subshells invoked from the interactive main shell is the only reliable and effective fix I've found. I've tried to fork the subshell conditionally in every other remotely plausible place I can think of in fault.c and xec.c, but I can't get anything to work properly. If anyone can get this to work without forking as much (or at all), please do submit a patch or PR that supersedes this fix. At least subshells of subshells don't need to fork, so the performance impact can be limited. Plus, it's not as if most people need maximum speed on the interactive command line. Scripts (including login/profile scripts) are not affected at all. Command substitutions can be handled differently. My testing shows that all shells except ksh93 simply block SIGTSTP (the ^Z signal) while they run. We should do the same, so they don't need to fork. NOTE for any backporters: the subshell.c and fault.c changes depend on commits 35b02626 and 48ba6964 to work correctly. src/cmd/ksh93/sh/subshell.c: sh_subshell(): - If the interactive shell state bit is on, then before executing the subshell's code: - for command substitutions, block SIGTSTP; - for other subshells, fork. - For command substitutions, release SIGTSTP if the interactive shell state bit was on upon invoking the subshell. src/cmd/ksh93/sh/fault.c: - Instead of checking for a virtual subshell, check the shell's interactive state bit to decide whether to handle SIGTSTP, as that is only turned on in the interactive main shell. src/cmd/ksh93/sh/main.c: sh_main(): - To avoid bugs, ignore SIGTSTP while running profile scripts. Blocking it doesn't work because delaying it until after sigrelease() will cause a crash. Thanks to @JohnoKing for this. - While we're here, prevent a possible overflow of the 'beenhere' static char variable by only incrementing it once. Co-authored-by: Johnothan King <johnothanking@protonmail.com> Resolves: https://github.com/ksh93/ksh/issues/390
2021-12-22 04:43:42 +00:00
- Fixed: after suspending (Ctrl+Z) a subshell that is running an external
command, resuming the subshell with 'fg' failed and the job was lost.
$ (vi) <--- press Ctrl+Z
[2] + Stopped (vi)
$ fg
(vi) <--- vi failed to resume; immediate return to command line
$ fg
ksh: no such job
Re-fix defining types conditionally or in subshells (re: f508660d) New version. I'm pretty sure the problems that forced me to revert it earlier are fixed. This commit mitigates the effects of the hack explained in the referenced commit so that dummy built-in command nodes added by the parser for declaration/assignment purposes do not leak out into the execution level, except in a relatively harmless corner case. Something like if false; then typeset -T Foo_t=(integer -i bar) fi will no longer leave a broken dummy Foo_t declaration command. The same applies to declaration commands created with enum. The corner case remaining is: $ ksh -c 'false && enum E_t=(a b c); E_t -a x=(b b a c)' ksh: E_t: not found Since the 'enum' command is not executed, this should have thrown a syntax error on the 'E_t -a' declaration: ksh: syntax error at line 1: `(' unexpected This is because the -c script is parsed entirely before being executed, so E_t is recognised as a declaration built-in at parse time. However, the 'not found' error shows that it was successfully eliminated at execution time, so the inconsistent state will no longer persist. This fix now allows another fix to be effective as well: since built-ins do not know about virtual subshells, fork a virtual subshell into a real subshell before adding any built-ins. src/cmd/ksh93/sh/parse.c: - Add a pair of functions, dcl_hactivate() and dcl_dehacktivate(), that (de)activate an internal declaration built-ins tree into which check_typedef() can pre-add dummy type declaration command nodes. A viewpath from the main built-ins tree to this internal tree is added, unifying the two for search purposes and causing new nodes to be added to the internal tree. When parsing is done, we close that viewpath. This hides those pre-added nodes at execution time. Since the parser is sometimes called recursively (e.g. for command substitutions), keep track of this and only activate and deactivate at the first level. (Fixed compared to previous version of this commit: calling dcl_dehacktivate() when the recursion level is already zero is now a harmless no-op. Since this only occurs in error handling conditions, who cares.) - We also need to catch errors. This is done by setting libast's error_info.exit variable to a dcl_exit() function that tidies up and then passes control to the original (usually sh_exit()). (Fixed compared to previous version of this commit: dcl_exit() immediately deactivates the hack, no matter the recursion level, and restores the regular sh_exit(). This is the right thing to do when we're in the process of erroring out.) - sh_cmd(): This is the most central function in the parser. You'd think it was sh_parse(), but $(modern)-form command substitutions use sh_dolparen() instead. Both call sh_cmd(). So let's simply add a dcl_hacktivate() call at the beginning and a dcl_deactivate() call at the end. - assign(): This function calls path_search(), which among many other things executes an FPATH search, which may execute arbitrary code at parse time (!!!). So, regardless of recursion level, forcibly dehacktivate() to avoid those ugly parser side effects returning in that context. src/cmd/ksh93/bltins/enum.c: b_enum(): - Fork a virtual subshell before adding a built-in. src/cmd/ksh93/sh/xec.c: sh_exec(): - Fork a virtual subshell when detecting typeset's -T option. Improves fix to https://github.com/ksh93/ksh/issues/256
2021-12-17 00:02:01 +00:00
2021-12-17:
Release 1.0.0-beta.2 Announcing: KornShell 93u+m 1.0.0-beta.2 https://github.com/ksh93/ksh In May 2020, when every KornShell (ksh93) development project was abandoned, development was rebooted in a new fork based on the last stable AT&T version: ksh 93u+. This new fork is called ksh 93u+m as a permanent nod to its origin. We're restarting it at version 1.0. Seven months after the first beta, the second one is ready. Please test this second beta and report any bugs you find, or help us fix known bugs. We're now the default ksh93 in some OS distributions, at least Debian and Slackware! Even though we don't think it's stable release quality yet, the consensus seems to be that 93u+m is already much better than the last AT&T release. Main developers: Martijn Dekker, Johnothan King, hyenias Contributors: Andy Fiddaman, Anuradha Weeraman, Chase, Gordon Woodhull, Govind Kamat, Harald van Dijk, Lev Kujawski, Marc Wilson, Ryan Schmidt, Sterling Jensen HOW TO GET IT Please download the source code tarball from our GitHub releases page: https://github.com/ksh93/ksh/releases To build, follow the instructions in README.md or src/cmd/ksh93/README. HOW TO GET INVOLVED To report a bug, please open an issue at our GitHub page (see above). Alternatively, email me at martijn@inlv.org with your report. To get involved in development, read the brief policy information in README.md and then jump right in with a pull request or email a patch. See the TODO file in the top-level directory for a to-do list. *** MAIN CHANGES between 1.0.0-beta.1 and 1.0.0-beta.2 *** New features in built-in commands: - 'cd' now supports an -e option that, when combined with -P, verifies that $PWD is correct after changing directories; this helps detect access permission problems. See: https://www.austingroupbugs.net/view.php?id=253 - 'printf' now supports a -v option as in bash. This assigns formatted output directly to variables, which is very fast and will not strip final newline (\n) characters. - The 'return' command, when used to return from a function, can now return any status value in the 32-bit signed integer range, like on zsh. However, due to a traditional Unix kernel limitation, $? is still trimmed to its least significant 8 bits whenever leaving a (sub)shell environment. - 'test'/'[' now supports all the same operators as [[ (including =~, \<, \>) except for the different 'and'/'or' operators. Note that 'test'/'[' remains deprecated due to its unfixable pitfalls; [[ ... ]] is recommended instead. Shell language changes: - Several improvements were made to the --noexec shell code linter. - Arithmetic expressions in native ksh mode no longer interpret a number with a leading zero as octal in any context. Use 8#octalnumber instead (e.g. 8#400 == 256). Arithmetic expressions now also behave identically within and outside ((...)) and $((...)). - POSIX compatibility mode fixes (only applicable with the --posix shell option on): - A leading zero is now consistently recognised as introducing an octal number in all arithmetic contexts. - $((inf)) and $((nan)) are now interpreted as regular variables. - The '.' built-in no longer runs ksh functions and now only runs files. Bugs fixed: - '.' and '..' are now once again completed by tab completion. - If SIGINT is set to ignore, the interactive shell no longer exits on Ctrl+C. - ksh now builds and runs on Apple's new M1 hardware. - The 'return' and 'exit' commands no longer risk triggering actual signals by returning or exiting with a status > 256. - Ksh no longer behaves badly when parsing a type definition command ('typeset -T' or 'enum') without executing it or when executing it in a subshell. Types can now safely be defined in subshells and defined conditionally as in 'if condition; then enum ...; fi'. - Discipline functions, especially those applied to PS2 or .sh.tilde, will no longer crash your shell upon being interrupted or throwing an error. - Fixed a bug that could corrupt output if standard output is closed upon initialising the shell. - Fixed a bug in the [[ ... ]] compound command: the '!' logical negation operator now correctly negates another '!', e.g., [[ ! ! 1 -eq 1 ]] now returns 0/true. Note that this has always been the case for 'test'/'['. - Fixed SHLVL so that replacing ksh by itself (exec ksh) will not increase it. - Arithmetic expressions are no longer allowed to assign out-of-range values to variables of types declared with enum. - The 'time' keyword no longer makes the --errexit shell option ineffective. - Various bugs in libcmd built-in commands (those bound to the /opt/ast/bin path by default) have been fixed. - Various other crashing bugs have been fixed. Fixes for the shcomp byte code compiler: - shcomp is now able to compile scripts that define types using enum. - shcomp now refuses to mess up your terminal by writing bytecode to it. *** MAIN CHANGES between ksh 93u+ 2012-08-01 and 93u+m 1.0.0-beta.1 *** Hundreds of bugs have been fixed, including many serious/critical bugs. This includes upstreamed patches from OpenSUSE, Red Hat, and Solaris, fixes backported from the abandoned 93v- beta and ksh2020 fork, as well as many new fixes from the community. See the NEWS file for more information, and the git commit log for complete documentation of every fix. Incompatible changes have been minimised, but not at the expense of fixing bugs. For a list of potentially incompatible changes, see src/cmd/ksh93/COMPATIBILITY. Though there was a "no new features, bugfixes only" policy, some new features were found necessary, either to fix serious design flaws or to complete functionality that was evidently intended, but not finished. Below is a summary of these new features. New command line editor features: - The forward-delete and End keys are now handled as expected in the emacs and vi built-in line editors. - In the vi and emacs line editors, repeat count parameters can now also be used for the arrow keys and the forward-delete key. E.g., in emacs mode, <ESC> 7 <left-arrow> will now move the cursor seven positions to the left. In vi control mode, this would be entered as: 7 <left-arrow>. New shell language features: - The &>file redirection shorthand (for >file 2>&1) is now available for all scripts and interactive sessions and not only for profile/login scripts, bringing ksh 93u+m in line with mksh, bash, and zsh. - File name generation (a.k.a. pathname expansion, a.k.a. globbing) now never matches the special navigational names '.' (current directory) and '..' (parent directory). This change makes a pattern like .* useful; it now matches all hidden files (dotfiles) in the current directory, without the harmful inclusion of '.' and '..'. - Tilde expansion can now be extended or modified by defining a .sh.tilde.get or .sh.tilde.set discipline function. This replaces a 2004 undocumented attempt to add this functionality via a .sh.tilde command, which never worked and crashed the shell. See the manual for details on the new method. New features in built-in commands: - Usage error messages now show the --help/--man self-documentation options. - Path-bound built-ins (such as /opt/ast/bin/cat) can now be executed by invoking the canonical path, so the following will now work as expected: $ /opt/ast/bin/cat --version version cat (AT&T Research) 2012-05-31 - 'command -x' now looks for external commands only, skipping built-ins. In addition, its xargs-like functionality no longer freezes the shell on Linux and macOS, making it effectively a new feature on these systems. - 'redirect' now checks if all arguments are valid redirections before performing them. If an error occurs, it issues an error message instead of terminating the shell. - 'suspend' now refuses to suspend a login shell, as there is probably no parent shell to return to and the login session would freeze. - 'times' now gives high precision output in a POSIX compliant format. - 'typeset' now gives an informative error message if an incompatible combination of options is given. - 'whence -v/-a' now reports the location of autoloadable functions. New features in shell options: - A new --globcasedetect shell option is added on OSs where we can check for a case-insensitive file system (currently Windows/Cygwin, macOS, Linux and QNX 7.0+). When this option is turned on, file name generation (globbing), as well as file name tab completion on interactive shells, automatically become case-insensitive on file systems where the difference between upper and lower case is ignored for file names. This is transparently determined for each directory, so a path pattern that spans multiple file systems can be part case-sensitive and part case-insensitive. - A new --nobackslashctrl shell option disables the special escaping behaviour of the backslash character in the emacs and vi built-in editors. Particularly in the emacs editor, this makes it much easier to go backward, insert a forgotten backslash into a command, and then continue editing without having your next cursor key replace your backslash with garbage. Note that Ctrl+V (or whatever other character was set using 'stty lnext') always escapes all control characters in either editing mode. - A new --posix shell option has been added to ksh 93u+m that makes the ksh language more compatible with other shells by following the POSIX standard more closely. See the manual page for details. It is enabled by default if ksh is invoked as sh, otherwise it is disabled by default. - Enhancement to -G/--globstar: symbolic links to directories are now followed if they match a normal (non-**) glob pattern. For example, if '/lnk' is a symlink to a directory, '/lnk/**' and '/l?k/**' now work as you would expect.
2021-12-17 03:18:03 +00:00
- Release 1.0.0-beta.2.
Re-fix defining types conditionally or in subshells (re: f508660d) New version. I'm pretty sure the problems that forced me to revert it earlier are fixed. This commit mitigates the effects of the hack explained in the referenced commit so that dummy built-in command nodes added by the parser for declaration/assignment purposes do not leak out into the execution level, except in a relatively harmless corner case. Something like if false; then typeset -T Foo_t=(integer -i bar) fi will no longer leave a broken dummy Foo_t declaration command. The same applies to declaration commands created with enum. The corner case remaining is: $ ksh -c 'false && enum E_t=(a b c); E_t -a x=(b b a c)' ksh: E_t: not found Since the 'enum' command is not executed, this should have thrown a syntax error on the 'E_t -a' declaration: ksh: syntax error at line 1: `(' unexpected This is because the -c script is parsed entirely before being executed, so E_t is recognised as a declaration built-in at parse time. However, the 'not found' error shows that it was successfully eliminated at execution time, so the inconsistent state will no longer persist. This fix now allows another fix to be effective as well: since built-ins do not know about virtual subshells, fork a virtual subshell into a real subshell before adding any built-ins. src/cmd/ksh93/sh/parse.c: - Add a pair of functions, dcl_hactivate() and dcl_dehacktivate(), that (de)activate an internal declaration built-ins tree into which check_typedef() can pre-add dummy type declaration command nodes. A viewpath from the main built-ins tree to this internal tree is added, unifying the two for search purposes and causing new nodes to be added to the internal tree. When parsing is done, we close that viewpath. This hides those pre-added nodes at execution time. Since the parser is sometimes called recursively (e.g. for command substitutions), keep track of this and only activate and deactivate at the first level. (Fixed compared to previous version of this commit: calling dcl_dehacktivate() when the recursion level is already zero is now a harmless no-op. Since this only occurs in error handling conditions, who cares.) - We also need to catch errors. This is done by setting libast's error_info.exit variable to a dcl_exit() function that tidies up and then passes control to the original (usually sh_exit()). (Fixed compared to previous version of this commit: dcl_exit() immediately deactivates the hack, no matter the recursion level, and restores the regular sh_exit(). This is the right thing to do when we're in the process of erroring out.) - sh_cmd(): This is the most central function in the parser. You'd think it was sh_parse(), but $(modern)-form command substitutions use sh_dolparen() instead. Both call sh_cmd(). So let's simply add a dcl_hacktivate() call at the beginning and a dcl_deactivate() call at the end. - assign(): This function calls path_search(), which among many other things executes an FPATH search, which may execute arbitrary code at parse time (!!!). So, regardless of recursion level, forcibly dehacktivate() to avoid those ugly parser side effects returning in that context. src/cmd/ksh93/bltins/enum.c: b_enum(): - Fork a virtual subshell before adding a built-in. src/cmd/ksh93/sh/xec.c: sh_exec(): - Fork a virtual subshell when detecting typeset's -T option. Improves fix to https://github.com/ksh93/ksh/issues/256
2021-12-17 00:02:01 +00:00
- Ksh no longer behaves badly when parsing a type definition command
('typeset -T' or 'enum') without executing it or when executing it in
a subshell. Types can now safely be defined in subshells and defined
conditionally as in 'if condition; then enum ...; fi'.
Backport ksh93v- bugfix for [[ 1<2 ]] (#380) Strings compared in [[ with the > and < operators should be compared lexically. This does not work when the strings are single digits, as the parser interprets it as a syntax error: $ [[ 10<2 ]] # 10 lexically sorts before 2 $ echo $? 0 $ [[ 1<2 ]] /usr/bin/ksh: syntax error: `<' unexpected $ echo $? 3 src/cmd/ksh93/sh/lex.c: - Don't interpret numbers next to > and < as a redirection while inside of [[. This bugfix was backported from ksh93v- 2014-06-25. src/cmd/ksh93/tests/bracket.sh: - Add regression tests for the > and < operators.
2021-12-17 02:24:56 +00:00
- Single digits can now be compared lexically in [[ ... ]] with the
< and > operators.
New selection of default libcmd /opt/ast/bin built-ins Note that this is only about the /opt/ast/bin built-in commands, not about the regular pathless builtins such as printf. To use these, either add /opt/ast/bin to your $PATH or use a command like 'builtin cp'. As usual, --man provides info. Removed as defaults for lack of convincing advantages over the OS's external commands: - chmod, cmp, head, logname, mkdir, sync, uname, wc Remain as useful defaults: - basename, cat, cut, dirname. These are commonly used in performance-sensitive code paths in scripts and having them as built-ins can be good for performance. - getconf: This is the only interface to some libast internals that is available to ksh. It's also has better functionality than most OS-shipped 'getconf' commands, e.g., it can list and query all the configuration values. Added as defaults: - cp, ln, mv: Having these built in can speed up scripts that manage files. Also the AST versions have extended functionality (see cp --man, etc.). - mktemp: External mktemp commands vary too widely and are incompatible, but it's important that scripts can securely make temporary files, so it's good to ship a known interface to this functionality. As a result, the statically linked ksh binary is very slightly smaller than before. Resolves: https://github.com/ksh93/ksh/issues/349
2021-12-16 07:48:06 +00:00
2021-12-16:
- Changed the default selection of compiled-in /opt/ast/bin built-in libcmd
command to: basename, cat, cp, cut, dirname, getconf, ln, mktemp, mv.
Add /opt/ast/bin to your $PATH to use these. Type 'cp --man', etc. for info.
Do not exit shell on Ctrl+C with SIGINT ignored (re: 7e5fd3e9) The killpg(getpgrp(),SIGINT) call added to ed_getchar() in that commit caused the interactive shell to exit on ^C even if SIGINT is being ignored. We cannot revert or remove that call without breaking job control. This commit applies a new fix instead. Reproducers fixed by this commit: SIGINT ignored by child: $ PS1='childshell$ ' ksh childshell$ trap '' INT childshell$ (press Ctrl+C) $ SIGINT ignored by parent: $ (trap '' INT; ENV=/./dev/null PS1='childshell$ ' ksh) childshell$ (press Ctrl+C) $ SIGINT ignored by parent, trapped in child: $ (trap '' INT; ENV=/./dev/null PS1='childshell$ ' ksh) childshell$ trap 'echo test' INT childshell$ (press Ctrl+C) $ I've experimentally determined that, under these conditions, the SFIO stream error state is set to 256 == 0400 == SH_EXITSIG. src/cmd/ksh93/sh/main.c: exfile(): - On EOF or error, do not return (exiting the shell) if the shell state is interactive and if sferror(iop)==SH_EXITSIG. - Refactor that block a little to make the new check fit in nicely. src/cmd/ksh93/tests/pty.sh: - Test the above three reproducers. Fixes: https://github.com/ksh93/ksh/issues/343
2021-12-16 15:40:42 +00:00
- A bug introduced on 2020-09-17 was fixed that caused interactive ksh to exit
if Ctrl+C was pressed while SIGINT was being ignored (as in "trap '' INT").
Re-match '.' and '..' in tab completion (re: 5312a59d, aad74597) Turns out there is a bona fide, honest-to-goodness use case for matching '.' and '..' in globbing after all. It's when globbing is used as the backend mechanism for file name completion in interactive shell editors. A tab invisibly adds a * at the end of the word to the left of your cursor and the resulting pattern is expanded. In 5312a59d, this broke for '.' and '..'. Typing '.' followed by two tabs should result in a menu that includes './' and '../'. Typing '..' followed by a tab should result in '../', (or a menu that includes it if there are files with names starting with '..'). This is the behaviour in 93u+ and we should maintain this. To restore this functionality without reintroducing the harmful behaviour fixed in the referenced commits, we should special-case this, allowing '.' and '..' to match only for file name completion. src/lib/libast/include/glob.h: - Fix an inaccurate comment: the GLOB_COMPLETE flag is used for command completion, not file name completion. This is very clear from reading the path_expand() function in sh/expand.c. - Add new GLOB_FCOMPLETE flag for file name completion. src/lib/libast/misc/glob.c: - Adapt flags mask to fit the new flag. - glob_dir(): If GLOB_FCOMPLETE is passed, allow '.' and '..' to match even if expanded from a pattern. - Clarify the fix from aad74597 with an extended comment based on <https://github.com/ksh93/ksh/issues/146#issuecomment-790991990>. src/cmd/ksh93/sh/expand.c: path_expand(): - If we're in the SH_FCOMPLETE (file name completion) state, then pass the new GLOB_FCOMPLETE flag to AST glob(3). Fixes: https://github.com/ksh93/ksh/issues/372 Thanks to @fbrau for the bug report.
2021-12-13 00:33:26 +00:00
2021-12-13:
- Fixed a bug introduced on 2020-08-09 that prevented '.' and '..' from
being completed when using file name tab completion.
The `chown` builtin should fail with the same error consistently (#378) This bug was first reported at <https://www.illumos.org/issues/3782>. The chown builtin when used on illumos can fail with different error messages after running the same command twice: $ touch /tmp/x $ /opt/ast/bin/chown -h 433:434 /tmp/px chown: /tmp/x: cannot change owner and group [Not owner] $ /opt/ast/bin/chown -h 433:434 /tmp/px chown: /tmp/x: cannot change owner and group [Invalid argument] The error messages differ because the libast struid and strgid functions will return -2 if the same nonexistent ID is used twice. The fix for this bug has been ported from here: https://github.com/illumos/illumos-gate/commit/4162633a7c5961f388fd src/lib/libcmd/chgrp.c: - Remove NOID macro and check for a < 0 error status instead. This is different from the Illumos fix at <https://github.com/illumos/illumos-gate/commit/4162633a7c59> which added another macro. src/lib/libast/man/{strgid,struid}.3: - Correct errors in the strgid and struid documentation. - Document that the strgid and struid functions will return -2 if the same invalid name is used twice. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-12-14 09:22:30 +00:00
- Fixed a bug on illumos that caused the chown builtin to fail with 'Invalid
argument' after failing to change the ownership of a file twice using an ID
that doesn't exist in /etc/passwd. (Note that the chown builtin is not
enabled by default.)
Fix two more PS2/SIGINT crashing bugs (re: 3023d53b) *** Crash 1: *** ksh crashed if the PS1 prompt contains one or more command substitutions and you enter a multi-line command substitution on the command line, then interrupt while on the PS2 prompt. $ ENV=/./dev/null /usr/local/bin/ksh -o emacs $ PS1='$(echo foo) $(echo bar) $(echo baz) ! % ' foo bar baz 16999 % echo $( > true <-- here, press Ctrl+C instead of Return Memory fault The crash occurred due to a corrupted lexer state while trying to display the PS1 prompt. Analysis: My fix for the crashing bug with Ctrl+C in commit 3023d53b is incorrect and only worked accidentally. sh_fault() is not the right place to reset the lexer state because, when we press Ctrl+C on a PS2 prompt, ksh had been waiting for input to finish lexing a multi-line command, so sh_lex() and other lexer functions are on the function call stack and will be returned to. src/cmd/ksh93/sh/fault.c: sh_fault(): - Remove incorrect SIGINT fix. src/cmd/ksh93/sh/io.c: io_prompt(): - Reset the lexer state immediately before printing every PS1 prompt. Even in situations where this is redundant it should be perfectly safe, the overhead is negligible, and it resolves this crash. It may pre-empt other problems as well. *** Crash 2: *** If an INT trap is set, and you start entering a multi-line command substitution, then press Ctrl+C on the PS2 prompt to trigger the crash, the lexer state is corrupted because the lexer is invoked to eval the trap action. A crash then occurs on entering the final ')' of the command substitution. $ trap 'echo TRAPPED' INT $ echo $( > trueTRAPPED <-- press Ctrl+C to output "TRAPPED" > ) Memory fault Technically, as SIGINT is trapped, it should not interrupt, so ksh should execute the trap, then continue with the PS2 prompt to let the user finish inputting the command. But I have been unsuccessful in many different attempts to make this work properly. I managed to get multi-line command substitutions to lex correctly by saving and restoring the lexer state, but command substitutions were still corrupted at the parser and/or execution level and I have not managed to trace the cause of that. My testing showed that all other shells interrupt the PS2 prompt and return to PS1 when the user presses Ctrl+C, even if SIGINT is trapped. I think that is a reasonable alternative, and it is something I managed to make work. src/cmd/ksh93/sh/fault.c: sh_chktrap(): - Immediately after invoking sh_trap() to run a trap action, check if we're in a PS2 prompt (sh.nextprompt == 2). If so, assume the lexer state is now overwritten. Closing the fcin stream with fcclose() seems to reliably force the lexer to stop doing anything else. Then we can just reset the prompt to PS1 and invoke sh_exit() to start new command line, which will now reset the lexer state as per above.
2021-12-11 03:27:54 +00:00
2021-12-11:
- Fixed two more crashing bugs that occurred if ksh received a signal (such
as SIGINT due to Ctrl+C) while the user is entering a multi-line command
substitution in an interactive shell.
Port more shell lint improvements from illumos and ksh93v- (#374) This commit adds onto <https://github.com/ksh93/ksh/pull/353> by porting over two additional improvements to the shell linter: 1) The changes in the aforementioned pull request were merged into illumos-gate with an additional change.[*] The illumos revision of the patch improved the warning for (( $foo = $? )) to specify '$foo' causes the warning.[**] Example: $ ksh -n -c '(( $? != $bar ))' ksh: warning: line 1: in '(( $? != $bar ))', using '$' as in '$bar' is slower and can introduce rounding errors While I was porting the illumos patch I did notice one problem. The string it uses from paramsub() skips over the initial '{' in '${var}', resulting in the warning printing '$var}' instead: $ ksh -n -c '(( ${.sh.pid} != $$ ))' ... in '(( ${.sh.pid} != $$ ))', using '$' as in '$.sh.pid}' is slower ... This was fixed by including the missing '{' in the string returned by paramsub for ${var} variables. 2) In ksh93v-, parsing x=$((expr)) with the shell linter will cause ksh to warn the user x=$((expr)) is slower than ((x=expr)). This improvement has been backported with a modified warning: # Result from this commit $ ksh -n -c 'x=$((1 + 2))' ksh: warning: line 1: x=$((1 + 2)) is slower than ((x=1 + 2)) # Result from ksh93v- $ ksh93v -n -c 'x=$((1 + 2))' ksh93v: warning: line 1: ((x=1 + 2)) is more efficient than x=$((1 + 2)) Minor note: the ksh93v- patch had an invalid use of memcmp; this version of the patch uses strncmp instead. References: https://github.com/illumos/illumos-gate/commit/be548e87bcb4979e0b8a7bce5620a9916b888fdc https://code.illumos.org/c/illumos-gate/+/1834/comment/65722363_22fdf8e7/
2021-12-12 21:25:07 +00:00
- The shell linter's warning for variable expansion in ((...)) now tells the
user which variable is causing performance degradation.
- The shell linter now warns the user x=$((expr)) is slower than ((x=expr))
when assigning a value to a variable.
Reset lexer state on syntax error and on SIGINT (Ctrl+C) ksh crashed if you pressed Ctrl+C or Ctrl+D on a PS2 prompt while you haven't finished entering a $(command substitution). It corrupts subsequent command substitutions. Sometimes the situation recovers, sometimes the shell crashes. Simple crash reproducer: $ PS1="\$(echo foo) \$(echo bar) \$(echo baz) > " foo bar baz > echo $( <-- now press Ctrl+D > ksh: syntax error: `(' unmatched Memory fault The same happens with Ctrl+C, minus the syntax error message. The problem is that the lexer state becomes inconsistent when the lexer is interrupted in the middle of reading a command substitution of the form $( ... ). This is tracked in the 'lexd.dolparen' variable in the lexer state struct. Resetting that variable is sufficient to fix this issue. However, in this commit I prefer to just reinitialise the lexer state completely to pre-empt any other possible issues. Whether there was a syntax error or the user pressed Ctrl+C, we just interrupted all lexing and parsing, so the lexer *should* restart from scratch. src/cmd/ksh93/sh/fault.c: sh_fault(): - If the shell is in an interactive state (e.g. not a subshell) and SIGINT was received, reinitialise the lexer state. This fixes the crash with Ctrl+C. src/cmd/ksh93/sh/lex.c: sh_syntax(): - When handling a syntax error, reset the lexer state. This fixes the crash with Ctrl+D. NEWS: - Also add the forgotten item for the previous fix (re: 2322f939).
2021-12-09 06:05:36 +00:00
2021-12-09:
- Increased the general robustness of discipline function handling, fixing
crashing bugs with PS2.get() and .sh.tilde.get() disciplines, among others.
- Fixed a crash that occurred on the interactive shell if the PS1 prompt
contains multiple command substitutions and the user interrupts input
while the shell is on a PS2 prompt waiting for the user to complete a
command substitution of the form $( ... ).
Fix 'return' emitting signals; allow arbitrary return values When a global EXIT trap is set, and a ksh-style function exits with a status > 256 that could have been the result of a signal, then the shell incorrectly issues that signal to itself. Depending on the signal, this causes ksh to terminate itself ungracefully: $ cat /tmp/exit267 trap 'echo OK' EXIT # This trap triggers the crash function foo { return 267; } foo $ bash /tmp/exit267 OK $ ksh-3aee10d7 /tmp/exit267 OK $ ksh /tmp/exit267 Memory fault(coredump) On most systems, status 267 corresponds to SIGSEGV. The reported memory fault is not real; it results from ksh incorrectly killing itself with that signal. The problem is caused by two factors: 1. As of 93u+ 2012-08-01, ksh explicitly allows 'return' to use an exit status corresponding to a signal (from 257 to end of signal range). The rest of the integer range is trunctated to 8 bits. This is contrary to both 'man ksh' and 'return --man' which both say it's always truncated to 8 bits. Plus, combined with point 2 below, this new behaviour is nonsensical, as 'return' has no business actually generating signals. However, a couple of regression tests now depend on this, as may some scripts. 2. When a ksh-style function does not handle a signal, the signal is passed down to the parent environment and ksh does this by reissuing the signal to its own process after leaving the function scope. However, it does this by checking the exit status, which is very bad practice as there is no guarantee that an exit status corresponding to a signal was in fact produced by a signal, particularly after they changed the behaviour of 'return' per 1 above. This commit fixes both issues. It also takes a proper decision on allowable 'return' exit status arguments. Since 93u+ was released nearly a decade ago and some scripts may now rely on being able to pass certain exit statuses out of the 8-bit range, we should not disallow this now. But neither should we be half-hearted in allowing only some arbitrary selection of 9-bit statuses; 'return' values categorically should have nothing to do with signals, so this is no basis for limiting them. We're now allowing the full unsigned integer range, which is usually 32 bits. This is like zsh, and may create some interesting possibilities for scripts. Just don't forget that $? will still lose all but its 8 least significant bits when leaving the current (sub)shell environment. src/cmd/ksh93/sh/xec.c: sh_funscope(): - Fix passing down unhandled signals from interrupted ksh functions (jumpval==SH_JMPFUN) to the parent environment. Do not pay any attention to the exit status. Instead, use sh.lastsig (a.k.a. shp->lastsig). It is set by sh_fault() in fault.c for just this purpose and contains the last signal handled for the current command. It is reset in sh_exec() before running any new command. So if it contains a signal, that is the one that interrupted the ksh function, so it's the correct one to pass down. (Further evidence: sh_subshell() was already using this in the same way.) src/cmd/ksh93/bltins/cflow.c: b_return(): - Allow any signed int return value when invoked as and behaving like 'return'. - Add warning if a passed value is out of int range. Set the exit status to 128 in that case; int overflow is undefined behaviour in C and we want consistent behaviour across platforms. It should be safe enough to check if the long and int values are equal. - Refactor for clarity. src/cmd/ksh93/sh/subshell.c: sh_subshell(): - If a function returns with a status out of the 8 bit range in a virtual subshell, this status could be passed down to the parent shell in full. However, if the subshell forks, then the kernel will enforce an 8-bit exit status. That is inconsistent. Scripts should not be able to tell the difference between forked and non-forked subshells, so artificially enforce that limit here. Other changed files: - Documentation updates and copy-edits. - Update an AT&T functions.sh regress test to allow arbitrary integer return values for functions. - Add regression tests based in part on @JohnoKing's reproducers. - Rework some vaguely related regression tests to fail gracefully. Thanks to Johnothan King for the report and the testing. Fixes: https://github.com/ksh93/ksh/issues/364
2021-12-08 03:14:15 +00:00
2021-12-08:
- Fixed: if a function returned with a status > 256 using the 'return' command
and the return value corresponded to a value that could have resulted from a
signal, and an EXIT trap was active, then the shell mistakenly issued that
signal to itself. Depending on the signal, this could cause the shell to
terminate ungracefully, e.g. 'return 267' caused SIGSEGV ("memory fault").
- For the 'return' built-in command, you can now freely specify any
return value that fits in a signed integer, typically a 32-bit value.
Note that $? is truncated to 8 bits when the current (sub)shell exits.
Fix handling of files without newlines in the `head` and `tail` builtins (#365) The head and tail builtins don't correctly handle files that lack newlines[*]: $ print -n foo > /tmp/bar $ /opt/ast/bin/head -1 /tmp/bar # No output $ print -n 'foo\nbar' > /tmp/bar $ /opt/ast/bin/tail -1 /tmp/bar foo bar$ This commit backports the required changes from ksh93v- to handle files without a newline in the head and tail builtins. (Also note that the required fix to sfmove was already backported in commit 1bd06207.) src/lib/libcmd/{head,tail}.c: - Backport the relevant ksh93v- code for handling files without newlines. src/cmd/ksh93/tests/builtins.sh: - Add a few regression tests for using 'head -1' and 'tail -1' on a file missing and ending newline. [*]: https://www.illumos.org/issues/4149
2021-12-08 06:56:52 +00:00
- The head and tail builtins now correctly handle files that do not have an
ending newline. (Note that the tail builtin is not compiled in by default.)
Port over illumos fixes for conf.sh (#361) This commit ports over two of Andy Fiddaman's bugfixes to conf.sh on illumos: - The compiler isn't passed on to an invocation of iffe. The bugfix is from this commit: <https://github.com/citrus-it/ast/commit/63563232> - The getconf builtin is missing several parameters on illumos. Reproducer: $ /opt/ast/bin/getconf ADDRESS_WIDTH getconf: Invalid argument (ADDRESS_WIDTH) # Should output '64' This bug occurs because conf.sh expects GNU sed and fails to work properly with other sed implementations. The bugfix and original bug report can be found here: https://www.illumos.org/issues/14044 https://github.com/citrus-it/ast/commit/ba443cfd
2021-12-05 18:55:15 +00:00
2021-12-05:
- Fixed an issue on illumos that caused some parameters in the getconf
builtin to fail.
Add the '-e' flag to the 'cd' builtin (#358) This change adds the -e flag to the cd builtin, as specified in <https://www.austingroupbugs.net/view.php?id=253>. The -e flag is used to verify if the the current working directory after 'cd -P' successfully changes the directory, and returns with exit status 1 if the cwd couldn't be determined. Additionally, it causes all other errors to return with exit status >1 (i.e., status 2 unless ENOMEM occurs) if -e and -P are both active. src/cmd/ksh93/bltins/cd_pwd.c: - Add -e option to the cd builtin command. It verifies $PWD by using test_inode() to execute the equivalent of [[ . -ef $PWD ]]. - The check for restricted mode has been moved after optget to allow 'cd -eP' to return with exit status 2 when in restricted mode. To avoid changing the previous behavior of cd when -e isn't passed, extra checks have been added to prevent cd from printing usage information in restricted mode. src/cmd/ksh93/tests/builtins.sh: - Add regression tests for the exit status when using the cd -P flag with and without -e. src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/sh.1: - Document the addition of -e to the cd builtin.
2021-12-06 05:48:32 +00:00
- The cd built-in command now supports a -e option (as specified in
https://www.austingroupbugs.net/view.php?id=253). Passing -e alongside -P
is used to guarantee the cd built-in returns with exit status 1 if the
current working directory couldn't be determined after successfully changing
the directory.
nvdisc.c: Fix crash after an error or signal in discipline function (#356) This patch fixes the crashes experienced when a discipline function exited because of a signal or an error from a special builtin. The crashes were caused by ksh entering an inconsistent state after performing a longjmp away from the assign() and lookup() functions in nvdisc.c. Fixing the crash requires entering a new context, then setting a nonlocal goto with sigsetjmp(3). Any longjmps that happen while running the discipline function will go back to assign/lookup, allowing ksh to do a proper cleanup afterwards. Resolves: https://github.com/ksh93/ksh/issues/346
2021-12-02 04:43:41 +00:00
2021-12-01:
- Fixed a memory fault that occurred when a discipline function exited
with an error from a special builtin or when a discipline function exited
because of a signal.
Stop the time keyword overriding errexit (#351) This bug was first reported in <https://www.illumos.org/issues/7694>. The time keyword currently overrides the errexit shell option, allowing failing scripts to continue after an error: $ cat 1.sh #!/bin/sh time false # This should cause the script to exit echo FAILURE true $ ksh -o errexit 1.sh real 0m0.00s user 0m0.00s sys 0m0.00s FAILURE src/cmd/ksh93/sh/xec.c: - When the time keyword runs a command, pass the errexit state flag to the sh_exec call. This state flag is required for ksh to exit when a command fails while the errexit option is on. src/cmd/ksh93/tests/basic.sh: - Add a regression test based on the reproducer.
2021-11-29 08:16:23 +00:00
2021-11-29:
Fix memory fault on ARM-based Macs (#354) Ksh segfaults on M1 Macs, apparently because Apple's compiler optimizer can't cope with overriding bzero(3) with libast's implementation (though it's nothing more than "memset(b, 0, n);"). Apple has disabled libast's bzero function since 2018-12-04: https://opensource.apple.com/source/ksh/ksh-27/patches/src__lib__libast__comp__omitted.c.diff.auto.html src/lib/libast/comp/omitted.c: - Only fall back to the libast bzero function if the OS lacks an implementation of bzero. - Remove the bzero undef since this fallback is only reached if the OS doesn't have bzero. NEWS: - Add compat info for macOS on ARM64. This notes that macOS Monterey can now compile and run ksh, although there is one regression test failure: builtins.sh[345]: printf %H: invalid UTF-8 characters (expected %3F%C2%86%3F%3F%3F; got %3F%C2%86%3F%3Fv%3F%3F) Thanks to @DesantBucie for the report and the testing. Resolves: https://github.com/ksh93/ksh/issues/329
2021-11-29 12:38:17 +00:00
- Fixed a memory fault that prevented ksh from functioning on ARM-based Macs.
Stop the time keyword overriding errexit (#351) This bug was first reported in <https://www.illumos.org/issues/7694>. The time keyword currently overrides the errexit shell option, allowing failing scripts to continue after an error: $ cat 1.sh #!/bin/sh time false # This should cause the script to exit echo FAILURE true $ ksh -o errexit 1.sh real 0m0.00s user 0m0.00s sys 0m0.00s FAILURE src/cmd/ksh93/sh/xec.c: - When the time keyword runs a command, pass the errexit state flag to the sh_exec call. This state flag is required for ksh to exit when a command fails while the errexit option is on. src/cmd/ksh93/tests/basic.sh: - Add a regression test based on the reproducer.
2021-11-29 08:16:23 +00:00
- A bug that caused the time keyword to override the errexit shell option has
been fixed.
Fix `KEYBD` trap crash when inputting a command substitution (#355) This change fixes a crash that can occur after setting a KEYBD trap then inputting a multi-line command substitution. The crash is similar to issue #347, but it's easier to reproduce since it doesn't require you to setup a kshrc file. Reproducer for the crash: $ ENV=/./dev/null ksh $ trap : KEYBD $ : $( > true) Memory fault(coredump) The bugfix was backported (with considerable changes) from ksh93v- 2013-10-08. The crash was first reported on the old mailing list: https://www.mail-archive.com/ast-users@lists.research.att.com/msg00313.html src/cmd/ksh93/{include/shlex.h,sh/lex.c}: - To fix this properly, we need sizeof(Lex_t) to work as expected in edit.c, but that is thwarted by the _SHLEX_PRIVATE macro in lex.c which shlex.h uses to add private structs to the Lex_t type in lex.c only. So get rid of that _SHLEX_PRIVATE macro and make those members part of the centrally defined struct, renaming them to make it clear they're considered private to lex.c. src/cmd/ksh93/edit/edit.c: - Now that we can get its size, save and restore the shell lexing context when a KEYBD trap is present. src/cmd/ksh93/tests/pty.sh: - Add a regression test for the KEYBD trap crash. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-11-30 03:26:53 +00:00
- Fixed a crash that could occur when a KEYBD trap was set and a multi-line
command substitution was input in an interactive shell.
Port illumos' shell linter improvements (#353) This commit ports over two improvements to the shell linter from illumos (original patch written by Andy Fiddaman). Links to the relevant bug reports and the original patch: https://www.illumos.org/issues/13601 https://www.illumos.org/issues/13631 https://github.com/illumos/illumos-gate/commit/c7b656fc7125adc5888ce934f1a3433da4b1bbc5 The first improvement is to the lint warning for arithmetic operators in [[ ... ]]. The ksh linter now suggests the correct equivalent operator to use in ((...)). Example: $ ksh -nc '[[ 30 -gt 25 ]]' # Original warning warning: line 1: -gt within [[ ... ]] obsolete, use ((...)) # New warning warning: line 1: [[ ... -gt ... ]] obsolete, use ((... > ...)) The second improvement pertains to variable expansion in arithmetic expressions. The ksh linter now suggests referencing variable names directly: $ ksh -nc 'integer foo=40; (($foo < 50 ))' # Old warning warning: line 1: variable expansion makes arithmetic evaluation less efficient # New warning warning: line 1: in '(($foo < 50))', using '$' is slower and can introduce rounding errors src/cmd/ksh93/{data/lexstates,sh/lex,sh/parse}.c: - Port the improved shell lint warnings from illumos to ksh93u+m. - The original checks for arithmetic operators involved a bunch of if statements with inefficient calls to strcmp(3). These were replaced with a more efficient switch statement that avoids strcmp.
2021-11-30 20:26:17 +00:00
- The shell linter's warnings for obsolete arithmetic operators in [[ ... ]]
and unnecessary variable expansion in ((...)) have been improved.
'.': disable ksh function lookup in POSIX mode POSIXly, '.' loads only files, not functions. This only applies to '.', not 'source' (which is not in POSIX). src/cmd/ksh93/bltins/misc.c: b_source(): - For ksh function lookup, add an additional check that we're not in POSIX mode and running the '.' (SYSDOT) builtin.
2021-11-24 08:10:25 +00:00
2021-11-24:
- The --posix mode was amended to stop the '.' command (but not 'source') from
looking up functions defined with the 'function' keyword. In the POSIX
standard and on other shells, the '.' command finds only script files.
Backport the ksh93v- rm builtin to fix 'rm -d' (#348) The -d flag implemented in the rm builtin is completely broken. No matter what you do it refuses to remove directories, even if -r is also passed. Reproducer: $ mkdir /tmp/empty $ PATH=/opt/ast/bin rm -d /tmp/empty rm: /tmp/empty: directory $ PATH=/opt/ast/bin rm -dr /tmp/empty rm: /tmp/empty: directory not removed [Is a directory] Additionally, the description of 'rm -d' in the man page contradicts how it's specified in <https://www.austingroupbugs.net/view.php?id=802>. The ksh93v- rm builtin fixed nearly all of these issues, so I've backported it to 93u+m and applied one additional fix for 'rm -rd'. src/lib/libcmd/rm.c: - Backported the fixes from the ksh93v- rm builtin's -d flag when used on empty directories. - Backported the man page update for rm(1) from ksh93v-. - The ksh93v- rm builtin had one additional bug that caused the -r option to fail when combined with -d. This was fixed by overriding -d if -r is also passed. src/cmd/ksh93/tests/builtins.sh: - Add regression tests for the rm builtin's -d option.
2021-11-25 02:45:21 +00:00
- The rm built-in's -d/--directory option has been fixed. It now properly
removes empty directories and refuses to remove non-empty directories
(as specified in https://www.austingroupbugs.net/view.php?id=802). Note
that the rm built-in command isn't compiled in by default.
arith: implement range checking for enum types Within arithmetic expressions, enumeration values of variables of a type created with the 'enum' command translate to index numbers from 0 to the number of elements minus 1. However, there was no range checking on this in the arithmetic subsystem, allowing the assignment of out-of-range values that did not correspond to any enumeration value. Variables of an enum type are internally unsigned short integers (NV_UINT16), like those created with 'integer -su', except with an additional discipline function (ENUM_disc). src/cmd/ksh93/bltins/enum.c, src/cmd/ksh93/include/builtins.h: - To implement range checking, the arithmetic system needs access to the 'nelem' (number of elements) member of 'struct Enum'. This is only defined locally in enum.c. We could move that to name.h so arith.c can access it, but enum.c has code that supports compiling as standalone. So, instead, define a quick extern function, b_enum_elem(), that does the necessary type conversion and returns a type's number of elements. - Add --man documentation for the arithmetic subsystem behaviour for enum types. Tell the enuminfo() function, which dynamically inserts values into the documentation, how to process new \f tags 'lastv' (the last-defined value) and 'lastn' (the number of the last element). src/cmd/ksh93/sh/arith.c: arith(): - For NV_UINT16 variables with an ENUM_disc discipline, check the range using b_enum_elem() and error out if necessary. Resolves: https://github.com/ksh93/ksh/issues/335
2021-11-23 20:35:30 +00:00
2021-11-23:
- A bug was fixed that allowed arithmetic expressions to assign out-of-range
values to variables of an enumeration type defined with the 'enum' command,
causing undefined behavior. Within arithmetic expressions, enumeration
values translate to index numbers from 0 to the number of elements minus 1.
That range is now checked for. Decimal fractions are ignored.
Fix enum type definition pre-parsing for shcomp and dot/source Parser limitations prevent shcomp or source from handling enum types correctly: $ cat /tmp/colors.sh enum Color_t=(red green blue orange yellow) Color_t -A Colors=([foo]=red) $ shcomp /tmp/colors.sh > /dev/null /tmp/colors.sh: syntax error at line 2: `(' unexpected $ source /tmp/colors.sh /bin/ksh: source: syntax error: `(' unexpected Yet, for types created using 'typeset -T', this works. This is done via a check_typedef() function that preliminarily adds the special declaration builtin at parse time, with details to be filled in later at execution time. This hack will produce ugly undefined behaviour if the definition command creating that type built-in is then not actually run at execution time before the type built-in is accessed. But the hack is necessary because we're dealing with a fundamental design flaw in the ksh language. Dynamically addable built-ins that change the syntactic parsing of the shell language on the fly are an absurdity that violates the separation between parsing and execution, which muddies the waters and creates the need for some kind of ugly hack to keep things like shcomp more or less working. This commit extends that hack to support enum. src/cmd/ksh93/sh/parse.c: - check_typedef(): - Add 'intypeset' parameter that should be set to 1 for typeset and friends, 2 for enum. - When processing enum arguments, use AST getopt(3) to skip over enum's options to find the name of the type to be defined. (getopt failed if we were running a -c script; deal with this by zeroing opt_info.index first.) - item(): Update check_typedef() call, passing lexp->intypeset. - simple(): Set lexp->intypeset to 2 when processing enum. The rest of the changes are all to support the above and should be fairly obvious, except: src/cmd/ksh93/bltins/enum.c: - enuminfo(): Return on null pointer, avoiding a crash upon executing 'Type_t --man' if Type_t has not been fully defined due to the definition being pre-added at parse time but not executed. It's all still wrong, but a crash is worse. Resolves: https://github.com/ksh93/ksh/issues/256
2021-11-21 06:12:27 +00:00
2021-11-21:
- It is now possible to use types defined by 'enum' in contexts where the
script is entirely parsed before (or without) being executed, such as
dotted/sourced scripts and scripts compiled by shcomp.
Add support for 'stty size' to the libcmd 'stty' builtin (#342) This commit adds support for 'stty size' to the stty builtin, as defined in <https://austingroupbugs.net/view.php?id=1053>. The size mode is used to display the terminal's number of rows and columns. Note that stty isn't included in the default list of builtin commands; testing this addition requires adding CMDLIST(stty) to the table of builtins in src/cmd/ksh93/data/builtins.c. src/lib/libcmd/stty.c: - Add support for the size mode to the stty builtin. This mode is only used to display the terminal's number of rows and columns, so error out if any arguments are given that attempt to set the terminal size.
2021-11-23 00:15:24 +00:00
- Added support for the size mode to the stty(1) built-in. This mode is used
to display the terminal's number of rows and columns. Note that the stty
built-in is not compiled in by default. This can be changed by adding
stty to the table of built-ins in src/cmd/ksh93/data/builtins.c.
`typeset -T` shouldn't list types created with `enum` (#340) Listing types with 'typeset -T' will list not only types created with typeset, but also types created with enum. However, the types created by enum are not displayed correctly in the resulting output: $ enum Foo_t=(foo bar) $ typeset -T typeset -T Foo_t typeset -T Foo_t=fo) The fix for this bug was backported from ksh93v- 2013-10-08. src/cmd/ksh93/sh/nvtype.c: - sh_outtype(): Skip over enums when listing types with 'typeset -T'.
2021-11-20 08:41:49 +00:00
2021-11-20:
- Listing types with 'typeset -T' no longer displays incomplete versions of
types created by the enum built-in.
Backport 'printf -v' from ksh 93v- 'printf' on bash and zsh has a popular -v option that allows assigning formatted output directly to variables without using a command substitution. This is much faster and avoids snags with stripping final linefeeds. AT&T had replicated this feature in the abandoned 93v- beta version. This backports it with a few tweaks and one user-visible improvement. The 93v- version prohibited specifying a variable name with an array subscript, such as printf -v var\[3\] foo. This works fine on bash and zsh, so I see no reason why this should not work on ksh, as nv_putval() deals with array subscripts just fine. src/cmd/ksh93/bltins/print.c: b_print(): - While processing the -v option when called as printf, get a pointer to the variable, creating it if necessary. Pass only the NV_VARNAME flag to enforce a valid variable name, and not (as 93v- does) the NV_NOARRAY flag to prohibit array subscripts. - If a variable was given, set the output file to an internal string buffer and jump straight to processing the format. - After processing the format, assign the contents to the string buffer to the variable. src/cmd/ksh93/data/builtins.c: - Document the new option, adding a warning that unquoted square brackets may trigger pathname expansion.
2021-11-18 22:36:28 +00:00
2021-11-18:
- The printf built-in command now supports a -v option as on bash and zsh.
This allows you to assign formatted output directly to a variable.
Fix a few issues with $RANDOM seeding in subshells (#339) This commit fixes an issue I found in the subshell $RANDOM reseeding code. The main issue is a performance regression in the shbench fibonacci benchmark, introduced in commit af6a32d1. Performance dropped in this benchmark because $RANDOM is always reseeded and restored, even when it's never used in a subshell. Performance results from before and after this performance fix (results are on Linux with CC=gcc and CCFLAGS='-O2 -D_std_malloc'): $ ./shbench -b bench/fibonacci.ksh -l 100 ./ksh-0f06a2e ./ksh-af6a32d ./ksh-f31e368 ./ksh-randfix benchmarking ./ksh-0f06a2e, ./ksh-af6a32d, ./ksh-f31e368, ./ksh-randfix ... *** fibonacci.ksh *** # ./ksh-0f06a2e # Recent version of ksh93u+m # ./ksh-af6a32d # Commit that introduced the regression # ./ksh-f31e368 # Commit without the regression # ./ksh-randfix # Ksh93u+m with this patch applied ------------------------------------------------------------------------------------------------- name ./ksh-0f06a2e ./ksh-af6a32d ./ksh-f31e368 ./ksh-randfix ------------------------------------------------------------------------------------------------- fibonacci.ksh 0.481 [0.459-0.515] 0.472 [0.455-0.504] 0.396 [0.380-0.442] 0.407 [0.385-0.439] ------------------------------------------------------------------------------------------------- src/cmd/ksh93/include/variables.h, src/cmd/ksh93/sh/{init,subshell}.c: - Rather than reseed $RANDOM every time a subshell is created, add a sh_save_rand_seed() function that does this only when the $RANDOM variable is used in a subshell. This function is called by the $RANDOM discipline functions nget_rand() and put_rand(). As a minor optimization, sh_save_rand_seed doesn't reseed if it's called from put_rand(). - Because $RANDOM may have a seed of zero (i.e., RANDOM=0), sp->rand_seed isn't enough to tell if $RANDOM has been reseeded. Add sp->rand_state for this purpose. - sh_subshell(): Only restore the former $RANDOM seed and state if it is necessary to prevent a subshell leak. src/cmd/ksh93/tests/variables.sh: - Add two regression tests for bugs I ran into while making this patch.
2021-11-19 07:17:25 +00:00
- Fixed a performance regression introduced on 2021-05-03 that caused
the shbench[*] fibonacci benchmark to run slower.
[*]: https://github.com/ksh-community/shbench
arithmetic: Fix the octal leading zero mess (#337) In C/POSIX arithmetic, a leading 0 denotes an octal number, e.g. 010 == 8. But this is not a desirable feature as it can cause problems with processing things like dates with a leading zero. In ksh, you should use 8#10 instead ("10" with base 8). It would be tolerable if ksh at least implemented it consistently. But AT&T made an incredible mess of it. For anyone who is not intimately familiar with ksh internals, it is inscrutable where arithmetic evaluation special-cases a leading 0 and where it doesn't. Here are just some of the surprises/inconsistencies: 1. The AT&T maintainers tried to honour a leading 0 inside of ((...)) and $((...)) and not for arithmetic contexts outside it, but even that inconsistency was never quite consistent. 2. Since 2010-12-12, $((x)) and $(($x)) are different: $ /bin/ksh -c 'x=010; echo $((x)) $(($x))' 10 8 That's a clear violation of both POSIX and the principle of least astonishment. $((x)) and $(($x)) should be the same in all cases. 3. 'let' with '-o letoctal' acts in this bizarre way: $ set -o letoctal; x=010; let "y1=$x" "y2=010"; echo $y1 $y2 10 8 That's right, 'let y=$x' is different from 'let y=010' even when $x contains the same string value '010'! This violates established shell grammar on the most basic level. This commit introduces consistency. By default, ksh now acts like mksh and zsh: the octal leading zero is disabled in all arithmetic contexts equally. In POSIX mode, it is enabled equally. The one exception is the 'let' built-in, where this can still be controlled independently with the letoctal option as before (but, because letoctal is synched with posix when switching that on/off, it's consistent by default). We're also removing the hackery that causes variable expansions for the 'let' builtin to be quietly altered, so that 'x=010; let y=$x' now does the same as 'let y=010' even with letoctal on. Various files: - Get rid of now-redundant sh.inarith (shp->inarith) flag, as we're no longer distinguishing between being inside or outside ((...)). src/cmd/ksh93/sh/arith.c: - arith(): Let disabling POSIX octal constants by skipping leading zeros depend on either the letoctal option being off (if we're running the "let" built-in") or the posix option being off. - sh_strnum(): Preset a base of 10 for strtonll(3) depending on the posix or letoctal option being off, not on the sh.inarith flag. src/cmd/ksh93/include/argnod.h, src/cmd/ksh93/sh/args.c, src/cmd/ksh93/sh/macro.c: - Remove astonishing hackery that violated shell grammar for 'let'. src/cmd/ksh93/sh/name.c (nv_getnum()), src/cmd/ksh93/sh/nvdisc.c (nv_getn()): - Remove loops for skipping leading zeroes that included a broken check for justify/zerofill attributes, thereby fixing this bug: $ typeset -Z x=0x15; echo $((x)) -ksh: x15: parameter not set Even if this code wasn't redundant before, it is now: sh_arith() is called immediately after the removed code and it ignores leading zeroes via sh_strnum() and strtonll(3). Resolves: https://github.com/ksh93/ksh/issues/334
2021-11-17 03:24:18 +00:00
2021-11-16:
- By default, arithmetic expressions in ksh no longer interpret a number
with a leading zero as octal in any context. Use 8#octalnumber instead.
Before, ksh would arbitrarily recognize the leading octal zero in some
contexts but not others, e.g., both of:
$ x=010; echo "$((x)), $(($x))"
$ set -o letoctal; x=010; let y=$x z=010; echo "$y, $z"
would output '10, 8'. These now output '10, 10' and '8, 8', respectively.
Arithmetic expressions now also behave identically within and outside
((...)) and $((...)). Setting the --posix compliance option turns on the
recognition of the leading octal zero for all arithmetic contexts.
arith: Fix variables 'nan' and 'inf' in arithmetic for POSIX mode The --posix compliance option now disables the case-insensitive special floating point constants Inf and NaN so that all case variants of $((inf)) and $((nan)) refer to the variables by those names as the standard requires. (BUG_ARITHNAN) src/cmd/ksh93/sh/arith.c: arith(): - Only do case-insensitive checks for "Inf" and "NaN" if the POSIX option is off.
2021-11-15 17:49:19 +00:00
2021-11-15:
- In arithmetic evaluation, the --posix compliance option now disables the
special floating point constants Inf and NaN so that $((inf)) and $((nan))
refer to the variables by those names as the standard requires. (BUG_ARITHNAN)
Fix file descriptor leaks in the `hist` builtin (#336) This commit fixes two file descriptor leaks in the hist built-in. The bugfix for the first file descriptor leak was backported from ksh2020. See: https://github.com/att/ast/issues/872 https://github.com/att/ast/commit/73bd61b5 Reproducer: $ echo no $ hist -s no=yes The second file descriptor leak occurs after a substitution error in the hist built-in (this leak wasn't fixed in ksh2020). Reproducer: $ echo no $ ls /proc/$$/fd $ hist -s no=yes $ hist -s no=yes $ ls /proc/$$/fd src/cmd/ksh93/bltins/hist.c: - Close leftover file descriptors when an error occurs and after 'hist -s' runs a command. src/cmd/ksh93/tests/builtins.sh: - Add two regression tests for both of the file descriptor leaks.
2021-11-16 00:46:47 +00:00
- Fixed two file descriptor leaks in the hist builtin that occurred when
the -s flag ran a command or encountered an error.
Fix [ \( str -a str \) ], [ \( str -o str \) ] Symptoms: $ test \( string1 -a string2 \) /usr/local/bin/ksh: test: argument expected $ test \( string1 -o string2 \) /usr/local/bin/ksh: test: argument expected The parentheses should be irrelevant and this should be a test for the non-emptiness of string1 and/or string2. src/cmd/ksh93/bltins/test.c: - b_test(): There is a block where the case of 'test' with five or less arguments, the first and last one being parentheses, is special-cased. The parentheses are removed as a workaround: argv is increased to skip the opening parenthesis and argc is decreased by 2. However, there is no corresponding increase of tdata.av which is a copy of this function's argv. This renders the workaround ineffective. The fix is to add that increase. - e3(): Do not handle '!' as a negator if not followed by an argument. This allows a right-hand expression that is equal to '!' (i.e. a test for the non-emptiness of the string '!').
2021-11-14 17:27:05 +00:00
2021-11-14:
Fix crash on unsetting .sh.match ksh crashed after unsetting .sh.match and then matching a pattern: $ unset .sh.match $ [[ bar == ba* ]] Memory fault src/cmd/ksh93/sh/init.c: sh_setmatch(): - Do nothing if we cannot get an array pointer to SH_MATCHNOD.
2021-11-15 02:18:18 +00:00
- Fixed: ksh crashed after unsetting .sh.match and then matching a pattern.
Fix [ \( str -a str \) ], [ \( str -o str \) ] Symptoms: $ test \( string1 -a string2 \) /usr/local/bin/ksh: test: argument expected $ test \( string1 -o string2 \) /usr/local/bin/ksh: test: argument expected The parentheses should be irrelevant and this should be a test for the non-emptiness of string1 and/or string2. src/cmd/ksh93/bltins/test.c: - b_test(): There is a block where the case of 'test' with five or less arguments, the first and last one being parentheses, is special-cased. The parentheses are removed as a workaround: argv is increased to skip the opening parenthesis and argc is decreased by 2. However, there is no corresponding increase of tdata.av which is a copy of this function's argv. This renders the workaround ineffective. The fix is to add that increase. - e3(): Do not handle '!' as a negator if not followed by an argument. This allows a right-hand expression that is equal to '!' (i.e. a test for the non-emptiness of the string '!').
2021-11-14 17:27:05 +00:00
- Another test/[ fix: "test \( string1 -a string2 \)" and "test \( string1 -o
string2 \)" no longer give an incorrect "argument expected" error message.
test/[: Fix binary -a/-o operators in POSIX mode POSIX requires test "$a" -a "$b" to return true if both $a and $b are non-empty, and test "$a" -o "$b" to return true if either $a or $b is non-empty. In ksh, this fails if "$a" is '!' or '(' as this causes ksh to interpret the -a and -o as unary operators (-a being a file existence test like -e, and -o being a shell option test). $ test ! -a ""; echo "$?" 0 (expected: 1/false) $ set -o trackall; test ! -o trackall; echo "$?" 1 (expected: 0/true) $ test \( -a \); echo "$?" ksh: test: argument expected 2 (expected: 0/true) $ test \( -o \) ksh: test: argument expected 2 (expected: 0/true) Unfortunately this problem cannot be fixed without risking breakage in legacy scripts. For instance, a script may well use test ! -a filename to check that a filename is nonexistent. POSIX specifies that this always return true as it is a test for the non-emptiness of both strings '!' and 'filename'. So this commit fixes it for POSIX mode only. src/cmd/ksh93/bltins/test.c: e3(): - If the posix option is active, specially handle the case of having at least three arguments with the second being -a or -o, overriding their handling as unary operators. src/cmd/ksh93/data/testops.c: - Update 'test --man --' date and say that unary -a is deprecated. src/cmd/ksh93/sh.1: - Document the fix under the -o posix option. - For test/[, explain that binary -a/-o are deprecated. src/cmd/ksh93/tests/bracket.sh: - Add tests based on reproducers in bug report. Resolves: https://github.com/ksh93/ksh/issues/330
2021-11-13 02:17:54 +00:00
2021-11-13:
test/[: binary operators: fix '<' and add '=~'; some more cleanups In ksh88, the test/[ built-in supported both the '<' and '>' lexical sorting comparison operators, same as in [[. However, in every version of ksh93, '<' does not work though '>' still does! Still, the code for both is present in test_binop(): src/cmd/ksh93/bltins/test.c 548: case TEST_SGT: 549: return(strcoll(left, right)>0); 550: case TEST_SLT: 551: return(strcoll(left, right)<0); Analysis: The binary operators are looked up in shtab_testops[] in data/testops.c using a macro called sh_lookup, which expands to a sh_locate() call. If we examine that function in sh/string.c, it's easy to see that on systems using ASCII (i.e. all except IBM mainframes), it assumes the table is sorted in ASCII order. src/cmd/ksh93/sh/string.c 64: while((c= *tp->sh_name) && (CC_NATIVE!=CC_ASCII || c <= first)) The problem was that the '<' operator was not correctly sorted in shtab_testops[]; it was sorted immediately before '>', but after '='. The ASCII order is: < (60), = (61), > (62). This caused '<' to never be found in the table. The test_binop() function is also used by [[, yet '<' always worked in that. This is because the parser has code that directly checks for '<' and '>' within [[ (in sh/parse.c, lines 1949-1952). This commit also adds '=~' to 'test', which took three lines of code and allowed eliminating error handling in test_binop() as test/[ and [[ now support the same binary ops. (re: fc2d5a60) src/cmd/ksh93/*/*.[ch]: - Rename a couple of very misleadingly named macros in test.h: . For == and !=, the TEST_PATTERN bit is off for pattern compares and on for literal string compares! Rename to TEST_STRCMP. . The TEST_BINOP bit does not denote all binary operators, but only the logical -a/-o ops in test/[. Rename to TEST_ANDOR. src/cmd/ksh93/bltins/test.c: test_binop(): - Add support for =~. This is only used by test/[. The method is implemented in two lines that convert the ERE to a shell pattern by prefixing it with ~(E), then call test_strmatch with that temporary string to match the ERE and update ${.sh.match}. - Since all binary ops from shtab_testops[] are now accounted for, remove unknown op error handling from this function. src/cmd/ksh93/data/testops.c: - shtab_testops[]: . Correctly sort the '<' (TEST_SLT) entry. . Remove ']]' (TEST_END). It's not an op and doesn't belong here. - Update sh_opttest[] documentation with =~, \<, \>. - Remove now-unused e_unsupported_op[] error message. src/cmd/ksh93/sh/lex.c: sh_lex(): - Check for ']]' directly instead of relying on the removed TEST_END entry from shtab_testops[]. src/cmd/ksh93/tests/bracket.sh: - Add relevant tests. src/cmd/ksh93/tests/builtins.sh: - Fix an old test that globally deleted the 'test' builtin. Delete it within the command substitution subshell only. - Remove the test for non-support of =~ in test/[. - Update the test for invalid test/[ op to use test directly.
2021-11-13 22:57:15 +00:00
- The test/[ built-in command now supports the '<' and '=~' operators from [[.
As of now, test/[ supports the same operators as [[ except for the different
and/or operators. Note: test/[ remains deprecated due to its many pitfalls.
test/[: Fix binary -a/-o operators in POSIX mode POSIX requires test "$a" -a "$b" to return true if both $a and $b are non-empty, and test "$a" -o "$b" to return true if either $a or $b is non-empty. In ksh, this fails if "$a" is '!' or '(' as this causes ksh to interpret the -a and -o as unary operators (-a being a file existence test like -e, and -o being a shell option test). $ test ! -a ""; echo "$?" 0 (expected: 1/false) $ set -o trackall; test ! -o trackall; echo "$?" 1 (expected: 0/true) $ test \( -a \); echo "$?" ksh: test: argument expected 2 (expected: 0/true) $ test \( -o \) ksh: test: argument expected 2 (expected: 0/true) Unfortunately this problem cannot be fixed without risking breakage in legacy scripts. For instance, a script may well use test ! -a filename to check that a filename is nonexistent. POSIX specifies that this always return true as it is a test for the non-emptiness of both strings '!' and 'filename'. So this commit fixes it for POSIX mode only. src/cmd/ksh93/bltins/test.c: e3(): - If the posix option is active, specially handle the case of having at least three arguments with the second being -a or -o, overriding their handling as unary operators. src/cmd/ksh93/data/testops.c: - Update 'test --man --' date and say that unary -a is deprecated. src/cmd/ksh93/sh.1: - Document the fix under the -o posix option. - For test/[, explain that binary -a/-o are deprecated. src/cmd/ksh93/tests/bracket.sh: - Add tests based on reproducers in bug report. Resolves: https://github.com/ksh93/ksh/issues/330
2021-11-13 02:17:54 +00:00
- The test/[ built-in command is fixed so that the binary -a (and) and -o (or)
operators, as in [ "$a" -a "$b" ] or [ "$a" -o "$b" ], work even if "$a" is
'!' or '('. To avoid breaking backwards compatibility with the nonstandard
unary [ -a "$file" ] and [ -o "$option" ] operators in combination with '!'
or parentheses, this fix is only activated if the posix option is on.
Fix bug on closed stdout; improve BUG_PUTIOERR fix (re: 93e15a30) Stéphane Chazelas reported: > As noted in this austin-group-l discussion[*] (relevant to this > issue): > > $ ksh93u+m -c 'pwd; echo "$?" >&2; echo test; echo "$?" >&2' >&- > 0 > 1 > /home/chazelas > > when stdout is closed, pwd does claim it succeeds (by returning a > 0 exit status), while echo doesn't (not really relevant to the > problem here, only to show it doesn't affect all builtins), and > the output that pwd failed to write earlier ends up being written > on stderr here instead of stdout upon exit (presumably) because > of that >&2 redirection. > > strace shows ksh93 attempting write(1, "/home/chazelas\n", 15) 6 > times (1, the last one, successful). > > It gets even weirder when redirecting to a file: > > $ ksh93u+m -c 'pwd; echo "$?" >&2; echo test; echo "$?" > file' >&- > 0 > $ cat file > 1 > 1 > ome/chazelas In my testing, the problem does not occur when closing stdout at the start of the -c script itself (using redirect >&- or exec >&-); it only occurs if stdout was closed before initialising the shell. That made me suspect that the problem had to do with an inconsistent file descriptor state in the shell. ksh uses internal sh_open() and sh_close() functions, among others, to maintain that state. src/cmd/ksh93/sh/main.c: sh_main(): - If the shell is initialised with stdin, stdout or stderr closed, then make the shell's file descriptor state tables reflect that fact by calling sh_close() for the closed file descriptors. This commit also improves the BUG_PUTIOERR fix from 93e15a30. Error checking after sfsync() is not sufficient. For instance, on FreeBSD, the following did not produce a non-zero exit status: ksh -c 'echo hi' >/dev/full even though this did: ksh -c 'echo hi >/dev/full' Reliable error checking requires not only checking the result of every SFIO command that writes output, but also synching the buffer at the end of the operation and checking the result of that. src/cmd/ksh93/bltins/print.c: - Make exitval variable global to allow functions called by b_print() to set a nonzero exit status. - Check the result of all SFIO output commands that write output. - b_print(): Always sfsync() at the end, except if the s (history) flag was given. This allows getting rid of the sfsync() call that required the workaround introduced in 846ad932. [*] https://www.mail-archive.com/austin-group-l@opengroup.org/msg08056.html Resolves: https://github.com/ksh93/ksh/issues/314
2021-11-07 10:14:54 +00:00
2021-11-07:
- Fixed a bug that could corrupt output if standard output is closed upon
initializing the shell.
- Improved BUG_PUTIOERR fix (2020-05-14) with more error checking. On
systems with the "disk full" error testing device /dev/full, an
echo/print/printf to /dev/full now always yields a non-zero exit status.
Fix octal number arguments in printf integer arithmetic Bug 1: POSIX requires numbers used as arguments for all the %d, %u... in printf to be interpreted as in the C language, so printf '%d\n' 010 should output 8 when the posix option is on. However, it outputs 10. This bug was introduced as a side effect of a change introduced in the 2012-02-07 version of ksh 93u+m, which caused the recognition of leading-zero numbers as octal in arithmetic expressions to be disabled outside ((...)) and $((...)). However, POSIX requires leading-zero octal numbers to be recognised for printf, too. The change in question introduced a sh.arith flag that is set while we're processing a POSIX arithmetic expression, i.e., one that recognises leading-zero octal numbers. Bug 2: Said flag is not reset in a command substitution used within an arithmetic expression. A command substitution should be a completely new context, so the following should both output 10: $ ksh -c 'integer x; x=010; echo $x' 10 # ok; it's outside ((…)) so octals are not recognised $ ksh -c 'echo $(( $(integer x; x=010; echo $x) ))' 8 # bad; $(comsub) should create new non-((…)) context src/cmd/ksh93/bltins/print.c: extend(): - For the u, d, i, o, x, and X conversion modifiers, set the POSIX arithmetic context flag before calling sh_strnum() to convert the argument. This fixes bug 1. src/cmd/ksh93/sh/subshell.c: sh_subshell(): - When invoking a command substitution, save and unset the POSIX arithmetic context flag. Restore it at the end. This fixes bug 2. Reported-by: @stephane-chazelas Resolves: https://github.com/ksh93/ksh/issues/326
2021-09-13 02:23:50 +00:00
2021-09-13:
- Disable the POSIX arithmetic context while running a command substitution
invoked from within an arithmetic expression. This fixes a bug that caused
integer arguments with a leading zero to be incorrectly interpreted as octal
numbers in non-POSIX arithmetic contexts within such command substitutions.
Fix 'ps' output for hashbangless scripts on Linux/macOS When invoking a script without an interpreter (#!hashbang) path, ksh forks, but there is no exec syscall in the child. The existing command line is overwritten in fixargs() with the name of the new script and associated arguments. In the generic/fallback version of fixargs() which is used on Linux and macOS, if the new command line is longer than the existing one, it is truncated. This works well when calling a script with a shorter name. However, it generates a misleading name in the common scenario where a script is invoked from an interactive shell, which typically has a short command line. For instance, if "/tmp/script" is invoked, "ksh" gets replaced with "/tm" in "ps" output. A solution is found in the fact that, on these systems, the environment is stored immediately after the command line arguments. This space can be made available for use by a longer command line by moving the environment strings out of the way. src/cmd/ksh93/sh/main.c: fixargs(): - Refactor BSD setproctitle(3) version to be more self-contained. - In the generic (Linux/macOS) version, on init (i.e. mode==0), if the command line is smaller than 128 bytes and the environment strings have not yet been moved (i.e. if they still immediately follow the command line arguments in memory), then strdup the environment strings, pointing the *environment[] members to the new strings and adding the length of the strings to the maximum command line buffer size. Reported-by: @gkamat Resolves: https://github.com/ksh93/ksh/pull/300
2021-09-12 03:16:51 +00:00
2021-09-12:
- When invoking a script without an interpreter/hashbang path on Linux and
macOS, ksh can now update 'ps' output to show longer command lines.
Fix `backtick` comsubs by making them act like $(modern) ones ksh93 currently has three command substitution mechanisms: - type 1: old-style backtick comsubs that use a pipe; - type 3: $(modern) comsubs that use a temp file, currently with fallback to a pipe if a temp file cannot be created; - type 2: ${ shared-state; } comsubs; same as type 3, but shares state with parent environment. Type 1 is buggy. There are at least two reproducers that make it hang. The Red Hat patch applied in 4ce486a7 fixed a hang in backtick comsubs but reintroduced another hang that was fixed in ksh 93v-. So far, no one has succeeded in making pipe-based comsubs work properly. But, modern (type 3) comsubs use temp files. How does it make any sense to have two different command substitution mechanisms at the execution level? The specified functionality between backtick and modern command substitutions is exactly the same; the difference *should* be purely syntactic. So this commit removes the type 1 comsub code at the execution level, treating them all like type 3 (or 2). As a result, the related bugs vanish while the regression tests all pass. The only side effect that I can find is that the behaviour of bug https://github.com/ksh93/ksh/issues/124 changes for backtick comsubs. But it's broken either way, so that's neutral. So this commit can now be added to my growing list of ksh93 issues fixed by simply removing code. src/cmd/ksh93/sh/xec.c: - Remove special code for type 1 comsubs from iousepipe(), sh_iounpipe(), sh_exec() and _sh_fork(). src/cmd/ksh93/include/defs.h, src/cmd/ksh93/sh/subshell.c: - Remove pipe support from sh_subtmpfile(). This also removes the use of a pipe as a fallback for $(modern) comsubs. Instead, panic and error out if temp file creation fails. If the shell cannot create a temporary file, there are fatal system problems anyway and a script should not continue. - No longer pass comsub type to sh_subtmpfile(). All other changes: - Update sh_subtmpfile() calls. src/cmd/ksh93/tests/subshell.sh: - Add two regression tests based on reproducers from bug reports. Resolves: https://github.com/ksh93/ksh/issues/305 Resolves: https://github.com/ksh93/ksh/issues/316
2021-08-13 06:01:46 +00:00
2021-08-13:
- An issue was fixed that could cause old-style `backtick` command
substitutions to hang in certain cases.
[[ ... ]]: fix '!' to negate another '!' Bug: [[ ! ! 1 -eq 1 ]] returns false, but should return true. This bug was reported for bash, but ksh has it too: https://lists.gnu.org/archive/html/bug-bash/2021-06/msg00006.html Op 24-05-21 om 17:47 schreef Chet Ramey: > On 5/22/21 2:45 PM, Vincent Menegaux wrote: >> Previously, these commands: >> >> [[ ! 1 -eq 1 ]]; echo $? >> [[ ! ! 1 -eq 1 ]]; echo $? >> >> would both result in `1', since parsing `!' set CMD_INVERT_RETURN >> instead of toggling it. > > Interestingly, ksh93 produces the same result as bash. I agree > that it's more intuitive to toggle it. Also interesting is that '!' as an argument to the simple 'test'/'[' command does work as expected (on both bash and ksh93): 'test ! ! 1 -eq 1' and '[ ! ! 1 -eq 1 ]' return 0/true. Even the man page for [[ is identical for bash and ksh93: | ! expression | True if expression is false. This suggests it's supposed to be a logical negation operator, i.e. '!' is implicitly documented to negate another '!'. Bolsky & Korn's 1995 ksh book, p. 167, is slightly more explicit about it: "! test-expression. Logical negation of test-expression." I also note that multiple '!' negators in '[[' work as expected on mksh, yash and zsh. src/cmd/ksh93/sh/parse.c: test_primary(): - Fix bitwise logic for '!': xor the TNEGATE bit into tretyp instead of or'ing it, which has the effect of toggling it.
2021-06-03 13:47:25 +00:00
2021-06-03:
- Fixed a bug in the [[ compound command: the '!' logical negation operator
now correctly negates another '!', e.g., [[ ! ! 1 -eq 1 ]] now returns
0/true. Note that this has always been the case for 'test'/'['.
Decrease SHLVL before doing 'exec' from main shell Problem: $ exec ksh $ echo $SHLVL 2 $ exec ksh $ echo $SHLVL 3 $ exec ksh $ echo $SHLVL 4 ...etc. SHLVL is supposed to acount the number of shell processes that you need to exit before you get logged out. Since ksh was replacing itself with a new shell in the same process using 'exec', SHLVL should not increase. src/cmd/ksh93/bltins/misc.c: b_exec(): - When about to replace the shell and we're not in a subshell, decrease SHLVL to cancel out a subsequent increase by the replacing shell. Bash and zsh also do this.
2021-05-18 22:02:49 +00:00
2021-05-18:
- Fixed SHLVL so that replacing ksh by itself (exec ksh) will not increase it.
Fix shell exit on function call redirection error (re: 23f2e23) This regression also exists on ksh 93v- and ksh2020, from which it was backported. Reproducer: $ (fn() { true; }; fn >/dev/null/ne; true) 2>/dev/null; echo $? 1 Expected output: 0 (as on ksh 93u+). FreeBSD sh and NetBSD sh are the only other known shells that share this behaviour. POSIX currently allows both behaviours, but may require the ksh 93u+ behaviour in future. In any case, this causes an incompatibility with established ksh behaviour that could easily break existing ksh scripts. src/cmd/ksh93/sh/xec.c: sh_exec(): - Commit 23f2e23 introduced a check for jmpval > SH_JMPIO (5). When a function call pushes context for a redirection, this is done with the jmpval exit value of SH_JMPCMD (6). Change that to SH_JMPIO to avoid triggering that check. src/cmd/ksh93/tests/exit.sh: - Add regression tests for exit behaviour on various kinds of shell errors as listed in the POSIX standard, including an error in a redirection of a function call. Fixes: https://github.com/ksh93/ksh/issues/310
2021-05-19 04:53:56 +00:00
- Fixed a regression introduced on 2020-08-05 that caused a non-interactive
shell to exit if an I/O redirection of a function call encountered an error.
Re-fix 'test -t 1' in command substitutions (re: 090b65e7) Since a command substitution no longer forks on non-permanently redirecting standard output within it for a specific command, test -t 1, [ -t 1 ], and [[ -t 1 ]] broke as follows: v=$(test -t 1 >/dev/tty && echo ok) did not assign 'ok' to v. This is because the assumption in tty_check() that standard output is never on a terminal in a non-forked command substitution, added in 55f0f8ce, was made invalid by 090b65e7. src/cmd/ksh93/edit/edit.c: tty_check(): - Implement a new method. Return false if the file descriptor stream is of type SF_STRING, which is the case for non-forked command substitutions -- it means the sfio stream writes directly into a memory area. This can be checked with the sfset(3) function (see src/lib/libast/man/sfio.3). To avoid a segfault when accessing sh.sftable, we need to validate the FD first. src/cmd/ksh93/tests/pty.sh: - Add the above reproducer.
2021-05-13 07:30:28 +00:00
2021-05-13:
- Fixed a bug with 'test -t 1' that was introduced on 2021-04-26:
v=$(test -t 1 >/dev/tty && echo ok) did not assign 'ok' to v.
Release 1.0.0-beta.1 In May 2020, when every KornShell (ksh93) development project was abandoned, development was rebooted in a new fork based on the last stable AT&T version: ksh 93u+. Now, one year and hundreds of bug fixes later, the first beta version is ready, and KornShell lives again. This new fork is called ksh 93u+m as a permanent nod to its origin; a standard semantic version number is added starting at 1.0.0-beta.1. Please test the beta and report any bugs you find, or help us fix known bugs.
2021-05-10 16:40:44 +00:00
2021-05-10:
- Release 1.0.0-beta.1.
Back port ksh93v- float, int, and exp10 changes from math.tab (#299) src/cmd/ksh93/data/math.tab: - Added exp10(). - Remove int() as being an alias to floor(). - Created entries for local float() and local int() which are defined in features/math.sh. src/cmd/ksh93/features/math.sh: - Backport floor() and int() related code from ksh93v-. src/cmd/ksh93/sh.1: - Sync man page to math.tab's potential functions.
2021-05-08 03:43:37 +00:00
2021-05-07:
- Backported three ksh 93v- math.tab changes, allowing for an exp10()
arithmetic function if one exists in the C library, a new float()
function, and lastly an updated int() function that rounds to zero
instead of being an alias to floor().
Fix two more 'command' bugs BUG 1: Though 'command' is specified/documented as a regular builtin, preceding assignments survive the invocation (as with special or declaration builtins) if 'command' has no command arguments in these cases: $ foo=wrong1 command; echo $foo wrong1 $ foo=wrong2 command -p; echo $foo wrong2 $ foo=wrong3 command -x; echo $foo wrong3 Analysis: sh_exec(), case TCOM (simple command), contains the following loop that skips over 'command' prefixes, preparsing any options and remembering the offset in the 'command' variable: src/cmd/ksh93/sh/xec.c 1059 while(np==SYSCOMMAND || !np && com0 && nv_search(com0,shp->fun_tree,0)==SYSCOMMAND) 1060 { 1061 register int n = b_command(0,com,&shp->bltindata); 1062 if(n==0) 1063 break; 1064 command += n; 1065 np = 0; 1066 if(!(com0= *(com+=n))) 1067 break; 1068 np = nv_bfsearch(com0, shp->bltin_tree, &nq, &cp); 1069 } This skipping is not done if the preliminary b_command() call on line 1061 (with argc==0) returns zero. This is currently the case for command -v/-V, so that 'command' is treated as a plain and regular builtin for those options. The cause of the bug is that this skipping is even done if 'command' has no arguments. So something like 'foo=bar command' is treated as simply 'foo=bar', which of course survives. So the fix is for b_command() to return zero if there are no arguments. Then b_command() itself needs changing to not error out on the second/main b_command() call if there are no arguments. src/cmd/ksh93/bltins/whence.c: b_command(): - When called with argc==0, return a zero offset not just for -v (X_FLAG) or -V (V_FLAG), but also if there are no arguments left (!*argv) after parsing options. - When called with argc>0, do not issue a usage error if there are no arguments, but instead return status 0 (or, if -v/-V was given, status 2 which was the status of the previous usage message). This way, 'command -v $emptyvar' now also works as you'd expect. BUG 2: 'command -p' sometimes failed after executing certain loops. src/cmd/ksh93/sh/path.c: defpath_init(): - astconf() returns a pointer to memory that may be overwritten later, so duplicate the string returned. Backported from ksh2020. (re: f485fe0f, aa4669ad, <https://github.com/att/ast/issues/959>) src/cmd/ksh93/tests/builtins.sh: - Update the test for BUG_CMDSPASGN to check every variant of 'command' (all options and none; invoking/querying all kinds of command and none) with a preceding assignment. (re: fae8862c) This also covers bug 2 as 'command -p' was failing on macOS prior to the fix due to a loop executed earlier in another test.
2021-05-05 01:42:45 +00:00
2021-05-05:
- Fixed: a preceding variable assignment like foo=bar in 'foo=bar command'
(with no command arguments after 'command') incorrectly survived the
'command' regular built-in command invocation.
- Fixed: 'command -p some_utility' intermittently failed to find the utility
under certain conditions due to a memory corruption issue.
Fix remaining bug in ${var:-'{}'} (re: d087b031) The following problems remained: $ var=x; echo ${var:-'{}'} x} $ var=; echo ${var:+'{}'} } src/cmd/ksh93/sh/macro.c: varsub(): - Use the new ST_MOD1 state table to skip over ${var-'foo'}, etc. instead of ST_QUOTE. In ST_MOD1 the ' is categorised as S_LIT which causes the single quotes to be skipped over correctly. See d087b031 for more info. src/cmd/ksh93/tests/quoting2.sh: - Add tests for this remaining bug. - Make the new test xtrace-proof. Resolves: https://github.com/ksh93/ksh/issues/290 (again)
2021-05-03 01:59:09 +00:00
2021-05-03:
Fix $RANDOM to act consistently in subshells (#294) This fixes the following: 1. Using $RANDOM in a virtual/non-forked subshell no longer influences the reproducible $RANDOM sequence in the parent environment. 2. When invoking a subshell $RANDOM is now re-seeded (as mksh and bash do) so that invocations in repeated subshells (including forked subshells) longer produce identical sequences by default. 3. Program flow corruption that occurred in scripts on executing ( ( simple_command & ) ). src/cmd/ksh93/include/variables.h: - Move 'struct rand' here as it will be needed in subshell.c. Add rand_seed member to save the pseudorandom generator seed. Remove the pointer to the shell state as it's redundant. src/cmd/ksh93/sh/init.c: - put_rand(): Store given seed in rand_seed while calling srand(). No longer pointlessly limit the number of possible seeds with the RANDMASK bitmask (that mask is to limit the values to 0-32767, it should not limit the number of possible sequences to 32768). - nget_rand(): Instead of using rand(), use rand_r() to update the random_seed value. This makes it possible to save/restore the current seed of the pseudorandom generator. - Add sh_reseed_rand() function that reseeds the pseudorandom generator by calling srand() with a bitwise-xor combination of the current PID, the current time with a granularity of 1/10000 seconds, and a sequence number that is increased on each invocation. - nv_init(): Set the initial seed using sh_reseed_rand() here instead of in sh_main(), as this is where the other struct rand members are initialised. src/cmd/ksh93/sh/main.c: sh_main(): - Remove the srand() call that was replaced by the sh_reseed_rand() call in init.c. src/cmd/ksh93/sh/subshell.c: sh_subshell(): - Upon entering a virtual subshell, save the current $RANDOM seed and state, then reseed $RANDOM for the subshell. - Upon exiting a virtual subshell, restore $RANDOM seed and state and reseed the generator using srand() with the restored seed. src/cmd/ksh93/sh/xec.c: sh_exec(): - When optimizing out a subshell that is the last command, still act like a subshell: reseed $RANDOM and increase ${.sh.subshell}. - Fix a separate bug discovered while implementing this. Do not optimize '( simple_command & )' when in a virtual subshell; doing this causes program flow corruption. - When optimizing '( simple_command & )', also reseed $RANDOM and increment ${.sh.subshell}. src/cmd/ksh93/tests/subshell.sh, src/cmd/ksh93/tests/variables.sh: - Add various tests for all of the above. Co-authored-by: Johnothan King <johnothanking@protonmail.com> Resolves: https://github.com/ksh93/ksh/issues/285
2021-05-03 03:03:46 +00:00
- Subshells (even if non-forked) now keep a properly separated state of the
pseudorandom generator used for $RANDOM, so that using $RANDOM in a
non-forked subshell no longer influences a reproducible $RANDOM sequence in
the parent environment. In addition, upon invoking a subshell, $RANDOM is now
reseeded (as mksh and bash do).
- Fixed program flow corruption that occurred in scripts on executing a
background job in a nested subshell, as in ( ( simple_command & ) ).
Fix remaining bug in ${var:-'{}'} (re: d087b031) The following problems remained: $ var=x; echo ${var:-'{}'} x} $ var=; echo ${var:+'{}'} } src/cmd/ksh93/sh/macro.c: varsub(): - Use the new ST_MOD1 state table to skip over ${var-'foo'}, etc. instead of ST_QUOTE. In ST_MOD1 the ' is categorised as S_LIT which causes the single quotes to be skipped over correctly. See d087b031 for more info. src/cmd/ksh93/tests/quoting2.sh: - Add tests for this remaining bug. - Make the new test xtrace-proof. Resolves: https://github.com/ksh93/ksh/issues/290 (again)
2021-05-03 01:59:09 +00:00
- Completed the 2021-04-30 fix for ${var<OP>'{}'} where <OP> is '-', '+',
':-' or ':+' by fixing a bug that caused an extra '}' to be output.
POSIX: 'command': don't disable declaration proprts (re: b9d10c5a) Following the resolution of Austin Group bug 1393[*] that is set to be included in the next version of the POSIX standard, the 'command' prefix in POSIX mode (set -o posix) no longer disables the declaration properties of declaration built-ins. [*] https://austingroupbugs.net/view.php?id=1393 src/cmd/ksh93/sh/parse.c: lex(): - Skip the 'command' prefix even in POSIX mode so that any declaration commands prefixed by it are treated as such in xec.c (sh_exec()). src/cmd/ksh93/sh/xec.c: sh_exec(): - The foregoing change reintroduced a variant of BUG_CMDSPEXIT: the shell exits on something like 'command export readonlyvar=foo'. This now fixes that bug for both POSIX and non-POSIX mode. When calling nv_setlist() to process true shell assignments, and there is a 'command' prefix, push a shell context and use sigsetjmp to intercept any errors in assignments and stop the shell exiting. src/cmd/ksh93/tests/builtins.sh: - Borrow the BUG_CMDSPEXIT regression test from modernish and adapt it for ksh. (I'm the author so yes, I can do this.) Original: https://github.com/modernish/modernish/blob/ae8fe9c3/lib/modernish/tst/builtin.t#L80-L109
2021-05-03 23:34:06 +00:00
- Following the resolution of Austin Group bug 1393[*] that is set to be
included in the next version of the POSIX standard, the 'command' prefix
in POSIX mode (set -o posix) no longer disables the declaration properties
of declaration built-ins. This reverts a change introduced on 2020-09-11.
[*] https://austingroupbugs.net/view.php?id=1393
Various minor capitalization and typo fixes (#371) This commit fixes various minor typos, punctuation errors and corrects the capitalization of many names.
2021-12-12 21:40:38 +00:00
- Fixed arithmetic assignment operations for multidimensional indexed arrays.
Fix arithmetic assignment operations for multidimensional indexed arrays (#296) This PR corrects #168 for indexed arrays having more than one level. Turns out ksh was only keeping track of the subscript number for assignment in lvalue's nosub variable. By saving the actual subscript reference, the result can be assigned to its proper destination instead of putting the result into the last looked value or subscript location. src/cmd/ksh93/include/streval.h: struct lval: - Create a new pointer named sub to hold the reference that nosub describes. src/cmd/ksh93/sh/arith.c: arith(): - Adjust LOOKUP: for lvalue ARITH_ASSIGNOP operations on indexed arrays to save the np of the destination subscript for later use. - Adjust ASSIGN: to act when lvalue's nosub > 0 which happens as the last step in the arithmetic parsing loop for assignment operations. Only indexed arrays will have a nosub value > 0. All others have a nosub of 0 unless they are involved in a unary operation (++, --) which sets nosub to -1. All said in the context of assignment operations like (( arr[0][1] += 1 )). src/cmd/ksh93/sh/streval.c: - Initialize the new sub pointer to 0. src/cmd/ksh93/tests/arrays2.sh: - Created a few multidimensional indexed array tests for assignment operations like += as an example. Resolves: https://github.com/ksh93/ksh/issues/168
2021-05-04 02:13:14 +00:00
Fix single quotes in expansion operator string (re: 5ed9ffd6) The referenced commit introduced the following bug: > The closing quote does not appear to be registering during the > parse of the following: > > echo ${var:+'{}'} > > Within a script, this will result in: > > syntax error at line 1: `'' unmatched src/cmd/ksh93/data/lexstates.c, src/cmd/ksh93/include/lexstates.h: - Add new ST_MOD1 state table that is a copy of ST_QUOTE, but adds a special meaning (ST_LIT) for the single quote (position 39). src/cmd/ksh93/sh/lex.c: sh_lex(): - For parameter expansion operators with old-style quoting (S_MOD1), use the new ST_MOD1 state table instead of ST_QUOTE. This causes single quotes within them to be processed properly. src/cmd/ksh93/tests/quoting2.sh: - Add tests. Thanks to @gkamat for the bug report. Resolves: https://github.com/ksh93/ksh/issues/290
2021-04-30 04:10:04 +00:00
2021-04-30:
Parse quotes when extracting words from command history (#291) This avoids splitting on quoted whitespace when extracting words from the command history using the emacs M-. or vi _ command. Example: if the prior command is $ ls Stairway\ To\ Heaven.mp3 then, M-. in Emacs editing mode (and _ in vi mode) now inserts Stairway\ To\ Heaven.mp3 instead of Heaven.mp3. The behavior is similar for 'Stairway To Heaven.mp3' and "Stairway To Heaven.mp3". src/cmd/ksh93/edit/history.c: hist_word(): - Skip over single-quoted and double-quoted strings and backslash-escaped characters. src/cmd/ksh93/tests/pty.sh: - Add regression test for this feature in vi mode. Since emacs and vi both use the same code for this, that should be good enough. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-04-30 18:45:50 +00:00
- The emacs 'ESC .' (M-.) and vi '_' commands now take shell quoting into
account when repeating a word from the previous command line. For example, if
the previous command is 'ls Stairway\ To\ Heaven.mp3', then they now insert
'Stairway\ To\ Heaven.mp3' instead of 'Heaven.mp3'. Thanks to Govind Kamat.
Fix single quotes in expansion operator string (re: 5ed9ffd6) The referenced commit introduced the following bug: > The closing quote does not appear to be registering during the > parse of the following: > > echo ${var:+'{}'} > > Within a script, this will result in: > > syntax error at line 1: `'' unmatched src/cmd/ksh93/data/lexstates.c, src/cmd/ksh93/include/lexstates.h: - Add new ST_MOD1 state table that is a copy of ST_QUOTE, but adds a special meaning (ST_LIT) for the single quote (position 39). src/cmd/ksh93/sh/lex.c: sh_lex(): - For parameter expansion operators with old-style quoting (S_MOD1), use the new ST_MOD1 state table instead of ST_QUOTE. This causes single quotes within them to be processed properly. src/cmd/ksh93/tests/quoting2.sh: - Add tests. Thanks to @gkamat for the bug report. Resolves: https://github.com/ksh93/ksh/issues/290
2021-04-30 04:10:04 +00:00
- Fixed a bug introduced on 2020-09-05 that caused "echo ${var:+'{}'}"
to be misparsed.
Fork before entering shared-state command substitution The code contains various checks to see if a subshell needs to fork, like this one in the ulimit builtin: if(shp->subshell && !shp->subshare) sh_subfork(); All checks of this form are fatally broken, as each one of them causes shared-state command substitutions to ignore parent virtual subshells. Currently the only feasible way to fix this is to fork a virtual subshell before executing a shared-state command substitution in it. In the long term I think shared-state command substitutions should probably be redesigned to disassociate them completely from the virtual subshell mechanism. src/cmd/ksh93/sh/macro.c: comsubst(): - If we're in a non-subshare virtual subshell, fork it before entering a type 2 (subshare) command substitution. src/cmd/ksh93/sh/subshell.c: - sh_assignok(): Remove subshare fix from 911d6b06 as it's redundant now that the parent of a subshare is never a virtual subshell. Go back to not doing anything if the current "subshell" is a subshare. - sh_subtracktree(), sh_subfuntree(): Similarly, remove the now-redundant subshare fixes from 13c57e4b. src/cmd/ksh93/sh/xec.c: sh_exec(): - Fix a separate bug: only fork a virtual subshell before running a background job if that "subshell" is not a subshare. src/cmd/ksh93/tests/subshell.sh: - Add test for bug fixed in xec.c. - Add tests for 'ulimit', 'builtin' and 'exec' run in subshare within subshell -- all commands that use checks of the form 'if(sh.subshell && !sh.subshare) sh_subfork();'. Resolves: https://github.com/ksh93/ksh/issues/289
2021-04-30 23:47:39 +00:00
- Fixed: the effects of 'builtin', 'exec' and 'ulimit' leaked out of a parent
virtual subshell if run from a ${ shared-state; } command substitution.
Fix fork after redirecting stdout in subshare (re: 500757d7) Previously, command substitutions executed as virtual subshells were always forked if any command was run within them that redireceted standard output, even if the redirection was local to that command. Commit 500757d7 removed the check for a shared-state command substitution (subshare), so introduced a bug where even that would fork, causing it to stop sharing its state. We can further improve on that fix by only forking if the redirection is permanent as with `exec` or `redirect`. There should be no need to do that if the redirection is local to a command run within the command substitution, as the file descriptor is restored when that command finishes, which is still within the command substitution. src/cmd/ksh93/sh/io.c: sh_redirect(): - Only fork upon redirecting stdout if the virtual subshell is a command substitution, and if the redirection is permanent (flag==1 or flag==2).
2021-04-26 17:09:36 +00:00
2021-04-26:
- Fixed a bug introduced on 2021-02-20 in which a shared-state command
substitution stopped sharing its state with the calling shell environment
Various minor capitalization and typo fixes (#371) This commit fixes various minor typos, punctuation errors and corrects the capitalization of many names.
2021-12-12 21:40:38 +00:00
if it executed a command that locally redirected standard output.
Fix fork after redirecting stdout in subshare (re: 500757d7) Previously, command substitutions executed as virtual subshells were always forked if any command was run within them that redireceted standard output, even if the redirection was local to that command. Commit 500757d7 removed the check for a shared-state command substitution (subshare), so introduced a bug where even that would fork, causing it to stop sharing its state. We can further improve on that fix by only forking if the redirection is permanent as with `exec` or `redirect`. There should be no need to do that if the redirection is local to a command run within the command substitution, as the file descriptor is restored when that command finishes, which is still within the command substitution. src/cmd/ksh93/sh/io.c: sh_redirect(): - Only fork upon redirecting stdout if the virtual subshell is a command substitution, and if the redirection is permanent (flag==1 or flag==2).
2021-04-26 17:09:36 +00:00
shcomp: fix redirection with process substitution The commands within a process substitution used as an argument to a redirection (e.g. < <(...) or > >(...)) are simply not included in parse trees dumped by shcomp. This can be verified with a command like hexdump -C. As a result, these process substitutions do not work when running a bytecode-compiled shell script. The fix is surprisingly simple. A process substitution is encoded as a complete parse tree. When used with a redirection, that parse tree is used as the file name for the redirection. All we need to do is treat the "file name" as a parse tree instead of a string if flags indicate a process substitution. A process substitution is detected by the struct ionod field 'iofile'. Checking the IOPROCSUB bit flag is not enough. We also need to exclude the IOLSEEK flag as that form of redirection may use the IOARITH flag which has the same bit value as IOPROCSUB (see include/shnodes.h). src/cmd/ksh93/sh/tdump.c: p_redirect(): - Call p_tree() instead of p_string() for a process substitution. src/cmd/ksh93/sh/trestore.c: r_redirect(): - Call r_tree() instead of r_string() for a process substitution. src/cmd/ksh93/include/version.h: - Bump the shcomp binary header version as this change is not backwards compatible; previous trestore.c versions don't know how to read the newly compiled process substitutions and would crash. src/cmd/ksh93/tests/io.sh: - Add test. src/cmd/ksh93/tests/builtins.sh, src/cmd/ksh93/tests/options.sh: - Revert shcomp workarounds. (re: 6701bb30) Resolves: https://github.com/ksh93/ksh/issues/165
2021-04-22 02:25:24 +00:00
2021-04-22:
- shcomp (the shell bytecode compiler) was fixed to correctly compile process
substitutions used as the file name to a redirection, as in 'cmd < <(cmd)'.
Fix LINENO after unsetting it a virtual subshell (#283) There is a TODO note in variables.sh that notes the value of LINENO is wrong after a virtual subshell. The following script should print '6', but the bug causes it to print '1' instead: $ cat /tmp/lineno #!/bin/ksh ( unset LINENO : ) echo $LINENO This bug started to occur after the bugfix applied in 7b994b6a. However, that commit is not where the cause of bug was (when that bugfix is applied to ksh versions 2008-07-25 through 2012-01-01, $LINENO works fine). Rather, the cause of this bug was introduced in 93u+ 2012-02-29. In that version, the mp->nvfun pointer was only copied from np->nvfun if the variable can be freed from memory. This is what caused 7b994b6a to break $LINENO in subshells, so to fix this bug the mp->nvfun and np->nvfun must point to the same object, even when the variable isn't freed from memory. src/cmd/ksh93/sh/subshell.c: nv_restore(): - Always copy the np->nvfun pointer to mp->nvfun. To prevent crashes, the value of np->nvfun->nofree is set to the value given by the nofree variable, which is set before _nv_unset. See also commit 7e7f1372, which fixed a crash that happened because _nv_unset discards the NV_NOFREE flag. src/cmd/ksh93/tests/variables.sh: - Remove the workaround for LINENO after a virtual subshell. - Add a regression test for the value of LINENO when unset in a virtual subshell, then used after the subshell. Note that before commit 997ad43b LINENO's value was corrupted after being unset in a subshell, so the test checks for corruption of the LINENO variable (in prior commits LINENO was set to '49' because of the previous bug).
2021-04-22 18:16:25 +00:00
- Fixed a bug introduced on 2020-07-13 that set LINENO to the wrong line
number after leaving a virtual subshell in which LINENO had been unset.
Fix ksh exit on syntax error in profile (re: cb67a01b, ceb77b13) Johnothan King writes: > There are two regressions related to how ksh handles syntax > errors in the .kshrc file. If ~/.kshrc or the file pointed to by > $ENV have a syntax error, ksh exits during startup. Additionally, > the error message printed is incorrect: > > $ cat /tmp/synerror > (( > echo foo > > # ksh93u+m > $ ENV=/tmp/synerror arch/*/bin/ksh -ic 'echo ${.sh.version}' > /tmp/synerror: syntax error: `/t/tmp/synerror' unmatched > > # ksh93u+ > $ ENV=/tmp/synerror ksh93u -ic 'echo ${.sh.version}' > /tmp/synerror: syntax error: `(' unmatched > Version AJM 93u+ 2012-08-01 > > The regression that causes the incorrect error message was > introduced by commit cb67a01. The other bug that causes ksh to > exit on startup was introduced by commit ceb77b1. src/cmd/ksh93/sh/lex.c: fmttoken(): - Call stakfreeze(0) to terminate a possible unterminated previous stack item before writing the token string onto the stack. This fixes the bug with garbage in a syntax error message. src/cmd/ksh93/sh/main.c: exfile(): - Revert Red Hat's ksh-20140801-diskfull.patch applied in ceb77b13. This fixes the bug with interactive ksh exiting on syntax error in a profile script. Testing by @JohnoKing showed the patch is no longer necessary to fix a login crash on disk full, as commit 970069a6 (which applied Red Hat patches ksh-20120801-macro.patch and ksh-20120801-fd2lost.patch) also fixes that crash. src/cmd/ksh93/README: - Fix typos. (re: fdc08b23) Co-authored-by: Johnothan King <johnothanking@protonmail.com> Resolves: https://github.com/ksh93/ksh/issues/281
2021-04-21 18:28:46 +00:00
2021-04-21:
- Fixed a bug introduced on 2020-09-28 that caused an interactive ksh to exit
if a profile script (such as ~/.kshrc) contains a syntax error.
Fix multiple problems with the getconf builtin (#280) This commit fixes three problems with getconf pathbound builtin: 1. The -l/--lowercase option did not change all variable names to lower case. 2. The -q/--quote option now quotes all string values. Previously, it only quoted string values that had a space or other non-shellsafe character. 3. The -c/--call, -n/--name and -s/--standard options matched all variable names provided by 'getconf -a', even if none were actual matches. Additionally, references to the confstr and sysconf functions have been updated to reference section 3 of the man pages instead of section 2. src/lib/libast/port/astconf.c: - Previously, only values that had spaces in them were quoted. Change that behavior to quote all string values by using the FMT_ALWAYS flag. Bug report: https://github.com/att/ast/issues/1173 - Not all variable names were printed in lowercase by 'getconf -l'. Fix it by adding a few missing instances of fmtlower. Bug report: https://github.com/att/ast/issues/1171 - Add the missing code to the '#if _pth_getconf_a' block to handle -c/-n/-s while parsing the OS's native 'getconf -a' output. This approach reuses code for name matching from other parts of astconflist(). Resolves: https://github.com/ksh93/ksh/issues/279 src/lib/libcmd/getconf.c: - Update the documentation to note the -q flag only quotes strings. src/cmd/ksh93/tests/bulitins.sh: - Add regression tests for the getconf bugs fixed in this commit. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-04-21 02:34:54 +00:00
2021-04-20:
- Fixed three problems with the /opt/ast/bin/getconf built-in command:
1. The -l/--lowercase option did not change all variable names to lower case.
2. The -q/--quote option now quotes all string values. Previously, it only
quoted string values that had a space or other non-shellsafe character.
3. The -c/--call, -n/--name and -s/--standard options matched all variable
names provided by 'getconf -a', even if none were actual matches.
Don't import/export readonly attribute via magic A__z env var While automagically importing/exporting ksh variable attributes via the environment is probably a misfeature in general (now disabled for POSIX standard mode), doing so with the readonly attribute is particularly problematic. Scripts can take into account the possibility of importing unwanted attributes by unsetting or typesetting variables before using them. But there is no way for a script to get rid of an unwanted imported readonly variable. This is a possible attack vector with no possible mitigation. This commit blocks both the import and the export of the readonly attribute through the environment. I consider it a security fix. src/cmd/ksh93/sh/init.c: env_import_attributes(): - Clear NV_RDONLY from imported attributes before applying them. src/cmd/ksh93/sh/name.c: sh_envgen(): - Remove NV_RDONLY from bitmask defining attributes to export.
2021-04-21 02:56:03 +00:00
- The readonly attribute of ksh variables is no longer imported from
or exported to other ksh shell instances through the environment.
emacs: Fix digits input after completion (re: 16e4824c, e8b3274a) Immediately after tab-completing the name of a directory, it is not possible to type digits after the slash; ksh eats them as it parses them as a menu selection for a nonexistent menu. Reproducer: $ mkdir -p emacstest/123abc $ cd emacste[tab]123abc Actual results: $ cd emacstest/abc Expected results: $ cd emacstest/123abc Workarounds are to press a non-numeric key followed by backspace, or hit [tab] again to get a list of options. Originally reported by Arnon Weinberg, 2012-12-23 07:15:19 UTC, at: https://bugzilla.redhat.com/889745 The fix had been partially backported from ksh 93v- by AT&T (16e4824c), which made things worse, so it was reverted (e8b3274a). This commit backports a slightly edited version of the complete fix. Thanks to @JohnoKing for finding the correct code. Discussion: https://github.com/ksh93/ksh/issues/198#issuecomment-820178514 src/cmd/ksh93/edit/emacs.c: escape(): - Backport the fix for this bug that was implemented in ksh 93v- alpha 2013-10-10. Immediately after a slash, do not stay in "\" mode (file name completion) and reset the tab count. src/cmd/ksh93/tests/pty.sh: - Test the fix. Resolves: https://github.com/ksh93/ksh/issues/198
2021-04-16 13:45:01 +00:00
2021-04-16:
- Fixed a bug in emacs mode: after using tab completion to complete the name
of a directory, it was not possible to type numbers after the slash.
Fix <>; redirection for final command exec optimization (#277) The <>; operator doesn't work correctly if it's used as the last command of a -c script. Reproducer: $ echo test > a; ksh -c 'echo x 1<>; a'; cat a x st This bug is caused by ksh running the last command of -c scripts with execve(2) instead of posix_spawn(3) or fork(2). The <>; operator is noted by the man page as being incompatible with the exec builtin (see also the ksh93u+ man page), so it's not surprising this bug occurs when ksh runs a command using execve: > <>;word cannot be used with the exec and redirect built-ins. The ksh2020 fix simply removed the code required for ksh to use this optimization at all. It's not a performance friendly fix and only papers over the bug, so this commit provides a better fix. This bug was first reported at: https://github.com/att/ast/issues/9 In addition, this commit re-enables the execve(2) optimization for the last command for scripts loaded from a file. It was enabled in in older ksh versions, and was only disabled in interactive shells: https://github.com/ksh93/ast-open-history/blob/2011-06-30/src/cmd/ksh93/sh/main.c#L593-L599 It was changed on 2011-12-24 to only be used for -c scripts: https://github.com/ksh93/ast-open-history/blob/2011-12-24/src/cmd/ksh93/sh/main.c#L593-L599 We think there is no good reason why scripts loaded from a file should be optimised less than scripts loaded from a -c argument. They're both scripts; there's no essential difference between them. So this commit reverts that change. If there is a bug left in the optimization after this fix, this revert increases the chance of exposing it so that it can be fixed. src/cmd/ksh93/sh/xec.c: - The IOREWRITE flag is set when handling the <>; operator, so to fix this bug, avoid exec'ing the last command if it uses <>;. See also commit 17ebfbf6, which fixed another issue related to the execve optimization. src/cmd/ksh93/tests/io.sh: - Enable a regression test that was failing because of this bug. - Add the reproducer from https://github.com/att/ast/issues/9 as a regression test. src/cmd/ksh93/sh/main.c: - Only avoid the non-forking optimization in interactive shells. src/cmd/ksh93/tests/signal.sh: - Add an extra comment to avoid the non-forking optimization in the regression test for rhbz#1469624. - If the regression test for rhbz#1469624 fails, show the incorrect exit status in the error message. src/cmd/ksh93/tests/builtins.sh, src/cmd/ksh93/tests/options.sh: - This bugfix was causing the options regression test to segfault when run under shcomp. The cause is the same as <https://github.com/ksh93/ksh/issues/165>, so as a workaround, avoid parsing process substitutions with shcomp until that is fixed. This workaround should also avoid the other problem detailed in <https://github.com/ksh93/ksh/issues/274>. Resolves: https://github.com/ksh93/ksh/issues/274
2021-04-15 17:29:50 +00:00
- Fixed an optimization bug that caused the <>; redirection operator to fail
Further fix '<>;' and fix crash on 32-bit systems (re: 6701bb30) Accessing t->tre.treio for every sh_exec() run is invalid because 't' is of type Shnode_t, which is a union that can contain many different kinds of structs. As all members of a union occupy the same address space, only one can be used at a time. Which member is valid to access depends on the node type sh_exec() was called with. The invalid access triggered a crash on 32-bit systems when executing an arithmetic command like ((x=1)). The t->tre.treio union member should be accessed for a simple command (case TCOM in sh_exec()). The fix is also needed for redirections attached to blocks (case TSETIO) in which case the union member to use is t->fork.forkio. src/cmd/ksh93/sh/xec.c: - Add check_exec_optimization() function that checks for all the conditions where the exec optimisation should not be done. For redirections we need to loop through the whole list to check for an IOREWRITE (<>;) one. - sh_exec(): case TCOM (simple command): Only bother to call check_exec_optimization() if there are either command arguments or redirections (IOW: don't bother for bare variable assignments), so move it to within the if(io||argn) block. - sh_exec(): case TSETIO: This needs a similar fix. To avoid the optimization breaking again if the last command is a subshell with a <>; redirection attached, we need to not only set execflg to 0 but also clear the SH_NOFORK state bit from the 'flags' variable which is passed on to the recursive sh_exec() call. src/cmd/ksh93/tests/io.sh: - Update and expand tests. Add tests for redirections attached to simple commands (TCOM) and various kinds of code block (TSETIO). Co-authored-by: Johnothan King <johnothanking@protonmail.com> Resolves: https://github.com/ksh93/ksh/issues/278
2021-04-17 20:31:38 +00:00
when used with the last command in a -c script.
Fix <>; redirection for final command exec optimization (#277) The <>; operator doesn't work correctly if it's used as the last command of a -c script. Reproducer: $ echo test > a; ksh -c 'echo x 1<>; a'; cat a x st This bug is caused by ksh running the last command of -c scripts with execve(2) instead of posix_spawn(3) or fork(2). The <>; operator is noted by the man page as being incompatible with the exec builtin (see also the ksh93u+ man page), so it's not surprising this bug occurs when ksh runs a command using execve: > <>;word cannot be used with the exec and redirect built-ins. The ksh2020 fix simply removed the code required for ksh to use this optimization at all. It's not a performance friendly fix and only papers over the bug, so this commit provides a better fix. This bug was first reported at: https://github.com/att/ast/issues/9 In addition, this commit re-enables the execve(2) optimization for the last command for scripts loaded from a file. It was enabled in in older ksh versions, and was only disabled in interactive shells: https://github.com/ksh93/ast-open-history/blob/2011-06-30/src/cmd/ksh93/sh/main.c#L593-L599 It was changed on 2011-12-24 to only be used for -c scripts: https://github.com/ksh93/ast-open-history/blob/2011-12-24/src/cmd/ksh93/sh/main.c#L593-L599 We think there is no good reason why scripts loaded from a file should be optimised less than scripts loaded from a -c argument. They're both scripts; there's no essential difference between them. So this commit reverts that change. If there is a bug left in the optimization after this fix, this revert increases the chance of exposing it so that it can be fixed. src/cmd/ksh93/sh/xec.c: - The IOREWRITE flag is set when handling the <>; operator, so to fix this bug, avoid exec'ing the last command if it uses <>;. See also commit 17ebfbf6, which fixed another issue related to the execve optimization. src/cmd/ksh93/tests/io.sh: - Enable a regression test that was failing because of this bug. - Add the reproducer from https://github.com/att/ast/issues/9 as a regression test. src/cmd/ksh93/sh/main.c: - Only avoid the non-forking optimization in interactive shells. src/cmd/ksh93/tests/signal.sh: - Add an extra comment to avoid the non-forking optimization in the regression test for rhbz#1469624. - If the regression test for rhbz#1469624 fails, show the incorrect exit status in the error message. src/cmd/ksh93/tests/builtins.sh, src/cmd/ksh93/tests/options.sh: - This bugfix was causing the options regression test to segfault when run under shcomp. The cause is the same as <https://github.com/ksh93/ksh/issues/165>, so as a workaround, avoid parsing process substitutions with shcomp until that is fixed. This workaround should also avoid the other problem detailed in <https://github.com/ksh93/ksh/issues/274>. Resolves: https://github.com/ksh93/ksh/issues/274
2021-04-15 17:29:50 +00:00
Allow invoking path-bound built-in commands by direct path or preceding `PATH` assignment (#275) Path-bound builtins on ksh (such as /opt/ast/bin/cat) break some basic assumptions about paths in the shell that should hold true, e.g., that a path output by whence -p or command -v should actually point to an executable command. This commit should fix the following: 1. Path-bound built-ins (such as /opt/ast/bin/cat) can now be executed by invoking the canonical path (independently of the value of $PATH), so the following will now work as expected: $ /opt/ast/bin/cat --version version cat (AT&T Research) 2012-05-31 $ (PATH=/opt/ast/bin:$PATH; "$(whence -p cat)" --version) version cat (AT&T Research) 2012-05-31 In the event an external command by that path exists, the path-bound builtin will now override it when invoked using the canonical path. To invoke a possible external command at that path, you can still use a non-canonical path, e.g.: /opt//ast/bin/cat or /opt/ast/./bin/cat 2. Path-bound built-ins will now also be found on a PATH set locally using an assignment preceding the command, so something like the following will now work as expected: $ PATH=/opt/ast/bin cat --version version cat (AT&T Research) 2012-05-31 The builtin is not found by sh_exec() because the search for builtins happens long before invocation-local preceding assignments are processsed. This only happens in sh_ntfork(), before forking, or in sh_fork(), after forking. Both sh_ntfork() and sh_fork() call path_spawn() to do the actual path search, so a check there will cover both cases. This does mean the builtin will be run in the forked child if sh_fork() is used (which is the case on interactive shells with job.jobcontrol set, or always after compiling with SHOPT_SPAWN disabled). Searching for it before forking would mean fundamentally redesigning that function to be basically like sh_ntfork(), so this is hard to avoid. src/cmd/ksh93/sh/path.c: path_spawn(): - Before doing anything else, check if the passed path appears in the builtins tree as a pathbound builtin. If so, run it. Since a builtin will only be found if a preceding PATH assignment temporarily changed the PATH, and that assignment is currently in effect, we can just sh_run() the builtin so a nested sh_exec() invocation will find and run it. - If 'spawn' is not set (i.e. we must return), set errno to 0 and return -2. See the change to sh_ntfork() below. src/cmd/ksh93/sh/xec.c: - sh_exec(): When searching for built-ins and the restricted option isn't active, also search bltin_tree for names beginning with a slash. - sh_ntfork(): Only throw an error if the PID value returned is exactly -1. This allows path_spawn() to return -2 after running a built-in to tell sh_ntfork() to do the right things to restore state. src/cmd/ksh93/sh/parse.c: simple(): - When searching for built-ins at parse time, only exclude names containing a slash if the restricted option is active. This allows finding pointers to built-ins invoked by literal path like /opt/ast/bin/cat, as long as that does not result from an expansion. This is not actually necessary as sh_exec() will also cover this case, but it is an optimisation. src/lib/libcmd/getconf.c: - Replace convoluted deferral to external command by a simple invocation of the path to the native getconf command determined at compile time (by src/lib/libast/comp/conf.sh). Based on: https://github.com/ksh93/ksh/issues/138#issuecomment-816384871 If there is ever a system that has /opt/ast/bin/getconf as its default native external 'getconf', then there would still be an infinite recursion crash, but this seems extremely unlikely. Resolves: https://github.com/ksh93/ksh/issues/138
2021-04-15 03:08:12 +00:00
2021-04-14:
- Path-bound built-ins (such as /opt/ast/bin/cat) can now be executed by
invoking the canonical path, so the following will now work as expected:
$ /opt/ast/bin/cat --version
version cat (AT&T Research) 2012-05-31
$ (PATH=/opt/ast/bin:$PATH; "$(whence -p cat)" --version)
version cat (AT&T Research) 2012-05-31
Non-canonical paths such as /opt/ast/./bin/cat will not find the built-ins.
- Path-bound built-ins will now also be found on a PATH set locally using an
assignment preceding the command, so the following will now work as expected:
$ PATH=/opt/ast/bin cat --version
version cat (AT&T Research) 2012-05-31
Fix the exit status returned when a command isn't executable (#273) Previous discussion: https://github.com/att/ast/issues/485 If ksh attempts to execute a non-executable command found in the PATH, in some instances the error message and return status are incorrect. In the example below, ksh returns with exit status 126 when using the -c execve(2) optimization or when using fork(2) in an interactive shell. However, using posix_spawn(3) causes the exit status to change: $ echo 'print cannot execute' > /tmp/x # Runs command with spawnveg (i.e., posix_spawn or vfork) $ ksh -c 'PATH=/tmp; x; echo $?' ksh: x: not found 127 # Runs command with execve $ ksh -c 'PATH=/tmp; x'; echo $? ksh: x: cannot execute [Permission denied] 126 # Runs command with fork $ ksh -ic 'PATH=/tmp; x; echo $?' ksh: x: cannot execute [Permission denied] 126 Since 'x' is in the PATH but can't be executed, the correct exit status is 126, not 127. It's worth noting this bug doesn't cause the regression tests to fail with ksh93u+m, but it does cause one test to fail when run under dtksh: path.sh[706]: Long nonexistent command name: got status 126, '' This commit backports various fixes for this bug from ksh2020, with additional fixes applied (since there were still some additional issues the ksh2020 patch didn't fix). The lacking regression test for exit status 126 in path.sh has been rewritten to test for more scenarios where ksh failed to return the correct error message and/or exit status. I can also confirm with this patch applied the path.sh regression tests now pass when run under dtksh. src/cmd/ksh93/sh/path.c: - Add a comment to path_absolute() describing 'oldpp' is the current pointer in the while loop and 'pp' is the next pointer. Backported from: https://github.com/att/ast/commit/a6cad450 - The patch from ksh2020 didn't fix this bug in the SHOPT_SPAWN code (because ksh2020 prefers fork(2)), so issues with the exit status could still occur when using spawnveg. To fix this, always set 'noexec' to the value of errno if can_execute fails. Before this fix, errno was discarded if 'pp' was a null pointer and can_execute failed. - If a command couldn't be executed and the error wasn't ENOENT, save errno in a 'not_executable' variable. If an executable command couldn't be found in the PATH, exit with status 126 and set errno to the saved value. This was based on a ksh2020 bugfix, but it has been reworked a little bit to fix a bug that caused a mismatch between the error message shown and errno. Example with a non-executable file in PATH: $ nonexec ksh2020: nonexec: cannot execute [No such file or directory] The ksh2020 patch: <https://github.com/att/ast/pull/493> - Backport a ksh2020 bugfix for directories in the PATH when running one of the added regression tests on OpenBSD: https://github.com/att/ast/pull/767 src/cmd/ksh93/data/msg.c, src/cmd/ksh93/include/shell.h, src/cmd/ksh93/sh/{path,xec}.c: - If a command name is too long (ENAMETOOLONG), then it wasn't found in the PATH. For that case return exit status 127, like for ENOENT. src/cmd/ksh93/tests/path.sh: - Replace the old test with a new set of more extensive tests. These tests check the error message and exit status when ksh attempts to run a command using any of the following: - execve(2), used with the last command run with -c (*A tests). - posix_spawn(3)/vfork(2), used in noninteractive scripts (*B tests). - fork(2), used in interactive shells with job control (*C tests). - command -x (*D tests). - exec(1) (*E tests). - Add a regression test from ksh2020 for attempting to execute a directory: https://github.com/att/ast/pull/758 src/lib/libast/include/ast.h, src/lib/libast/include/wait.h: - Avoid bitshifts in macros for static error codes. The return values of command not found and exec related errors are static values and should not require any macro magic for calculation. Backported from: https://github.com/att/ast/commit/c073b102 - Simplify EXIT_* and W* macros to use 8 bits.
2021-04-15 02:37:57 +00:00
2021-04-13:
- Fixed a few bugs that could cause ksh to show the wrong error message and/or
return the wrong exit status if a command couldn't be executed. In
scenarios where the command was found in the PATH but it was not executable,
ksh now returns with exit status 126. Otherwise, ksh will return with exit
status 127 (such as if the command isn't found or if the command name is
too long).
Correct memory fault when removing default nameref KSH_VERSION (#271) This commit fixes a segmentation fault when an attempt was made to unset the default KSH_VERSION variable prior any other nameref activity such as creating another nameref or even reassigning the nameref KSH_VERSION to something else. (new shell without prior nameref activity) $ nameref KSH_VERSION=.sh.version $ unset -n KSH_VERSION Memory fault src/cmd/ksh93/sh/name.c: _nv_unset(): - Add a 'Refdict' check before attempting to remove a value from it as apparently one does not exist until some sort of nameref activity occurs after shell startup as the default nameref of 'KSH_VERSION=.sh.version' does not create one.
2021-04-13 02:15:34 +00:00
2021-04-12:
- Corrected a memory fault when an attempt was made to unset the default
nameref KSH_VERSION from the shell environment prior to any other name
Lots of man page fixes and some other minor fixes (#284) Noteworthy changes: - The man pages have been updated to fix a ton of instances of runaway underlining (this was done with `sed -i 's/\\f5/\\f3/g'` commands). This commit dramatically increased in size because of this change. - The documentation for spawnveg(3) has been extended with information about its usage of posix_spawn(3) and vfork(2). - The documentation for tmfmt(3) has been updated with the changes previously made to the man pages for the printf and date builtins (though the latter builtin is disabled by default). - The shell's tracked alias tree (hash table) is now documented in the shell(3) man page. - Removed the commented out regression test for an ERRNO variable as the COMPATIBILITY file states it was removed in ksh93.
2021-04-23 21:02:30 +00:00
reference variable creation or modification.
Correct memory fault when removing default nameref KSH_VERSION (#271) This commit fixes a segmentation fault when an attempt was made to unset the default KSH_VERSION variable prior any other nameref activity such as creating another nameref or even reassigning the nameref KSH_VERSION to something else. (new shell without prior nameref activity) $ nameref KSH_VERSION=.sh.version $ unset -n KSH_VERSION Memory fault src/cmd/ksh93/sh/name.c: _nv_unset(): - Add a 'Refdict' check before attempting to remove a value from it as apparently one does not exist until some sort of nameref activity occurs after shell startup as the default nameref of 'KSH_VERSION=.sh.version' does not create one.
2021-04-13 02:15:34 +00:00
Fix += operator regressions (re: fae8862c) (#270) The bugfix for BUG_CMDSPASGN backported in commit fae8862c caused two regressions with the += operator: 1. The += operator did not append to variables. Reproducer: $ integer foo=3 $ foo+=2 command eval 'echo $foo' 2 2. The += operator ignored the readonly attribute, modifying readonly variables in the same manner as above. Reproducer $ readonly bar=str $ bar+=ing command eval 'echo $bar' ing Both of the regressions above were caused by nv_putval() failing to clone the variable from the previous scope into the invocation-local scope. As a result, 'foo+=2' was effectively 0 + 2 (since ksh didn't clone 3). The first regression was noticed during the development of ksh93v-, so to fix both bugs I've backported the bugfix for the regression from the ksh93v- 2013-10-10 alpha version: https://www.mail-archive.com/ast-users@lists.research.att.com/msg00369.html src/cmd/ksh93/sh/name.c: - To fix both of the bugs above, find the variable to modify with nv_search(), then clone it into the invocation local scope. To fix the readonly bug as well, this is done before the NV_RDONLY check (otherwise np will be missing that attribute and be incorrectly modified in the invocation-local scope). - Update a nearby comment describing what sh_assignok() does (per this comment: https://github.com/ksh93/ksh/pull/249#issuecomment-811381759) src/cmd/ksh93/tests/builtins.sh: - Add regression tests for both of the now fixed regressions, loosely based on the regression tests in ksh93v-.
2021-04-12 00:24:33 +00:00
2021-04-11:
- Fixed two related regressions introduced on 2020-06-16:
1. The += assignment failed to append the value of variables when used
in an invocation-local scope. The following should print '5', but
the regression resulted in '3' being printed instead:
$ integer foo=2; foo+=3 command eval 'echo $foo'
3
2. Any += assignment used in an invocation-local scope could modify
readonly variables.
Reset arithmetic recursion level on all errors (re: 264ba48b) The recursion level for arithmetic expressions is kept track of in a static 'level' variable in streval.c. It is reset when arithmetic expressions throw an error. But an error for an arithmetic expression may also occur elsewhere -- at least in one case: when an arithmetic expression attempts to change a read-only variable. In that case, the recursion level is never reset because that code does not have access to the static 'level' variable. If many such conditions occur (as in the new readonly.sh regression tests), an arithmetic command like 'i++' may eventually fail with a 'recursion too deep' error. To mitigate the problem, MAXLEVEL in streval.c was changed from 9 to 1024 in 264ba48b (as in the ksh 93v- beta). This commit leaves that increase, but adds a proper fix. src/cmd/ksh93/include/defs.h: - Add global sh.arithrecursion (a.k.a. shp->arithrecursion) variable to keep track of the arithmetic recursion level, replacing the static 'level' variable in streval.c. src/cmd/ksh93/sh/xec.c: sh_exec(): - Reset sh.arithrecursion before starting a new simple command (TCOM), a new subshell with parentheses (TPAR), a new pipe (TFIL), or a new [[ ... ]] command (TTST). These are the same places where 'echeck' is set to 1 for --errexit and ERR trap checks, so it should cover everything. src/cmd/ksh93/sh/streval.c: - Change all uses of 'level' to sh.arithrecursion. - _seterror, aritherror(): No longer bother to reset the level to zero here; xec.c should have this covered for all cases now. src/cmd/ksh93/tests/arith.sh: - Add tests for main shell and subshell.
2021-04-10 23:08:26 +00:00
2021-04-10:
- Fixed: the internal count of the recursion level for arithmetic expressions
was not reset when certain errors occurred in a virtual subshell. This could
cause an erroneous "recursion to deep" error when a loop executed many
subshells containing arithmetic expressions with errors, e.g. for testing.
Fix handling of '--posix' and '--default' (#265) src/cmd/ksh93/sh/args.c: sh_argopts(): - Remove special-casing for --posix (see also data/builtins.c) and move the case -5: to the case ':' instead, so this option is handled like all other long options. This change fixes two bugs: 1. 'set --posix' had no effect on the letoctal or braceexpand options. Reproducer: $ set --posix $ [[ -o braceexpand ]]; echo $? 0 $ [[ -o letoctal ]]; echo $? 1 2. 'ksh --posix' could not run scripts correctly because it wrongly enabled '-c'. Reproducer: $ ksh --posix < <(echo 'exit 0') ksh: -c requires argument Usage: ksh [--posix] [arg ...] Help: ksh [ --help | --man ] 2>&1 - Don't allow 'set --default' to unset the restricted option. src/cmd/ksh93/tests/options.sh: - Add regression tests for the bugs described above, using -o posix and --posix. src/cmd/ksh93/tests/restricted.sh: - Add a regression test for 'set --default' in rksh. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-04-09 22:26:07 +00:00
2021-04-09:
- Fixed a bug that caused ksh to enable -c during the shell's initialization
if the only argument passed was --posix.
- Fixed a related bug that caused 'set --posix' to leave the braceexpand and
letoctal shell options unchanged.
- Fixed a bug that caused 'set --default' to unset the restricted option
in restricted shells.
Allow path-bound builtins on restricted shells If a system administrator prefixes /opt/ast/bin to the path and then invokes the shell in restricted mode, they clearly intend for the user to run those AST utilities. Similarly, if a system administrator sets a PATH for a restricted shell that includes libraries listed in the .paths file, they must have intended for the user to use those loadable built-ins, as they will be associated with the pathnames of their respective libraries. Since the user cannot change PATH or use the builtin command, they still cannot load just any built-in they choose. src/cmd/ksh93/sh/path.c: - Remove SH_RESTRICTED check when handling path-bound builtins or dynamic libaries containining builtins in $PATH. src/cmd/ksh93/tests/builtins.sh: - Add test verifying a restricted user can use /opt/ast/bin/cat via a PATH search. Progresses: https://github.com/ksh93/ksh/issues/138
2021-04-08 13:39:30 +00:00
2021-04-08:
- Path-bound builtins will now be used by restricted shells if /opt/ast/bin
is in the $PATH upon invoking the shell or before setting it to restricted.
Fix 'printf %T' ignoring the current locale in LC_TIME (#263) src/lib/libast/tm/tmlocale.c: - Load the locale set by LC_TIME or LC_ALL if it hasn't been loaded before or if it was loaded previously but isn't the current locale. src/cmd/ksh93/tests/locale.sh: - Add a regression test using the nl_NL.UTF-8 and ja_JP.UTF-8 locales. Fixes: https://github.com/ksh93/ksh/issues/261
2021-04-09 02:49:48 +00:00
- Fixed a bug that caused "printf '%T\n' now" to ignore $LC_ALL and $LC_TIME
if the current locale was previously set, unset then set again.
Add LC_TIME to the supported locale variables (#257) The current version of 93u+m does not have proper support for the LC_TIME variable. Setting LC_TIME has no effect on printf %T, and if the locale is invalid no error message is shown: $ LC_TIME=ja_JP.UTF-8 $ printf '%T\n' now Wed Apr 7 15:18:13 PDT 2021 $ LC_TIME=invalid.locale $ # No error message src/cmd/ksh93/data/variables.c, src/cmd/ksh93/include/variables.h, src/cmd/ksh93/sh/init.c: - Add support for the $LC_TIME variable. ksh93v- attempted to add support for LC_TIME, but the patch from that version was extended because the variable still didn't function correctly. src/cmd/ksh93/tests/variables.sh: - Add LC_TIME to the regression tests for LC_* variables.
2021-04-08 12:06:22 +00:00
2021-04-07:
- The $LC_TIME variable is now recognized by ksh and if set to an invalid
locale will show an error.
Backport bugfix for BUG_CSUBSTDO from ksh93v- 2012-08-24 (#259) This commit fixes BUG_CSUBSTDO, which could break stdout inside of non-forking command substitutions. The breakage only occurred when stdout was closed outside of the command substitution and a file descriptor other than stdout was redirected in the command substitution (such as stderr). Thanks to the ast-open-history repo, I was able to identify and backport the bugfix from ksh93v- 2012-08-24. This backport may fix other bugs as well. On 93v- 2012-08-24 it fixed the regression below, though it was not triggered on 93u+(m). src/cmd/ksh93/tests/heredoc.sh 487 print foo > $tmp/foofile 488 x=$( $SHELL 2> /dev/null 'read <<< $(<'"$tmp"'/foofile) 2> /dev/null;print -r "$REPLY"') 489 [[ $x == foo ]] || err_exit '<<< $(<file) not working' src/cmd/ksh93/sh/io.c: sh_open(): - If the just-opened file descriptor exists in sftable and is flagged with SF_STRING (as in non-forking command substitutions, among other situations), then move the file descriptor to a number >= 10. src/cmd/ksh93/tests/io.sh: - Add a regression test for BUG_CSUBSTDO, adapted from the one in modernish.
2021-04-08 12:24:17 +00:00
- Fixed BUG_CSUBSTDO: If standard output is closed before running a command
substitution, redirecting any other file descriptor no longer closes standard
output inside of the command substitution.
Fix unsetting array element after expanding array subscript range Simple reproducer: set -A arr a b c d; : ${arr[1..2]}; unset arr[1]; echo ${arr[@]} Output: a Expected output: a c d The ${arr[1..2]} expansion broke the subsequent 'unset' command so that it unsets element 1 and on, instead of only 1. This regression was introduced in nv_endsubscript() on 2009-07-31: https://github.com/ksh93/ast-open-history/commit/c47896b4/src/cmd/ksh93/sh/array.c That change checks for the ARRAY_SCAN attribute which enables processing ranges of array elements instead of single array elements, and restores it after. That restore is evidently not correct as it causes the subsequent unset command to malfunction. If we revert that change, the bug disappears and the regression tests show no failures. However, I don't know what this was meant to accomplish and what other bug we might introduce by reverting this. However, no corresponding regression test was added along with the 2009-07-31 change, nor is there any corresponding message in the changelog. So this looks to be one of those mystery changes that we'll never know the reason for. Since we currently have proof that this change causes breakage and no evidence that it fixes anything, I'll go ahead and revert it (and add a regression test, of course). If that causes another regression, hopefully someone will find it at some point. src/cmd/ksh93/sh/array.c: nv_endsubscript(): - Revert the 2009-07-31 change that saves/restores the ARRAY_SCAN attribute. - Keep the 'ap' pointer as it is now used by newer code. Move the declaration up to the beginning of the block, as is customary. src/cmd/ksh93/sh/init.c: - Cosmetic change: remove an unused array_scan() macro that I found when grepping the code for ARRAY_SCAN. The macro was introduced in version 2001-06-01 but the code that used it was replaced in version 2001-07-04, without removing the macro itself. Resolves: https://github.com/ksh93/ksh/issues/254
2021-04-05 21:11:23 +00:00
2021-04-05:
- Fixed a regression, introduced in ksh 93t+ 2009-07-31, that caused a command
like 'unset arr[3]' to unset not just element 3 of the array but all elements
starting from 3, if a range expansion like ${arr[5..10]} was previously used.
Backport bugfixes for arrays of 'enum' types from ksh 93v- beta These fixes are applied rather blindly as no one has yet managed to understand the almost entirely uncommented arrays and variables handling code (arrays.c, name.c, nvdisc.c, nvtree.c, nvtype.c). Hopefully we'll figure all that out at some point. In the meantime these backported fixes appear to work fine, and these bugs impact the usability of 'enum', so I'm just going to have to violate my own policy and backport these fixes without understanding them. Thanks to @JohnoKing for putting in a lot of work tracing these. Further discussion at: https://github.com/ksh93/ksh/issues/87 src/cmd/ksh93/sh/array.c: - nv_arraysettype(): * Further simplify the function. After my initial simplification of it (re: 5491fe97), I don't believe there's actually a need to save a duplicate copy of the value. Use the pointer returned by nv_getval() directly to restore the value. * Cope with a null value (nv_getval() returning a NULL pointer). This is needed for compatibility with the backported fix in nvtype.c (below). - array_putval(): If the array's value pointer (up->cp) is a pointer to the empty string, it is set to NULL before calling nv_putv() to prevent an empty string from being deleted. Backport a fix from 93v- that restores the pointer to the empty string if the NV_NOFREE attribute is set. Removing it somehow causes these regressions: enum.sh[86]: ${array[@]} doesn't yield all values for associative enum arrays (expected 'green blue blue red yellow green red orange'; got 'green blue blue yellow green orange') enum.sh[94]: unsetting associative enum array does not work (got 'Color_t -A Colors=([foo]=red [rood]=red)') enum.sh[116]: assigning first enum element to indexed array failed (expected 'red red'; got 'BUG BUG') - nv_associative(): Do not increase the 'nelem' (number of elements) value of the array's 'header' struct if the array is associative and of an enum type. The original 93v- fix only checked for the NV_INTEGER attribute, but backporting that caused several regressions. Using a debug output command I've determined that the exact value of 'type' is somehow consistently set to 0x26 if the array is associative and of an enum type, which is NV_INTEGER | NV_LTOU | NV_RJUST as defined in include/nval.h. I cannot find where/how that value is determined. In any case this fix, based on but more specific than the 93v- one, appears to work fine. Removing it somehow causes this regression: enum.sh[94]: unsetting associative enum array does not work (got 'Color_t -A Colors=()') src/cmd/ksh93/sh/nvtype.c: nv_settype(): - Another fix backported from 93v-. If the variable is an array, also set the type of element 0 of that array using a call to nv_arraysettype(). The value may be null. Removing this somehow causes this regression: enum.sh[94]: unsetting associative enum array does not work (got 'Color_t -A Colors=()') src/cmd/ksh93/tests/enum.sh: - Add tests for all the bugs fixed here, plus some hypothetical bugs (e.g., do the same tests for indexed enum type arrays as for associative enum type arrays, even though indexed enum type arrays didn't have all the same problems). Co-authored-by: Johnothan King <johnothanking@protonmail.com> Resolves: https://github.com/ksh93/ksh/issues/87
2021-04-06 04:45:46 +00:00
- Several fixes for arrays of a type created by 'enum' were backported from ksh
93v-, further to the two enum array fixes already applied on 2021-02-01:
1. The array[@]} expansion was fixed for associative arrays of an enum type.
2. Assignments now work correctly for all enum values for both indexed and
associative arrays.
3. 'unset' will now completely unset an associative array of an enum type.
Hardening of readonly variables (#239) Ksh currently restricts readonly scalar variables from having their values directly changed via a value assignment. However, since ksh allows variable attributes to be altered, the variable's value can be indirectly altered. For instance, if TMOUT=900 (for a 15 minute idle timeout) was set to readonly, all that is needed to alter the value of TMOUT from 900 to 0 is to issue 'typeset -R1 TMOUT', perhaps followed by a 'typeset -i TMOUT' to turn off the shell's timeout value. In addition, there are problems with arrays. The following is incorrectly allowed: typeset -a arr=((a b c) 1) readonly arr arr[0][1]=d arr=(alphas=(a b c);name=x) readonly arr.alphas arr.alphas[1]=([b]=5) arr=(alphas=(a b c);name=x) readonly arr.alphas arr.alphas[1]=(b) typeset -C arr=(typeset -r -a alphas=(a b c);name=x) arr.alphas[1]=() src/cmd/ksh93/bltins/typeset.c: setall(): - Relocate readonly attribute check higher up the code and widen its application to issue an error message if the pre-existing name-pair has the readonly bit flag set. - To avoid compatibility problems, don't check for readonly if NV_RDONLY is the only attribute set (ignoring NV_NOFREE). This allows 'readonly foo; readonly foo' to keep working. src/cmd/ksh93/sh/array.c: nv_endsubscript(): - Apply a readonly flag check when an array subscript or append assignment occurs, but allow type variables (typeset -T) as they utilize '-r' for 'required' sub-variables. src/cmd/ksh93/tests/readonly.sh: - New file. Create readonly tests that validate the warning message and validate that the readonly variable did not change. src/cmd/ksh93/sh/streval.c: - Bump MAXLEVEL from 9 to 1024 as a workaround for arithmetic expansion, avoiding a spurious error about too much recursion when the readonly.sh tests are run. This change is backported from ksh 93v-. TODO: debug a spurious increase in arithmetic recursion level variable when readonly.sh tests with 'typeset -i' are run. That is a different bug for a different commit. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-04-05 05:43:19 +00:00
2021-04-04:
Fix bell character handling when redrawing command line (#250) To set a window title in bash and zsh, the $PS1 prompt can be set with the title placed between $'\E]0;' and $'\a': set -o emacs # Or vi mode typeset -A fmt=( [start_title]=$'\E]0;' [end_title]=$'\a' ) PS1="${fmt[start_title]}$(hostname): $(uname)${fmt[end_title]}\$ " This also works in ksh unless the shell receives SIGWINCH. With a $PS1 that sets a window title, the prompt breaks until two interrupts are received. This is caused by ed_setup() skipping $'\a' (the bell character) when setting up the e_prompt buffer which is an edited version of the final line of the PS1 prompt for use when redrawing the command line. One fix would be to avoid cutting out the bell character. But if the prompt contains a bell, we only want the terminal to beep when a new prompt is printed, and not upon refreshing the command line, e.g. when receiving SIGWINCH or pressing Ctrl+L. To avoid the problem, this commit adds code that cuts out sequences of the form ESC ] <number> ; <text> BELL from the prompt redraw buffer altogether. They are not needed there because these sequences will already have taken effect when the full prompt was printed by io_prompt(). This commit also adds a tweak that should improve the recognition of other escape sequences to count their length. src/cmd/ksh93/edit/edit.c: ed_setup(): - When preparing the e_prompt buffer, cut out dtterm/xterm Operating System Commands that set window/icon title, etc. See: https://invisible-island.net/xterm/ctlseqs/ctlseqs.html - When counting the length of escape sequences in that part of PS1, try to recognize some more types of sequences. These changes are part of a ksh2020 patch: https://github.com/att/ast/issues/399 src/cmd/ksh93/sh.1: - Document that any '!' in escape sequences in the PS1 prompt needs to be changed to '!!'. To avoid breaking compatibility, this requirement is documented instead of backporting the changes to io_prompt() from https://github.com/att/ast/issues/399 which try to remove that requirement for specific escape sequences. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-04-05 07:06:53 +00:00
- A bug was fixed that caused a broken prompt display upon redrawing the
command line if the last line of the prompt includes an xterm escape
sequence that is terminated by $'\a' (the bell character).
Hardening of readonly variables (#239) Ksh currently restricts readonly scalar variables from having their values directly changed via a value assignment. However, since ksh allows variable attributes to be altered, the variable's value can be indirectly altered. For instance, if TMOUT=900 (for a 15 minute idle timeout) was set to readonly, all that is needed to alter the value of TMOUT from 900 to 0 is to issue 'typeset -R1 TMOUT', perhaps followed by a 'typeset -i TMOUT' to turn off the shell's timeout value. In addition, there are problems with arrays. The following is incorrectly allowed: typeset -a arr=((a b c) 1) readonly arr arr[0][1]=d arr=(alphas=(a b c);name=x) readonly arr.alphas arr.alphas[1]=([b]=5) arr=(alphas=(a b c);name=x) readonly arr.alphas arr.alphas[1]=(b) typeset -C arr=(typeset -r -a alphas=(a b c);name=x) arr.alphas[1]=() src/cmd/ksh93/bltins/typeset.c: setall(): - Relocate readonly attribute check higher up the code and widen its application to issue an error message if the pre-existing name-pair has the readonly bit flag set. - To avoid compatibility problems, don't check for readonly if NV_RDONLY is the only attribute set (ignoring NV_NOFREE). This allows 'readonly foo; readonly foo' to keep working. src/cmd/ksh93/sh/array.c: nv_endsubscript(): - Apply a readonly flag check when an array subscript or append assignment occurs, but allow type variables (typeset -T) as they utilize '-r' for 'required' sub-variables. src/cmd/ksh93/tests/readonly.sh: - New file. Create readonly tests that validate the warning message and validate that the readonly variable did not change. src/cmd/ksh93/sh/streval.c: - Bump MAXLEVEL from 9 to 1024 as a workaround for arithmetic expansion, avoiding a spurious error about too much recursion when the readonly.sh tests are run. This change is backported from ksh 93v-. TODO: debug a spurious increase in arithmetic recursion level variable when readonly.sh tests with 'typeset -i' are run. That is a different bug for a different commit. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-04-05 05:43:19 +00:00
- Harden readonly variables. Readonly variables or arrays no longer allow
attribute changes which would otherwise allow their value to be altered.
Expanded support for readonly variables within multidimensional arrays.
Fix bugs related to 'uname -d' in the 'uname' builtin (#251) This commit fixes a bug in the ksh uname builtin's -d option that could change the output of -o (I was only able to reproduce this on Linux): $ builtin uname $ uname -o GNU/Linux $ uname -d (none) $ uname -o (none) I identified this patch from ksh2020 as a fix for this bug: <https://github.com/att/ast/pull/1187> The linked patch was meant to fix a crash in 'uname -d', although I've had no luck reproducing it: <https://github.com/att/ast/issues/1184> src/lib/libcmd/uname.c: - Pass correct buffer to getdomainname() while executing uname -d. src/cmd/ksh93/tests/builtins.sh: - Add a regression test for the reported 'uname -d' crash. - Add a regression test for the output of 'uname -o' after 'uname -d'. - To handle potential crashes when running the regression tests in older versions of ksh, fork the command substitutions that run 'uname -d'.
2021-04-04 21:18:43 +00:00
2021-04-03:
- Fixed a bug that caused the uname builtin's -d option to change the output
of the -o option.
- Fixed a possible crash that could occur when showing the domain name
with the uname builtin's -d option.
`cd -` shouldn't ignore `$OLDPWD` when in a new scope (#249) This bug was first reported at <https://github.com/att/ast/issues/8>. The 'cd' command currently takes the value of $OLDPWD from the wrong scope. In the following example 'cd -' will change the directory to /bin instead of /tmp: $ OLDPWD=/bin ksh93 -c 'OLDPWD=/tmp cd -' /bin src/cmd/ksh93/bltins/cd_pwd.c: - Use sh_scoped() to obtain the correct value of $OLDPWD. - Fix a use-after-free bug. Make the 'oldpwd' variable a static char that points to freeable memory. Each time cd is used, this variable is freed if it points to a freeable memory address and isn't also a pointer to shp->pwd. src/cmd/ksh93/sh/path.c: path_pwd(): - Simplify and add comments. - Scope $PWD properly. src/cmd/ksh93/tests/builtins.sh, src/cmd/ksh93/tests/leaks.sh: - Backport the ksh2020 regression tests for 'cd -' when $OLDPWD is set. - Add test for $OLDPWD and $PWD after subshare. - Add test for $PWD after 'cd'. - Add test for possible memory leak. - Add testing for 'unset' on OLDPWD and PWD. src/cmd/ksh93/COMPATIBILITY: - Add compatibility note about changes to $PWD and $OLDPWD. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-04-02 00:19:19 +00:00
2021-03-31:
- Fixed a bug that caused 'cd -' to ignore the current value of $OLDPWD
when it's set to a different directory in a new scope.
- Fixed a related bug that caused ksh to use the wrong value for $PWD
when in a new scope.
Fix vi mode crashes when going back one word (#246) This bug was originally reported at <https://github.com/att/ast/issues/1467>. A crash can occur when using the 'b' or 'B' vi mode commands to go back one word. I was able to reproduce these crashes with 100% consistency on an OpenBSD virtual machine when ksh is compiled with -D_std_malloc. Reproducer: $ set -o vi $ asdf <ESC> <b or B> The fix is based on Matthew DeVore's analysis: > I suspect this is caused by this line: >> while (vi_isalph(tcur_virt) && tcur_virt >= first_virt) --tcur_virt; > which is in the b codepath. It checks vi_isalph(tcur_virt) before checking > if tcur_virt is in range. These two clauses should be reversed. Note that > line 316 is a similar check for pressing B, and there the tcur_virt value > is checked first. src/cmd/ksh93/edit/vi.c: - Check tcur_virt before using isalph() or isblank() to fix both crashes. At the start of the backword() while loop this check was performed twice, so the redundant check has been removed. src/cmd/ksh93/tests/pty.sh: - Add a regression test for the b, B, w and W editor commands.
2021-03-30 10:25:20 +00:00
2021-03-29:
- Fixed an intermittent crash that could occur in vi mode when using the 'b'
or 'B' commands to go back one word.
`test foo =~ foo` should fail with exit status 2 (#245) When test is passed the '=~' operator, it will silently fail with exit status 1: $ test foo =~ foo; echo $? 1 This bug is caused by test_binop reaching the 'NOTREACHED' area of code. The bugfix was adapted from ksh2020: https://github.com/att/ast/issues/1152 src/cmd/ksh93/bltins/test.c: test_binop(): - Error out with a message suggesting usage of '[[ ... ]]' if '=~' is passed to the test builtin. - Special-case TEST_END (']]') as that is not really an operator. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-03-27 21:51:16 +00:00
2021-03-27:
- The 'test' builtin will now show an error message when given the invalid ']]'
or '=~' operators; it also properly returns with exit status 2 now (instead
of exit status 1). If the invalid operator is supported by [[ ... ]] (such
as '=~'), test will now suggest the usage of [[ ... ]] instead.
Add --globcasedetect shell option for globbing and completion One of the best-kept secrets of libast/ksh93 is that the code includes support for case-insensitive file name generation (a.k.a. pathname expansion, a.k.a. globbing) as well as case-insensitive file name completion on interactive shells, depending on whether the file system is case-insensitive or not. This is transparently determined for each directory, so a path pattern that spans multiple file systems can be part case-sensitive and part case- insensitive. In more precise terms, each slash-separated path name component pattern P is treated as ~(i:P) if its parent directory exists on a case-insensitive file system. I recently discovered this while dealing with <https://github.com/ksh93/ksh/issues/223>. However, that support is dead code on almost all current systems. It depends on pathconf(2) having a _PC_PATH_ATTRIBUTES selector. The 'c' attribute is supposedly returned if the given directory is on a case insensitive file system. There are other attributes as well (at least 'l', see src/lib/libcmd/rm.c). However, I have been unable to find any system, current or otherwise, that has _PC_PATH_ATTRIBUTES. Google and mailing list searches yield no relevant results at all. If anyone knows of such a system, please add a comment to this commit on GitHub, or email me. An exception is Cygwin/Windows, on which the "c" attribute was simply hardcoded, so globbing/completion is always case- insensitive. As of Windows 10, that is wrong, as it added the possibility to mount case-sensitive file systems. On the other hand, this was never activated on the Mac, even though macOS has always used a case-insensitive file like Windows. But, being UNIX, it can also mount case-sensitive file systems. Finally, Linux added the possibility to create individual case- insensitive ext4 directories fairly recently, in version 5.2. https://www.collabora.com/news-and-blog/blog/2020/08/27/using-the-linux-kernel-case-insensitive-feature-in-ext4/ So, since this functionality latently exists in the code base, and three popular OSs now have relevant file system support, we might as well make it usable on those systems. It's a nice idea, as it intuitively makes sense for globbing and completion behaviour to auto-adapt to file system case insensitivity on a per-directory basis. No other shell does this, so it's a nice selling point, too. However, the way it is coded, this is activated unconditionally on supported systems. That is not a good idea. It will surprise users. Since globbing is used with commands like 'rm', we do not want surprises. So this commit makes it conditional upon a new shell option called 'globcasedetect'. This option is only compiled into ksh on systems where we can actually detect FS case insensitivity. To implement this, libast needs some public API additions first. *** libast changes *** src/lib/libast/features/lib: - Add probes for the linux/fs.h and sys/ioctl.h headers. Linux needs these to use ioctl(2) in pathicase(3) (see below). src/lib/libast/path/pathicase.c, src/lib/libast/include/ast.h, src/lib/libast/man/path.3, src/lib/libast/Mamfile: - Add new pathicase(3) public API function. This uses whatever OS-specific method it can detect at compile time to determine if a particular path is on a case-insensitive file system. If no method is available, it only sets errno to ENOSYS and returns -1. Currently known to work on: macOS, Cygwin, Linux 5.2+, QNX 7.0+. - On systems (if any) that have the mysterious _PC_PATH_ATTRIBUTES selector for pathconf(2), call astconf(3) and check for the 'c' attribute to determine case insensitivity. This should preserve compatibility with any such system. src/lib/libast/port/astconf.c: - dynamic[]: As case-insensitive globbing is now optional on all systems, do not set the 'c' attribute by default on _WINIX (Cygwin/Windows) systems. - format(): On systems that do not have _PC_PATH_ATTRIBUTES, call pathicase(3) to determine the value for the "c" (case insensitive) attribute only. This is for compatibility as it is more efficient to call pathicase(3) directly. src/lib/libast/misc/glob.c, src/lib/libast/include/glob.h: - Add new GLOB_DCASE public API flag to glob(3). This is like GLOB_ICASE (case-insensitive matching) except it only makes the match case-insensitive if the file system for the current pathname component is determined to be case-insensitive. - gl_attr(): For efficiency, call pathicase(3) directly instead of via astconf(3). - glob_dir(): Only call gl_attr() to determine file system case insensitivity if the GLOB_DCASE flag was passed. This makes case insensitive globbing optional on all systems. - glob(): The options bitmask needs to be widened to fit the new GLOB_DCASE option. Define this centrally in a new GLOB_FLAGMASK macro so it is easy to change it along with GLOB_MAGIC (which uses the remaining bits for a sanity check bit pattern). src/lib/libast/path/pathexists.c: - For efficiency, call pathicase(3) directly instead of via astconf(3). *** ksh changes *** src/cmd/ksh93/features/options, src/cmd/ksh93/SHOPT.sh: - Add new SHOPT_GLOBCASEDET compile-time option. Set it to probe (empty) by default so that the shell option is compiled in on supported systems only, which is determined by new iffe feature test that checks if pathicase(3) returns an ENOSYS error. src/cmd/ksh93/data/options.c, src/cmd/ksh93/include/shell.h: - Add -o globcasedetect shell option if compiling with SHOPT_GLOBCASEDET. src/cmd/ksh93/sh/expand.c: path_expand(): - Pass the new GLOB_DCASE flag to glob(3) if the globcasedetect/SH_GLOBCASEDET shell option is set. src/cmd/ksh93/edit/completion.c: - While file listing/completion is based on globbing and automatically becomes case-insensitive when globbing does, it needs some additional handling to make a string comparison case-insensitive in corresponding cases. Otherwise, partial completions may be deleted from the command line upon pressing tab. This code was already in ksh 93u+ and just needs to be made conditional upon SHOPT_GLOBCASEDET and globcasedetect. - For efficiency, call pathicase(3) directly instead of via astconf(3). src/cmd/ksh93/sh.1: - Document the new globcasedetect shell option.
2021-03-22 15:01:03 +00:00
2021-03-22:
- A new --globcasedetect shell option is added to ksh on OSs where we can check
for a case-insensitive file system (currently macOS, Windows/Cygwin, Linux
Fix more compiler warnings, typos and other minor issues (#260) Many of these changes are minor typo fixes. The other changes (which are mostly compiler warning fixes) are: NEWS: - The --globcasedetect shell option works on older Linux kernels when used with FAT32/VFAT file systems, so remove the note about it only working with 5.2+ kernels. src/cmd/ksh93/COMPATIBILITY: - Update the documentation on function scoping with an addition from ksh93v- (this does apply to ksh93u+). src/cmd/ksh93/edit/emacs.c: - Check for '_AST_ksh_release', not 'AST_ksh_release'. src/cmd/INIT/mamake.c, src/cmd/INIT/ratz.c, src/cmd/INIT/release.c, src/cmd/builtin/pty.c: - Add more uses of UNREACHABLE() and noreturn, this time for the build system and pty. src/cmd/builtin/pty.c, src/cmd/builtin/array.c, src/cmd/ksh93/sh/name.c, src/cmd/ksh93/sh/nvtype.c, src/cmd/ksh93/sh/suid_exec.c: - Fix six -Wunused-variable warnings (the name.c nv_arrayptr() fixes are also in ksh93v-). - Remove the unused 'tableval' function to fix a -Wunused-function warning. src/cmd/ksh93/sh/lex.c: - Remove unused 'SHOPT_DOS' code, which isn't enabled anywhere. https://github.com/att/ast/issues/272#issuecomment-354363112 src/cmd/ksh93/bltins/misc.c, src/cmd/ksh93/bltins/trap.c, src/cmd/ksh93/bltins/typeset.c: - Add dictionary generator function declarations for former aliases that are now builtins (re: 1fbbeaa1, ef1621c1, 3ba4900e). - For consistency with the rest of the codebase, use '(void)' instead of '()' for print_cpu_times. src/cmd/ksh93/sh/init.c, src/lib/libast/path/pathshell.c: - Move the otherwise unused EXE macro to pathshell() and only search for 'sh.exe' on Windows. src/cmd/ksh93/sh/xec.c, src/lib/libast/include/ast.h: - Add an empty definition for inline when compiling with C89. This allows the timeval_to_double() function to be inlined. src/cmd/ksh93/include/shlex.h: - Remove the unused 'PIPESYM2' macro. src/cmd/ksh93/tests/pty.sh: - Add '# err_exit #' to count the regression test added in commit 113a9392. src/lib/libast/disc/sfdcdio.c: - Move diordwr, dioread, diowrite and dioexcept behind '#ifdef F_DIOINFO' to fix one -Wunused-variable warning and multiple -Wunused-function warnings (sfdcdio() only uses these functions when F_DIOINFO is defined). src/lib/libast/string/fmtdev.c: - Fix two -Wimplicit-function-declaration warnings on Linux by including sys/sysmacros.h in fmtdev().
2021-04-08 18:58:07 +00:00
and QNX 7.0+). When this option is turned on, file name generation
Add --globcasedetect shell option for globbing and completion One of the best-kept secrets of libast/ksh93 is that the code includes support for case-insensitive file name generation (a.k.a. pathname expansion, a.k.a. globbing) as well as case-insensitive file name completion on interactive shells, depending on whether the file system is case-insensitive or not. This is transparently determined for each directory, so a path pattern that spans multiple file systems can be part case-sensitive and part case- insensitive. In more precise terms, each slash-separated path name component pattern P is treated as ~(i:P) if its parent directory exists on a case-insensitive file system. I recently discovered this while dealing with <https://github.com/ksh93/ksh/issues/223>. However, that support is dead code on almost all current systems. It depends on pathconf(2) having a _PC_PATH_ATTRIBUTES selector. The 'c' attribute is supposedly returned if the given directory is on a case insensitive file system. There are other attributes as well (at least 'l', see src/lib/libcmd/rm.c). However, I have been unable to find any system, current or otherwise, that has _PC_PATH_ATTRIBUTES. Google and mailing list searches yield no relevant results at all. If anyone knows of such a system, please add a comment to this commit on GitHub, or email me. An exception is Cygwin/Windows, on which the "c" attribute was simply hardcoded, so globbing/completion is always case- insensitive. As of Windows 10, that is wrong, as it added the possibility to mount case-sensitive file systems. On the other hand, this was never activated on the Mac, even though macOS has always used a case-insensitive file like Windows. But, being UNIX, it can also mount case-sensitive file systems. Finally, Linux added the possibility to create individual case- insensitive ext4 directories fairly recently, in version 5.2. https://www.collabora.com/news-and-blog/blog/2020/08/27/using-the-linux-kernel-case-insensitive-feature-in-ext4/ So, since this functionality latently exists in the code base, and three popular OSs now have relevant file system support, we might as well make it usable on those systems. It's a nice idea, as it intuitively makes sense for globbing and completion behaviour to auto-adapt to file system case insensitivity on a per-directory basis. No other shell does this, so it's a nice selling point, too. However, the way it is coded, this is activated unconditionally on supported systems. That is not a good idea. It will surprise users. Since globbing is used with commands like 'rm', we do not want surprises. So this commit makes it conditional upon a new shell option called 'globcasedetect'. This option is only compiled into ksh on systems where we can actually detect FS case insensitivity. To implement this, libast needs some public API additions first. *** libast changes *** src/lib/libast/features/lib: - Add probes for the linux/fs.h and sys/ioctl.h headers. Linux needs these to use ioctl(2) in pathicase(3) (see below). src/lib/libast/path/pathicase.c, src/lib/libast/include/ast.h, src/lib/libast/man/path.3, src/lib/libast/Mamfile: - Add new pathicase(3) public API function. This uses whatever OS-specific method it can detect at compile time to determine if a particular path is on a case-insensitive file system. If no method is available, it only sets errno to ENOSYS and returns -1. Currently known to work on: macOS, Cygwin, Linux 5.2+, QNX 7.0+. - On systems (if any) that have the mysterious _PC_PATH_ATTRIBUTES selector for pathconf(2), call astconf(3) and check for the 'c' attribute to determine case insensitivity. This should preserve compatibility with any such system. src/lib/libast/port/astconf.c: - dynamic[]: As case-insensitive globbing is now optional on all systems, do not set the 'c' attribute by default on _WINIX (Cygwin/Windows) systems. - format(): On systems that do not have _PC_PATH_ATTRIBUTES, call pathicase(3) to determine the value for the "c" (case insensitive) attribute only. This is for compatibility as it is more efficient to call pathicase(3) directly. src/lib/libast/misc/glob.c, src/lib/libast/include/glob.h: - Add new GLOB_DCASE public API flag to glob(3). This is like GLOB_ICASE (case-insensitive matching) except it only makes the match case-insensitive if the file system for the current pathname component is determined to be case-insensitive. - gl_attr(): For efficiency, call pathicase(3) directly instead of via astconf(3). - glob_dir(): Only call gl_attr() to determine file system case insensitivity if the GLOB_DCASE flag was passed. This makes case insensitive globbing optional on all systems. - glob(): The options bitmask needs to be widened to fit the new GLOB_DCASE option. Define this centrally in a new GLOB_FLAGMASK macro so it is easy to change it along with GLOB_MAGIC (which uses the remaining bits for a sanity check bit pattern). src/lib/libast/path/pathexists.c: - For efficiency, call pathicase(3) directly instead of via astconf(3). *** ksh changes *** src/cmd/ksh93/features/options, src/cmd/ksh93/SHOPT.sh: - Add new SHOPT_GLOBCASEDET compile-time option. Set it to probe (empty) by default so that the shell option is compiled in on supported systems only, which is determined by new iffe feature test that checks if pathicase(3) returns an ENOSYS error. src/cmd/ksh93/data/options.c, src/cmd/ksh93/include/shell.h: - Add -o globcasedetect shell option if compiling with SHOPT_GLOBCASEDET. src/cmd/ksh93/sh/expand.c: path_expand(): - Pass the new GLOB_DCASE flag to glob(3) if the globcasedetect/SH_GLOBCASEDET shell option is set. src/cmd/ksh93/edit/completion.c: - While file listing/completion is based on globbing and automatically becomes case-insensitive when globbing does, it needs some additional handling to make a string comparison case-insensitive in corresponding cases. Otherwise, partial completions may be deleted from the command line upon pressing tab. This code was already in ksh 93u+ and just needs to be made conditional upon SHOPT_GLOBCASEDET and globcasedetect. - For efficiency, call pathicase(3) directly instead of via astconf(3). src/cmd/ksh93/sh.1: - Document the new globcasedetect shell option.
2021-03-22 15:01:03 +00:00
(globbing), as well as file name tab completion on interactive shells,
automatically become case-insensitive on file systems where the difference
between upper- and lowercase is ignored for file names. This is transparently
determined for each directory, so a path pattern that spans multiple file
systems can be part case-sensitive and part case-insensitive.
The option is not compiled into ksh on systems where we do not know of a
method to check for file system case insensitivity. The shell option can be
force-compiled by setting SHOPT_GLOBCASEDET to 1 in src/cmd/ksh93/SHOPT.sh,
but it won't have any effect on non-supported systems, so this is not
recommended. It can be removed from ksh by setting SHOPT_GLOBCASEDET to 0.
File completion: fix incomplete multibyte support Upon encountering two filenames with multibyte characters starting with the same byte, a partial multibyte character was completed. Reproducer (to run in UTF-8 locale): $ touch XXXá XXXë $ : XX <== pres tab $ : XXX^? <== partial multibyte character appears Note: á is $'\xc3\xa1' and ë is $'\xc3\xab' (same initial byte). src/cmd/ksh93/edit/completion.c: - Add multibyte support to the charcmp() and overlaid() functions. Thanks to Harald van Dijk for useful code and suggestions. - Add a few missing mbinit() calls. The state of multibyte processing must be reset before starting a new loop in case a previous processing run was interrupted mid-character. src/cmd/ksh93/tests/pty.sh: - Add test based on Harald's reproducer. Resolves: https://github.com/ksh93/ksh/issues/223
2021-03-17 22:34:45 +00:00
2021-03-17:
- Fixed a bug with file name completion on the interactive shell in multibyte
locales. Upon encountering two filenames with multibyte characters starting
with the same byte, a partial multibyte character was autocompleted.
Save $? when discipline triggered without command (#226) A discipline function could incorrectly influence the value of $? (exit status of last command) outside its context if it was triggered without another command being run, e.g. when a prompt variable is read, or COLUMNS or LINES is set. Reproducers include: PS1 prompt: $ PS1.get() { true; } $ false $ echo $? 0 PS2 prompt: $ PS2.get() { return 13; } $ \ > $ echo $? 13 The set discipline is affected too, e.g. COLUMNS and LINES: $ COLUMNS.set() { return 13; } $ true $ (press return) $ echo $? 13 There are probably other contexts where the shell reads or changes variables without running commands, allowing their get or set disciplines to influence $?. So this commit makes ksh save $? for all .get, .set, .append, and .unset discipline calls. src/cmd/ksh93/sh/nvdisc.c: - assign(): Save/restore $? when running a .set/.append/.unset discipline function. - lookup(): Save/restore $? when running a .get discipline. src/cmd/ksh93/tests/pty.sh: - Add a regression test for $? after displaying a prompt and when setting a LINES.set discipline function. src/cmd/ksh93/tests/return.sh: - The above test fails in script form on ksh93u+ and ksh2020, as it exposes another form of #117 that occurs after running a subshell. Add the above regression test here as well (re: 092b90da). Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-03-16 16:13:13 +00:00
2021-03-16:
Revert "[1.0 release prep] Remove tilde expansion discipline" This reverts c0334e32, thereby restoring 936a1939. After the fixes in 0a343244 and a2bc49be, the tilde expansion disciplines work nicely, so they can come back to the 1.0 branch.
2021-12-09 06:31:37 +00:00
- Tilde expansion can now be extended or modified by defining a .sh.tilde.get
or .sh.tilde.set discipline function. This replaces a 2004 undocumented
attempt to add this functionality via a .sh.tilde built-in, which never
worked and crashed the shell. See the manual for details on the new method.
Save $? when discipline triggered without command (#226) A discipline function could incorrectly influence the value of $? (exit status of last command) outside its context if it was triggered without another command being run, e.g. when a prompt variable is read, or COLUMNS or LINES is set. Reproducers include: PS1 prompt: $ PS1.get() { true; } $ false $ echo $? 0 PS2 prompt: $ PS2.get() { return 13; } $ \ > $ echo $? 13 The set discipline is affected too, e.g. COLUMNS and LINES: $ COLUMNS.set() { return 13; } $ true $ (press return) $ echo $? 13 There are probably other contexts where the shell reads or changes variables without running commands, allowing their get or set disciplines to influence $?. So this commit makes ksh save $? for all .get, .set, .append, and .unset discipline calls. src/cmd/ksh93/sh/nvdisc.c: - assign(): Save/restore $? when running a .set/.append/.unset discipline function. - lookup(): Save/restore $? when running a .get discipline. src/cmd/ksh93/tests/pty.sh: - Add a regression test for $? after displaying a prompt and when setting a LINES.set discipline function. src/cmd/ksh93/tests/return.sh: - The above test fails in script form on ksh93u+ and ksh2020, as it exposes another form of #117 that occurs after running a subshell. Add the above regression test here as well (re: 092b90da). Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-03-16 16:13:13 +00:00
- Fixed a bug in interactive shells: if a variable used by the shell called
a discipline function (such as PS1.get() or COLUMNS.set()), the value of $?
was set to the exit status of the discipline function instead of the last
command run.
Make ~ expand to home directory after unsetting HOME There was an issue with tilde expansion if the HOME var is unset. $ unset HOME $ echo ~ martijn Only the username is returned. Users are more likely to expect the current user's home directory as configured in the OS. POSIXly, the expansion of ~ is based on the value of HOME. If HOME is unset, the results are unspecified. After unsetting HOME, in bash, ~ returns the user's home directory as specified by the OS, whereas in all other shells, ~ expands to the empty string. Only ksh93 returns the username. The behaviour of bash is more useful. Discussion: https://github.com/ksh93/ksh/pull/225#issuecomment-799074107 src/cmd/ksh93/sh/macro.c, src/cmd/ksh93/tests/tilde.sh: - sh_tilde(): Backport fix by Mike Gilbert from ksh2020. See: https://github.com/att/ast/issues/1391 https://github.com/att/ast/pull/1396 https://github.com/att/ast/commit/070d365d - Add test. src/cmd/ksh93/COMPATIBILITY: - Note this change.
2021-03-15 21:49:02 +00:00
2021-03-15:
- If the HOME variable is unset, the bare tilde ~ now expands to the current
user's system-configured home directory instead of merely the username.
typeset: Allow last numeric type given to be used (#221) For most numeric types the last provided one wins out. This commit closes the gap for -F and -i numerics to not be covered up by other preceding float types. Note: -u for requesting an unsigned float or integer was considered and decided to be left alone as it stands, so as to not allow the variable to become an uppercased string if the requested options ended with a -u. As it stands for a case when multiple numeric types are requested, a -u option may be applied after the last numeric type is processed. Examples: -EF becomes -F -Fi becomes -i -Fu becomes -F -uF becomes -F -Fui becomes -i (because isfloat==1, unsigned is not applied) -Fiu becomes -iu (isfloat is reset and allows unsigned to be set) src/cmd/ksh93/bltins/typeset.c: b_typeset(): - Reset attribute bit flags for -E and -X when -F is requested by adding in NV_EXPNOTE to be removed. - For -i option if a float precedes it, reset isfloat and -E/-F attribute bit flags. - Take into account the impact of the shortint flag on floats. src/cmd/ksh93/tests/attributes.sh: - Add some validation tests to confirm that, when a -F follows either -E or -X, -F is used. - Add some validation tests to confirm that, when -F/E/X precede a -i, the variable becomes an integer and not a float. - Add in various tests when -s followed a float.
2021-03-16 10:19:00 +00:00
- Tighten up potential invalid typeset attribute combos when more than
one numeric type has been requested. In particular, -F and -i are no
longer masked over by previously given float types.
Re-enable SHOPT_DEVFD, fixing process substitution fd leaks (#218) This commit fixes a long-standing bug (present since at least ksh93r) that caused a file descriptor leak when passing a process substitution to a function, or (if compiled with SHOPT_SPAWN) to a nonexistent command. The leaks only occurred when ksh was compiled with SHOPT_DEVFD; the FIFO method was unaffected. src/cmd/ksh93/sh/xec.c: sh_exec(): - When a process substitution is passed to a built-in, the remaining file descriptor is closed with sh_iorestore. Do the same thing when passing a process substitution to a function. This is done by delaying the sh_iorestore() call to 'setexit:' where both built-ins and functions terminate and set the exit status ($?). This means that call now will not be executed if a longjmp is done, e.g. due to an error in a special built-in. However, there is already another sh_iorestore() call in main.c, exfile(), line 418, that handles that scenario. - sh_ntfork() can fail, so rather than assume it will succeed, handle a failure by closing extra file descriptors with sh_iorestore(). This fixes the leak on command not found with SHOPT_SPAWN. src/cmd/ksh93/include/defs.h: - Since the file descriptor leaks are now fixed, remove the workaround that forced ksh to use the FIFO method. src/cmd/ksh93/SHOPT.sh: - Add SHOPT_DEVFD as a configurable option (default: probe). src/cmd/ksh93/tests/io.sh: - Add a regression test for the 'not found' file descriptor leak. - Add a test to ensure it keeps working with 'command'. Fixes: https://github.com/ksh93/ksh/issues/67
2021-03-13 13:46:42 +00:00
2021-03-13:
- Fixed a file descriptor leak that occurred when ksh used /dev/fd for
process substitutions passed to functions.
- Fixed a separate file descriptor leak that happened when a process
substitution was passed to a nonexistent command.
Fix unused process substitutions hanging (#214) On systems where ksh needs to use the older and less secure FIFO method for process substitutions (which is currently all of them as the more modern and solid /dev/fd method is still broken, see #67), process substitutions could leave background processes hanging in these two scenarios: 1. If the parent process exits without opening a pipe to the child process forked by the process substitution. The fifo_check() function in xec.c, which is periodically called to check if the parent process still exists while waiting for it to open the FIFO, verified the parent process's existence by checking if the PPID had reverted to 1, the traditional PID of init. However, POSIX specifies that the PPID can revert to any implementation- defined system process in that case. So this breaks on certain systems, causing unused process substitutions to hang around forever as they never detect that the parent disappeared. The fix is to save the current PID before forking and having the child check if the PPID has changed from that saved PID. 2. If command invoked from the main shell is passed a process substitution, but terminates without opening the pipe to the process substitution. In that case, the parent process never disappears in the first place, because the parent process is the main shell. So the same infinite wait occurs in unused process substitutions, even after correcting problem 1. The fix is to remember all FIFOs created for any number of process substitutions passed to a single command, and unlink any remaining FIFOs as they represent unused command substitutions. Unlinking them FIFOs causes sh_open() in the child to fail with ENOENT on the next periodic check, which can easily be handled. Fixing these problems causes the FIFO method to act identically to the /dev/fd method, which is good for compatibility. Even when #67 is fixed this will still be important, as ksh also runs on systems that do not have /dev/fd (such as AIX, HP-UX, and QNX), so will fall back to using FIFOs. --- Fix problem 1 --- src/cmd/ksh93/sh/xec.c: - Add new static fifo_save_ppid variable. - sh_exec(): If a FIFO is defined, save the current PID in fifo_save_ppid for the forked child to use. - fifo_check(): Compare PPID against the saved value instead of 1. --- Fix problem 2 --- To keep things simple I'm abusing the name-value pair routines used for variables for this purpose. The overhead is negligible. A more elegant solution is possible but would involve adding more code. src/cmd/ksh93/include/defs.h: _SH_PRIVATE: - Define new sh.fifo_tree pointer to a new FIFO cleanup tree. src/cmd/ksh93/sh/args.c: sh_argprocsubs(): - After launching a process substitution in the background, add the FIFO to the cleanup list before freeing it. src/cmd/ksh93/sh/xec.c: - Add fifo_cleanup() that unlinks all FIFOs in the cleanup list and clears/closes the list. They should only still exist if the command never used them, however, just run 'unlink' and don't check for existence first as that would only add overhead. - sh_exec(): * Call fifo_cleanup() on finishing all simple commands (when setting $?) or when a special builtin fails. * When forking, clear/close the cleanup list; we do not want children doing duplicate cleanup, particularly as this can interfere when using multiple process substitutions in one command. * Process substitution handling: > Change FIFO check frequency from 500ms to 50ms. Note that each check sends a signal that interrupts open(2), causing sh_open() to reinvoke it. This causes sh_open() to fail with ENOENT on the next check when the FIFO no longer exists, so we do not need to add an additional check for existence to fifo_check(). Unused process substitutions now linger for a maximum of 50ms. > Do not issue an error message if errno == ENOENT. - sh_funct(): Process substitutions can be passed to functions as well, and we do not want commands within the function to clean up the FIFOs for the process substitutions passed to it from the outside. The problem is solved by simply saving fifo_tree in a local variable, setting it to null before running the function, and cleaning it up before restoring the parent one at the end. Since sh_funct() is called recursively for multiple-level function calls, this correctly gives each function a locally scoped fifo_tree. --- Tests --- src/cmd/ksh93/tests/io.sh: - Add tests covering the failing scenarios. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-03-12 11:43:23 +00:00
2021-03-11:
- Fixed an intermittent bug that caused process substitutions to infinitely
loop in Linux virtual machines that use systemd.
- Fixed a bug that caused process substitutions to leave lingering processes
if the command invoking them never reads from them.
Fix ${!foo@} and ${!foo*} to include 'foo' itself in search These expansions are supposed to yield all variable names beginning with the indicated prefix. This should include the variable name that is identical to the prefix (as 'prefix' begins with 'prefix'). This bugfix is backported from the abandoned ksh 93v- beta, so AT&T intended this change. It also makes ksh work like bash in this. src/cmd/ksh93/sh/macro.c: varsub(): M_NAMESCAN: - Check if the prefix itself exists. If so, start with that. src/cmd/ksh93/tests/variables.sh: - Add tests for these expansions. src/cmd/ksh93/sh.1: - Fix the incomplete documentation of these expansions. src/cmd/ksh93/COMPATIBILITY: - Note the change as it's potentially incompatible in corner cases. Resolves: https://github.com/ksh93/ksh/issues/183
2021-03-09 04:35:45 +00:00
2021-03-09:
- The ${!foo@} and ${!foo*} expansions yield variable names beginning with foo,
but excluded 'foo' itself. The fix for this is now backported from 93v- beta.
Fix test -v for numeric types & set/unset state for short int This commit fixes two interrelated problems. 1. The -v unary test/[/[[ operator is documented to test if a variable is set. However, it always returns true for variable names with a numeric attribute, even if the variable has not been given a value. Reproducer: $ ksh -o nounset -c 'typeset -i n; [[ -v n ]] && echo $n' ksh: n: parameter not set That is clearly wrong; 'echo $n' should never be reached and the error should not occur, and does not occur on mksh or bash. 2. Fixing the previous problem revealed serious breakage in short integer type variables that was being masked. After applying that fix and then executing 'typeset -si var=0': - The conditional assignment expansions ${var=123} and ${var:=123} assigned 123 to var, even though it was set to 0. - The expansions ${var+s} and ${var:+n} incorrectly acted as if the variable was unset and empty, respectively. - '[[ -v var ]]' and 'test -v var' incorrectly returned false. The problems were caused by a different storage method for short ints. Their values were stored directly in the 'union Value' member of the Namval_t struct, instead of allocated on the stack and referred to by a pointer, as regular integers and all other types do. This inherently broke nv_isnull() as this leaves no way to distinguish between a zero value and no value at all. (I'm also pretty sure it's undefined behaviour in C to check for a null pointer at the address where a short int is stored.) The fix is to store short ints like other variables and refer to them by pointers. The NV_INT16P combined bit mask already existed for this, but nv_putval() did not yet support it. src/cmd/ksh93/bltins/test.c: test_unop(): - Fix problem 1. For -v, only check nv_isnull() and do not check for the NV_INTEGER attribute (which, by the way, is also used for float variables by combining it with other bits). See also 5aba0c72 where we recently fixed nv_isnull() to work properly for all variable types including short ints. src/cmd/ksh93/sh/name.c: nv_putval(): - Fix problem 2, part 1. Add support for NV_INT16P. The code is simply copied and adapted from the code for regular integers, a few lines further on. The regular NV_SHORT code is kept as this is still used for some special variables like ${.sh.level}. src/cmd/ksh93/bltins/typeset.c: b_typeset(): - Fix problem 2, part 2. Use NV_INT16P instead of NV_SHORT. src/cmd/ksh93/tests/attributes.sh: - Add set/unset/empty/nonempty tests for all numeric types. src/cmd/ksh93/tests/bracket.sh, src/cmd/ksh93/tests/comvar.sh: - Update a couple of existing tests. - Add test for [[ -v var ]] and [[ -n ${var+s} ]] on unset and empty variables with many attributes. src/cmd/ksh93/COMPATIBILITY: - Add a note detailing the change to test -v. src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/sh.1: - Correct 'typeset -C' documentation. Variables declared as compound are *not* initially unset, but initially have the empty compound value. 'typeset' outputs them as: typeset -C foo=() and not: typeset -C foo and nv_isnull() is never true for them. This may or may not technically be a bug. I don't think it's worth changing, but it should at least be documented correctly.
2021-03-10 00:38:00 +00:00
- test -v var, [ -v var ], and [[ -v var ]] did not correctly test if a
variable is set or unset after it has been given a numeric attribute with
'typeset' but not yet assigned a value. This has been fixed so that
[[ -v var ]] is now equivalent to [[ -n ${var+set} ]] as documented.
Fixes for -G/--globstar (re: 5312a59d) The fix for '.' and '..' in regular globbing broke '.' and '..' in globstar. No globstar pattern that contains '.' or '..' as any pathname component still matched. This commit fixes that. This commit also makes symlink/** mostly work, which it never has done in any ksh93 version. It is correct and expected that symlinks found by patterns are not resolved, but symlinks were not resolved even when specified as explicit non-pattern pathname components. For example, /tmp/** breaks if /tmp is a symlink (e.g. on macOS), which looks like a bug. src/lib/libast/include/glob.h, src/lib/libast/misc/glob.c: glob_dir(): - Make symlink/** work. we can check if the string pointed to by pat is exactly equal to *. If so, we are doing regular globbing for that particular pathname element, and it's okay to resolve symlinks. If not (if it's **), we're doing globstar and we should not be matching symlinks. - Let's also introduce proper identification of symlinks (GLOB_SYM) and not lump them in with other special files (GLOB_DEV). - Fix the bug with literal '.' and '..' components in globstar patterns. In preceding code, the matchdir pointer gets set to the complete glob pattern if we're doing globstar for the current pathname element, null if not. The pat pointer gets set to the elements of the pattern that are still left to be processed; already-done elements are trimmed from it by increasing the pointer. So, to do the right thing, we need to make sure that '.' or '..' is skipped if, and only if, it is the final element in the pattern (i.e., if pat does not contain a slash) and is not specified literally as '.' or '..', i.e., only if '.' or '..' was actually resolved from a glob pattern. After this change, '**/.*', '**/../.*', etc. do the right thing, showing all your hidden files and directories without undesirable '.' and '..' results; '.' and '..' are skipped as final elements, unless you literally specify '**/.', '**/..', '**/foo/bar/..', etc. src/cmd/ksh93/COMPATIBILITY: - Note the symlink/** globstar change. src/cmd/ksh93/sh.1: - Try to document the current globstar behaviour more exhausively. src/cmd/ksh93/tests/glob.sh: - Add tests. Try to cover all the corner cases. src/cmd/ksh93/tests/shtests: - Since tests in glob.sh do not use err_exit, they were not counted. Special-case glob.sh for counting the tests: count the lines starting with a test_* function call. Resolves: https://github.com/ksh93/ksh/issues/146
2021-03-06 18:45:42 +00:00
2021-03-07:
Fix set/unset state for short integer (typeset -si) (#211) This commit fixes at least three bugs: 1. When issuing 'typeset -p' for unset variables typeset as short integer, a value of 0 was incorrectly diplayed. 2. ${x=y} and ${x:=y} were still broken for short integer types (re: 9f2389ed). ${x+set} and ${x:+nonempty} were also broken. 3. A memory fault could occur if typeset -l followed a -s option with integers. Additonally, now the last -s/-l wins out as the option to utilize instead of it always being short. src/cmd/ksh93/include/name.h: - Fix the nv_isnull() macro by removing the direct exclusion of short integers from this set/unset test. This breaks few things (only ${.sh.subshell} and ${.sh.level}, as far as we can tell) while potentially correcting many aspects of short integer use (at least bugs 1 and 2 above), as this macro is widely used. - union Value: add new pid_t *pidp pointer member for PID values (see further below). src/cmd/ksh93/bltins/typeset.c: b_typeset(): - To fix bug 3 above, unset the 'shortint' flag and NV_SHORT attribute bit upon encountering the -l optiobn. *** To fix ${.sh.subshell} to work with the new nv_isnull(): src/cmd/ksh93/sh/defs.h: - Add new 'realsubshell' member to the shgd (aka shp->gd) struct which will be the integer value for ${.sh.subshell}. src/cmd/ksh93/sh/init.c, src/cmd/ksh93/data/variables.c: - Initialize SH_SUBSHELLNOD as a pointer to shgd->realsubshell instead of using a short value (.s) directly. Using a pointer allows nv_isnull() to return a positive for ${.sh.subshell} as a non-null pointer is what it checks for. - While we're at it, initialize PPIDNOD ($PPID) and SH_PIDNOD (${.sh.pid}) using the new pdip union member, which is more correct as they are values of type pid_t. src/cmd/ksh93/sh/subshell.c, src/cmd/ksh93/sh/xec.c: - Update the ${.sh.subshell} increases/decreases to refer to shgd->realsubshell (a.k.a. shp->gd->realsubshell). *** To fix ${.sh.level} after changing nv_isnull(): src/cmd/ksh93/sh/macro.c: varsub(): - Add a specific exception for SH_LEVLNOD to the nv_isnull() test, so that ${.sh.level} is always considered to be set. Its handling throughout the code is too complex/special for a simple fix, so we have to special-case it, at least for now. *** Regression test additions: src/cmd/ksh93/tests/attributes.sh: - Add in missing short integer tests and correct the one that existed. The -si test now yields 'typeset -x -r -s -i foo' instead of 'typeset -x -r -s -i foo=0' which brings it in line with all the others. - Add in some other -l attribute tests for floats. Note, -lX test was not added as the size of long double is platform dependent. src/cmd/ksh93/tests/variables.sh: - Add tests for ${x=y} and ${x:=y} used on short int variables. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-03-08 04:19:36 +00:00
- Fixed the typeset -p display of short integers without an assigned value.
Also, the last -s or -l attribute option supplied for an integer is used.
Fixes for -G/--globstar (re: 5312a59d) The fix for '.' and '..' in regular globbing broke '.' and '..' in globstar. No globstar pattern that contains '.' or '..' as any pathname component still matched. This commit fixes that. This commit also makes symlink/** mostly work, which it never has done in any ksh93 version. It is correct and expected that symlinks found by patterns are not resolved, but symlinks were not resolved even when specified as explicit non-pattern pathname components. For example, /tmp/** breaks if /tmp is a symlink (e.g. on macOS), which looks like a bug. src/lib/libast/include/glob.h, src/lib/libast/misc/glob.c: glob_dir(): - Make symlink/** work. we can check if the string pointed to by pat is exactly equal to *. If so, we are doing regular globbing for that particular pathname element, and it's okay to resolve symlinks. If not (if it's **), we're doing globstar and we should not be matching symlinks. - Let's also introduce proper identification of symlinks (GLOB_SYM) and not lump them in with other special files (GLOB_DEV). - Fix the bug with literal '.' and '..' components in globstar patterns. In preceding code, the matchdir pointer gets set to the complete glob pattern if we're doing globstar for the current pathname element, null if not. The pat pointer gets set to the elements of the pattern that are still left to be processed; already-done elements are trimmed from it by increasing the pointer. So, to do the right thing, we need to make sure that '.' or '..' is skipped if, and only if, it is the final element in the pattern (i.e., if pat does not contain a slash) and is not specified literally as '.' or '..', i.e., only if '.' or '..' was actually resolved from a glob pattern. After this change, '**/.*', '**/../.*', etc. do the right thing, showing all your hidden files and directories without undesirable '.' and '..' results; '.' and '..' are skipped as final elements, unless you literally specify '**/.', '**/..', '**/foo/bar/..', etc. src/cmd/ksh93/COMPATIBILITY: - Note the symlink/** globstar change. src/cmd/ksh93/sh.1: - Try to document the current globstar behaviour more exhausively. src/cmd/ksh93/tests/glob.sh: - Add tests. Try to cover all the corner cases. src/cmd/ksh93/tests/shtests: - Since tests in glob.sh do not use err_exit, they were not counted. Special-case glob.sh for counting the tests: count the lines starting with a test_* function call. Resolves: https://github.com/ksh93/ksh/issues/146
2021-03-06 18:45:42 +00:00
- Fixed a bug with -G/--globstar introduced on 2020-08-09: patterns did not
match anything if any pathname component was '.' or '..', e.g. '**/./glob.c'
never matched. The 2020-08-09 fix does still apply to patterns like '.*'.
- Enhancement to -G/--globstar: symbolic links to directories are now followed
if they match a normal (non-**) glob pattern. For example, if '/lnk' is a
symlink to a directory, '/lnk/**' and '/l?k/**' now work as you would expect.
job_init(): fix init on setpgid() permission denied (re: 41ebb55a) Symptoms of this bug below. These only seem to occur on Linux and only if you replace your initial login shell by ksh using 'exec'. 1. An erroneous 'Interrupt' message is printed after stopping the read builtin in a script. Reproducer: $ exec arch/*/bin/ksh $ cat ./reproducer.sh #!/bin/sh read foo $ ./reproducer.sh ^C$ <Enter> [1] + Interrupt ../reproducer.sh 2. Ctrl+C fails to stop /bin/package make. Reproducer: $ exec arch/*/bin/ksh $ mv arch arch.old $ bin/package make # Press Ctrl+C multiple times Analysis: In 41ebb55a, I made an error in changing job_init() to work correctly on non-interactive shells. This line from before: 552| if(possible = (setpgid(0,job.mypgid)>=0) || errno==EPERM) was changed to: 555| possible = (setpgid(0,job.mypgid) >= 0); 556| if(sh_isoption(SH_INTERACTIVE) && (possible || errno==EPERM)) That is wrong. Before, 'possible' was set to 1 (true) if setpgid() either succeeded or failed with EPERM. After, it is only set to 1 if setpgid() succeeds. As a result, job control initialisation is aborted later on upon a test for non-zero 'possible'. src/cmd/ksh93/sh/jobs.c: job_init(): - Once again set possible to 1 even if setpgid() fails with EPERM. Thanks to @JohnoKing for the bug report and reproducers. Resolves: https://github.com/ksh93/ksh/issues/210
2021-03-07 17:01:17 +00:00
- Fixed a bug introduced on 2021-02-11 that caused job control on interactive
ksh sessions to misbehave if the login shell was replaced by ksh using 'exec'.
Fix ${x=y} and ${x:=y} for numeric types of x These POSIX expansions first assign y to x if x is unset or empty, respectively, and then they yield the value of x. This was not working on any ksh93 version if x was typeset as numeric (integer or float) but still unset, as in not assigned a value. $ unset a; typeset -i a; printf '%q\n' "${a:=42}" "$a" 0 '' Expected output: 42 42 src/cmd/ksh93/sh/macro.c: - Fix the test for set/unset variable. It was broken because it only checked for the existence of the node, which exists after 'typeset', but did not check if a value had been assigned. This additional check needs to be done with the nv_isnull() macro, but only for expansions of the regular M_BRACE type. Special expansions cannot have an unset state. - As of commit 95294419, we know that an nv_optimize() call may be needed before using nv_isnull() if the shell is compiled with SHOPT_OPTIMIZE. Move the nv_optimize() call from that commit forward to before the new check that calls nv_isnull(), and only bother with it if the type is M_BRACE. src/cmd/ksh93/tests/variables.sh: - Add tests for this bug. Test float and integer, and also check that ${a=b} and ${a:=b} correctly treat the value of 'b' as an arithmetic expression of which the result is assigned to 'a' if 'a' was typeset as numeric. src/cmd/ksh93/tests/attributes.sh, src/cmd/ksh93/tests/comvar.sh, src/cmd/ksh93/tests/nameref.sh, src/cmd/ksh93/tests/types.sh: - Fix a number of tests to report failures correctly. Resolves: https://github.com/ksh93/ksh/issues/157
2021-03-06 03:56:52 +00:00
2021-03-06:
- Fixed an old expansion bug: expansions of type ${var=value} and ${var:=value}
did not perform an assignment and yielded the value 0 if 'var' was typeset as
numeric (integer or float) but had not yet been assigned a value.
Fix Ctrl+D after ksh receives SIGWINCH (#208) src/cmd/ksh93/edit/edit.c: ed_read(): - The loop that handles SIGWINCH assumes sfpkrd will return and set errno to EINTR if ksh is sent SIGWINCH. This only occurs when select(2) is used to wait for input, so tell sfpkrd to use select if possible. This is only done if the last argument given to sfpkrd is '2', which should avoid regressions. src/lib/libast/sfio/sfpkrd.c: sfpkrd(): - Always use select if the last argument is 2. This allows sfpkrd() to intercept SIGWINCH when necessary. Fixes: https://github.com/ksh93/ksh/issues/202
2021-03-06 06:43:38 +00:00
- Fixed a bug introduced on 2020-08-19: Ctrl+D would break after an
interactive shell received SIGWINCH.
Fix command history corruption on syntax error (re: e999f6b1) Analysis: When a syntax error occurs, the shell performs a longjmp(3) back to exfile() in main.c on line 417: 415| if(jmpval) 416| { 417| Sfio_t *top; 418| sh_iorestore((void*)shp,0,jmpval); 419| hist_flush(shp->gd->hist_ptr); 420| sfsync(shp->outpool); The first thing it does is restore the file descriptor state (sh_iorestore), then it flushes the history file (hist_flush), then it synchronises sfio's logical stream state with the physical stream state using (sfsync). However, the fix applied in e999f6b1 caused sh_iorestore() to sync all sfio streams unconditionally. So this was done before hist_flush(), which caused unpredictable behaviour, including temporary and/or permanent history corruption, as this also synched shp->outpool before hist_flush() had a chance to do its thing. The fix is to only call sfsync() in sh_iorestore() if we're actually about to call ftruncate(2), and not otherwise. Moral of the story: bug fixes should be as specific as possible to minimise the risk of side effects. src/cmd/ksh93/sh/io.c: sh_iorestore(): - Only call sfsync() if we're about to truncate a file. src/cmd/ksh93/tests/pty.sh: - Add test. Thanks to Marc Wilson for reporting the bug and to Johnothan King for finding the commit that introduced it. Resolves: https://github.com/ksh93/ksh/issues/209 Relevant: https://github.com/att/ast/issues/61
2021-03-07 00:27:33 +00:00
- Fixed a bug introduced on 2020-05-21: on an interactive shell, command lines
containing a syntax error were not added to the command history file and
sometimes corrupted the command history.
Remove obsolete quote balancing hack The old Bourne shell failed to check for closing quotes and command substitution backticks when encountering end-of-file in a parser context (such as a script). ksh93 implemented a hack for partial compatibility with this bug, tolerating unbalanced quotes and backticks in backtick command subsitutions, 'eval', and command line invocation '-c' scripts only. This hack became broken for backtick command substitutions in fe20311f/350b52ea as a memory leak was fixed by adding a newline to the stack at the end of the command substitution. That extra newline becomes part of any string whose quotes are not properly terminated, causing problems such as the one detailed here: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01889.html $ touch abc $ echo `ls "abc` ls: abc : not found No other fix for the memory leak is known that doesn't cause other problems. (The alternative fix detailed in the referenced mailing list post causes a different corner-case regression.) Besides, the hack has always caused other corner case bugs as well: $ ksh -c '((i++' Actual: ksh: i++(: not found (If an external command 'i++(' existed, it would be run) Expect: ksh: syntax error at line 1: `(' unmatched $ ksh -c 'i=0; echo $((++i' Actual: (empty line; the arithmetic expansion is ignored) Expect: ksh: syntax error at line 1: `(' unmatched $ ksh -c 'echo $(echo "hi)' Actual: ksh: syntax error at line 1: `(' unmatched Expect: ksh: syntax error at line 1: `"' unmatched So, it's time to get rid of this hack. The old Bourne shell is dead and buried. No other shell tries to support this breakage. Tolerating syntax errors is just asking for strange side effects, inconsistent states, and corner case bugs. We should not want to do that. Old scripts that rely on this will just need to be fixed. src/cmd/ksh93/sh/lex.c: - struct lexdata: Remove 'char balance' member for remembering an unbalanced quote or backtick. - sh_lex(): Remove the back to remember and compensate for unbalanced quotes/backticks that was executed only if we were executing a script from a string, as opposed to a file. src/cmd/ksh93/COMPATIBILITY: - Note the change. Resolves: https://github.com/ksh93/ksh/issues/199
2021-03-05 21:55:25 +00:00
2021-03-05:
- Unbalanced quotes and backticks now correctly produce a syntax error
in -c scripts, 'eval', and backtick-style command substitutions.
Fix arbitrary command execution vuln in array subscripts in arith This commit fixes an arbitrary command execution vulnerability in array subscripts used within the arithmetic subsystem. One of the possible reproducers is: var='1$(echo INJECTION >&2)' ksh -c \ 'typeset -A a; ((a[$var]++)); typeset -p a' Output before this commit: INJECTION typeset -A a=([1]=1) The 'echo' command has been surreptitiously executed from an external environment variable. Output after this commit: typeset -A a=(['1$(echo INJECTION >&2)']=1) The value is correctly used as an array subscript and nothing in it is parsed or executed. This is as it should be, as ksh93 supports arbitrary subscripts for associative arrays. If we think about it logically, the C-style arithmetic subsystem simply has no business messing around with shell expansions or quoting at all, because those don't belong to it. Shell expansions and quotes are properly resolved by the main shell language before the arithmetic subsystem is even invoked. It is particularly important to maintain that separation because the shell expansion mechanism also executes command substitutions. Yet, the arithmetic subsystem subjected array subscripts that contain `$` (and only array subscripts -- how oddly specific) to an additional level of expansion and quote resolution. For some unfathomable reason, there are two lines of code doing specifically this. The vulnerability is fixed by simply removing those. Incredibly, variants of this vulnerability are shared by bash, mksh and zsh. Instead of fixing it, it got listed in Bash Pitfalls! http://mywiki.wooledge.org/BashPitfalls#y.3D.24.28.28_array.5B.24x.5D_.29.29 src/cmd/ksh93/sh/arith.c: - scope(): Remove these two lines that implement the vulnerability. if(strchr(sub,'$')) sub = sh_mactrim(shp,sub,0); - scope(), arith(): Remove the NV_SUBQUOTE flag from two nv_endsubscript() calls. That flag causes the array subscript to retain the current level of shell quoting. The shell quotes everything as in "double quotes" before invoking the arithmetic subsystem, and the bad sh_mactrim() call removed one level of quoting. Since we're no longer doing that, this flag should no longer be passed, or subscripts may get extra backslash escapes. src/cmd/ksh93/include/name.h, src/cmd/ksh93/sh/array.c: - nv_endsubscript(): The NV_SUBQUOTE flag was only passed from arith.c. Since it is now unused, remove it. src/cmd/ksh93/tests/arith.sh: - Tweak some tests: fix typos, report wrong values. - Add 21 tests. Most are based on reproducers contributed by @stephane-chazelas and @hyenias. They verify that this vulnerability is gone and that no quoting bugs were introduced. Resolves: https://github.com/ksh93/ksh/issues/152
2021-03-04 13:37:13 +00:00
2021-03-04:
- Fixed an arbitrary command execution vulnerability that occurred when
parsing the subscripts of arrays within arithmetic commands and expansion.
Readonly attribute size fix (#201) Corrected the size of attribute(s) being overwritten with 0 when 'readonly' or 'typeset -r' was applied to an existing variable. Since one cannot set any attributes with the 'readonly' command, its function call to setall() needs to be adjusted to acquire the current size from the old size or existing size of the variable. A plain 'typeset -r' is the same as 'readonly' in that it needs to load the old size as its current size for use in the subsequent to call to nv_newattr(). src/cmd/ksh93/bltins/typeset.c: setall(): - Both 'readonly' and 'typeset -r' end up calling setall(). setall() has full visibility into all user supplied values and existing values that are needed to differentiate whereas name.c newattr() acquires combined state flags. - Added a conditional check if the readonly flag was requested by user then meets the criteria of having present size of 0, cannot be a numeric nor binary string, and is void of presence of any of the justified string attributes. - -L/R/Z justified string attributes if not given a value default to a size of 0 which means to autosize. A binary string can have a fixed field size, e.g. -bZ. The present of any of the -L/R/Z attribules means that current size is valid and should be used even if it is zero. src/cmd/ksh93/tests/attributes.sh: - Added various tests to capture and reiterate that 'readonly' should be equivalent to 'typeset -r' and applying them should not alter the previous existing size unless additional attributes are set along with typeset command.
2021-03-03 03:26:39 +00:00
2021-03-01:
- Fixed the retention of size attributes when 'readonly' or 'typeset -r'
was applied to an existing variable.
emacs: Fix three tab completion bugs 1. The editor accepted literal tabs without escaping in certain cases, causing buggy and inconsistent completion behaviour. https://github.com/ksh93/ksh/issues/71#issuecomment-656970959 https://github.com/ksh93/ksh/issues/71#issuecomment-657216472 2. After completing a filename by choosing from a file completion menu, the terminal cursor was placed one position too far to the right, corrupting command line display. This happened with multiline active. https://github.com/ksh93/ksh/issues/71#issue-655093805 3. A completion menu was displayed if the file name to be completed was at the point where the rest of it started with a number, even if that part uniquely identified it so the menu had 1 item. https://www.mail-archive.com/ast-users@lists.research.att.com/msg00436.html src/cmd/ksh93/edit/emacs.c: - Cosmetic consistency: change two instances of cntl('[') to ESC. - ed_emacsread(): Fix number 1 by refusing to continue into default processing if a tab character was not used for tab completion. Instead, beep and continue to the next read loop iteration. This behaviour is consistent with most other shells, so I doubt there will be objections. To enter a literal tab it's simple enough to escape it with ^V (the 'stty lnext' character) or \. - draw(): Fix number 2 by correcting an off-by-one error in the ed_setcursor() call that updates the terminal's cursor display in multiline mode. The 'old' and 'new' parameters need to have identical values in this particular call to avoid the cursor position being off by one to the right. This change makes it match the corresponding ed_setcursor() call in vi.c. See below* for details. Thanks to Lev Kujawski for the help in analysing. src/cmd/ksh93/edit/completion.c: ed_expand(): - Fix number 3 by changing from '=' mode (menu-based completion) to '\' mode (ordinary filename completion) if the menu would only show one option, which was pointless and annoying. This never happened in vi mode, so possibly the ed_expand() call in emacs.c could have been improved instead. But I'm comfortable with fixing it here and not in emacs.c, because this fixes it at a more fundamental level, plus it's straightforward and obvious here. Resolves: https://github.com/ksh93/ksh/issues/71 ____ * Further details on bug number 2: At https://github.com/ksh93/ksh/issues/71#issuecomment-786391565 Martijn Dekker wrote: > I'm back to my original hypothesis that there is somehow an > off-by-one error related to the ed_setcursor() call that gets > executed when in multiline mode. I cannot confirm whether that > off-by-one error is actually in the call itself, or occurs > sometime earlier on one of the many possible occasions where > ep->cursor is changed. But everything else appears to work > correctly, so it's not unlikely that the problem is in the call > itself. > > For reference, this is the original version of that call in > emacs.c: > > ksh/src/cmd/ksh93/edit/emacs.c > Lines 1556 to 1557 in df2b9bf > if(ep->ed->e_multiline && option == REFRESH) > ed_setcursor(ep->ed, ep->screen, ep->cursor-ep->screen, ep->ed->e_peol, -1); > > There is a corresponding call in the vi.c refresh() function > (which does the same thing as draw() in emacs.c), where the third > (old) and fourth (new) arguments are actually identical: > > ksh/src/cmd/ksh93/edit/vi.c > > Lines 2086 to 2087 in df2b9bf > if(vp->ed->e_multiline && vp->ofirst_wind==INVALID) > ed_setcursor(vp->ed, physical, last_phys+1, last_phys+1, -1); > > The expectation for this particular call is in fact that they > should be identical, so that a delta of zero is calculated in > that function. Delta not being zero is what causes the cursor to > be positioned wrong. > > In vi.c, last_phys is a macro that is defined as editb.e_peol, > and editb is a macro that is defined as (*vp->ed). Which means > last_phys means vp->ed->e_peol, which is the same as > ep->ed->e_peol in emacs.c. (These editors were originally > separate programs by different authors, and I suppose this is how > it shows. Korn didn't want to change all the variable names to > integrate them, so made macros instead.) > > That leaves the question of why vi.c adds 1 to both last_phys > a.k.a. e_peol arguments, and emacs.c uses e_peol for new without > adding anything. Analysing the ed_setcursor() code could answer > that question. > > So, this patch makes emacs.c do it the same way vi.c does. Let's > make the third argument identical to the fourth. My brief testing > shows the bug is fixed, and the regression tests yield no > failures. This fix is also the most specific change possible, so > there are few opportunities for side effects (I hope). At https://github.com/ksh93/ksh/issues/71#issuecomment-786466652 Lev Kujawski wrote: > I did a bit of research on this, and I think the fix to have the > Emacs editing mode do the same as Vi is correct. > > From RELEASE: > 08-05-01 In multiline edit mode, the refresh operation will now clear > the remaining portion of the last line. > > Here's a fragment from the completion.c of the venerable but > dated CDE DtKsh: > > else > while (*com) > { > *out++ = ' '; > out = strcopy(out,*com++); > } > *cur = (out-outbuff); > /* restore rest of buffer */ > out = strcopy(out,stakptr(0)); > *eol = (out-outbuff); > > Noticeably missing is the code to add a space after file name > completions. So, it seems plausible that if multiline editing > mode was added beforehand,the ep->ed->p_eol != > ep->cursor-ep->screen case might never have occurred during > testing. > > Setting the 'first' parameter to -1 seems to be a pretty explicit > indicator that the author(s) intended the line clearing code to > run, hence the entry in RELASE. > > The real issue is that if we update the cursor by calling > ed_setcursor on line 1554 with old != new, the later call to > setcursor on line 1583, here: > > I = (ncursor-nscreen) - ep->offset; > setcursor(ep,i,0); > > will use outdated screen information to call setcursor, which, > coincidentally, calls ed_setcursor.
2021-02-26 11:20:58 +00:00
2021-02-26:
- Fixed three long-standing bugs with tab completion in the emacs editor:
1. The editor accepted literal tabs without escaping in certain cases,
causing buggy and inconsistent completion behaviour. Details:
https://github.com/ksh93/ksh/issues/71#issuecomment-656970959
https://github.com/ksh93/ksh/issues/71#issuecomment-657216472
To enter a literal tab in emacs, you need to escape it with ^V or \.
2. After completing a filename by choosing from a file completion menu,
the terminal cursor was placed one position too far to the right,
corrupting command line display. This happened with multiline active.
Details: https://github.com/ksh93/ksh/issues/71#issue-655093805
3. A completion menu was displayed if the file name to be completed was
at the point where the rest of it started with a number, even if that
part uniquely identified it so the menu only showed one item. Details:
https://www.mail-archive.com/ast-users@lists.research.att.com/msg00436.html
Fix ${.sh.fun} leaking out of DEBUG trap The value of the ${.sh.fun} variable, which is supposed to contain the name of the function currently being executed, leaks out of the DEBUG trap if it executes a function. Reproducer: $ fn() { echo "executing the function"; } $ trap fn DEBUG $ trap - DEBUG executing the function $ echo ${.sh.fun} fn ${.sh.fun} should be empty outside the function. Annalysis: The sh_debug() function in xec.c, which executes the DEBUG trap action, contains these lines, which are part of restoring the state after running the trap action with sh_trap(): nv_putval(SH_PATHNAMENOD,shp->st.filename,NV_NOFREE); nv_putval(SH_FUNNAMENOD,shp->st.funname,NV_NOFREE); shp->st = savst; First the SH_PATHNAMENOD (${.sh.file}) and SH_FUNNAMENOD (${.sh.fun}) variables get restored from the values in the shell's scoped information struct (shp->st), but that is done *before* restoring the parent scope with 'shp->st = savst;'. It should be done after. Fixing the order is sufficient to fix the bug. However, I am not convinced that these nv_putval() calls are good for anything at all. Setting, unsetting, restoring, etc. the ${.sh.fun} and ${.sh.file} variables is already being handled perfectly well elsewhere in the code for executing functions and sourcing dot scripts. The DEBUG trap is neither here nor there. There's no reason for it to get involved with these variables. I was unable to break anything after simply removing those two lines. So I strongly suspect this is another case, out of many now, where a bug in ksh93 is properly fixed by removing some code. I couldn't get ${.sh.file} to leak similarly -- I think this is because SH_PATHNAMENOD (and not SH_FUNNOD) is set explicitly in exfile() in main.c, masking this incorrect restore. It is the only place where SH_PATHNAMENOD and SH_FUNNOD are not both set. src/cmd/ksh93/sh/xec.c: - Remove these two spurious nv_putval() calls. src/cmd/ksh93/tests/variables.sh: - Add regression test for leaking ${.sh.fun}.
2021-02-27 01:01:30 +00:00
- A bug with ${.sh.fun} in combination with the DEBUG trap has been fixed.
The ${.sh.fun} variable wrongly continued to contain the name of the last
function executed by the DEBUG trap after the trap action completed.
editors: fix broken SIGWINCH handling In the emacs editor: 1. press ESC 2. change the size of your terminal window and your screen is mysteriously cleared. (Until recent fixes, the shell probably also crashed somewhere in the job control code.) The cause is the way SIGWINCH is handled by ed_read() in edit.c. For the emacs editor, it sends a Ctrl+L character to the input buffer. The Ctrl+L command refreshes the command line. And it so happens that ESC plus Ctrl+L is a command to clear the screen in the emacs editor. With the exeption of vi insert/command mode for which it uses a shared flag, edit.c does not know the state of the editor, because their data are internal to emacs.c and vi.c. So it doesn't know whether you're in some mode that treats keyboard input specially. Which means this way of dealing with SIGWINCH is fundamentally misdesigned and is not worth fixing. It gets sillier: in addition to sending keyboard commands, edit.c was also communicating directly with emacs.c and vi.c via a flag, e_nocrnl, which means 'please don't make Ctrl+L emit a linefeed' (it normally refreshes on a new line but that is undesirable for SIGWINCH). So there is already a hack that breaks the barrier between edit.c and emacs.c/vi.c. Let's do that properly instead. As of this commit, ed_read() does not send any fake keystrokes. Instead, two extern functions, emacs_redraw() and vi_redraw(), are defined for redrawing the command line. These are put in emacs.c and vi.c so they have access to relevant static data and functions. Then, instead of sending keyboard commands to the editor and returning, ed_read() simply calls the redraw function for the active editor, then continues and waits for input. Much cleaner. src/cmd/ksh93/include/edit.h: - Remove e_nocrnl flag from Edit_t struct. - Define externs emacs_redraw() and vi_redraw(). Since Emacs_t and Vi_t types are not known here, we have to declare void* pointers and the functions will have to use typecasts. src/cmd/ksh93/edit/edit.c: - ed_read(): Call emacs_redraw() or vi_redraw() as per above. - ed_getchar(): Remove comment about a nonexistent while loop. src/cmd/ksh93/edit/emacs.c: - Updates corresponding to removal of e_nocrnl flag. - Add emacs_redraw(). This one is pretty simple. Refresh the command line, then ed_flush() to update the cursor display. src/cmd/ksh93/edit/vi.c: - Updates corresponding to removal of e_nocrnl flag. Also remove a similar internal 'nonewline' flag which is now equally redundant. - Move the Ctrl+L handling code (minus writing the newline) into the vi_redraw() function. - Change two cases where vi set nonewline and sent Ctrl+L to itself into simple vi_redraw() calls. - Add vi_redraw(). This is more complicated as it incorporates the previous Ctrl+L code. It needs an added refresh() call with a check whether we're currently in command or insert mode, as those use different refresh methods. Luckily edit.c already maintains an *e_vi_insert flag in ed_getchar() that we can use. Since vi's refresh() already calls ed_flush(), we don't need to add that.
2021-02-21 23:07:30 +00:00
2021-02-21:
- Fixed: The way that SIGWINCH was handled (i.e. the signal emitted when the
terminal window size changes) could cause strange emacs/vi editor behaviour.
DEBUG trap: restore status 2 trigger to skip command (re: d00b4b39) So now we know what that faulty check for shp->indebug in sh_trap() was meant to do: it was meant to pass down the trap handler's exit status, via sh_debug(), down to sh_exec() (xec.c) so that it could then skip the execution of the next command if the trap's exit status is 2, as documented in the manual page. As of d00b4b39, exit status 2 was not passed down, so this stopped working. This commit reinstates that functionality, but without the exit status bug in command substitutions caused by the old way. src/cmd/ksh93/sh/fault.c: sh_trap(): - Save the trap's exit status before restoring the parent envionment's exit status. Make this saved exit status the return value of the function. (This does not break anything, AFAICT; the majority of sh_trap() calls ignore the return value, and the few that don't ignore it seem to expect it to return exactly this.) src/cmd/ksh93/sh/xec.c: sh_exec(): - The sh_trap() fix has one side effect: whereas the exit status of a skipped command was always 2 (as per the trap handler), now it is always 0, because it gets reset in sh_exec() but no command is executed. That is probably not a desirable change in behaviour, so let's fix that here instead: set sh.exitval to 2 when skipping commands. src/cmd/ksh93/sh.1: - Document that ${.sh.command} shell-quotes its arguments for use by 'eval' and such. This fact was not documented anywhere, AFAIK. src/cmd/ksh93/shell.3: - Document that $? (exit status) is made local to trap handlers. - Document that sh_trap() returns the trap handler's exit status. src/cmd/ksh93/tests/basic.sh: - Add test for this bug. - Add a missing test for the exit status 255 functionality (if a DEBUG trap handler yields this exit status and we're executing a function or dot script, a return is triggered). Fixes: https://github.com/ksh93/ksh/issues/187
2021-02-20 05:13:51 +00:00
2021-02-20:
- Fixed a bug introduced on 2021-01-20: if a DEBUG trap action yielded exit
status 2, the execution of the next command was not skipped as documented.
Fix multiple buffer overflows with justified strings (-L/-R/-Z) ksh crashed in various different and operating system-dependent ways when attempting to create or apply justification strings using typeset -L/-R/-Z, especially if large sizes are used. The crashes had two immediate causes: - In nv_newattr(), when applying justification attributes, a buffer was allocated for the justified string that was exactly 8 bytes longer than the original string. Any larger justification string caused a buffer overflow (!!!). - In nv_putval(), when applying existing attributes to a new value, the corresponding memmove() either did not zero-terminate the justified string (if the original string was longer than the justified string) or could read memory past the original string (if the original string was shorter than the justified string). Both scenarios can cause a crash. This commit fixes other minor issues as well, such as a mysterious 8 extra bytes allocated by several malloc/realloc calls. This may have been some naive attempt to paper over the above bugs. It seems no one can make any other kind of sense of it. A readjustment bug with zero-filling was also fixed. src/cmd/ksh93/sh/name.c: - nv_putval(): . Get rid of the magical +8 bytes for malloc and realloc. Just allocate one extra byte for the terminating zero. . Fix the memmove operation to use strncpy instead, so that buffer overflows are avoided in both scenarios described above. Also make it conditional upon a size adjustment actually happening (i.e. if 'dot' is nonzero). . Mild refactoring: combine two 'if(sp)' blocks into one; declare variables only used there locally for legibility. - nv_newattr(): * Replace the fatally broken "let's allocate string length + 8 bytes no matter the size of the adjustment" routine with a new one based on work by @hyenias (see comments in #142). It is efficient with memory, taking into account numeric types, growing strings, and shrinking strings. * Fix zero-filling in readjustment after changing the initial size of a -Z attribute. If the number was zero, all zeros were still skipped, leaving an empty string. Thanks to @hyenias for originally identifying this breakage and laying the groundwork for fixing nv_newattr(), and to @lijog for the crash analysis that revealed the key to the nv_putval() fix. Resolves: https://github.com/ksh93/ksh/issues/142 Resolves: https://github.com/ksh93/ksh/issues/181
2021-02-20 12:27:53 +00:00
- Fixed multiple buffer overflows causing crashes in typeset -L/-R-/-Z.
- Fixed typeset -Z zero-filling: if the number was zero, all zeros
were skipped when changing the initial size value of the -Z attribute,
leaving an empty string.
Make 'read' compatible with Shift-JIS This commit fixes a bug in the 'read' built-in: it did not properly skip over multibyte characters. The bug never affects UTF-8 locales because all UTF-8 bytes have the high-order bit set. But Shift-JIS characters may include a byte corresponding to the ASCII backslash character, which cauased buggy behaviour when using 'read' without the '-r' option that disables backslash escape processing. It also makes the regression tests compatible with Shift-JIS locales. They failed with syntax errors. src/cmd/ksh93/bltins/read.c: - Use the multibyte macros when skipping over word characters. Based on a patch from the old ast-developers mailing list: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01848.html src/cmd/ksh93/include/defs.h: - Be a bit smarter about causing the compiler to optimise out multibyte code when SHOPT_MULTIBYTE is disabled. See the updated comment for details. src/cmd/ksh93/tests/locale.sh: - Put all the supported locales in an array for future tests. - Add test for the 'read' bug. Include it in a loop that tests 64 SHIFT-JIS character combinations. Only one fails on old ksh: the one where the final byte corresponds to the ASCII backslash. It doesn't hurt to test all the others anyway. src/cmd/ksh93/tests/basic.sh, src/cmd/ksh93/tests/builtins.sh, src/cmd/ksh93/tests/quoting2.sh: - Fix syntax errors that occurred in SHIFT-JIS locales as the parser was processing literal UTF-8 characters. Not executing that code is not enough; we need to make sure it never gets parsed as well. This is done by wrapping the commands containing literal UTF-8 strings in an 'eval' command as a single-quoted operand. .github/workflows/ci.yml: - Run the tests in the ja_JP.SJIS locale instead of ja_JP.UTF-8. UTF-8 is already covered by the nl_NL.UTF-8 test run; that should be good enough.
2021-02-18 14:46:10 +00:00
2021-02-18:
- A bug was fixed in the 'read' builtin that caused it to fail to process
multibyte characters properly in Shift-JIS locales.
Fix emacs backslash escaping behavior (#179) This commit fixes the following: 1. Emacs mode ignores --nobackslashctrl (re: 24598fed) when in reverse search. 2. When entering more than one backslash, emacs reverse search mode deletes multiple backslashes after pressing backspace once. Reproducer: $ set --emacs --nobackslashctrl $ <Ctrl+R> \\\\<Backspace> 3. Except when in reverse search, the backslash fails to escape a subsequent interrupt character (^C). Reproducer: $ set --emacs --backslashctrl $ teststring \<Ctrl+C> src/cmd/ksh93/edit/emacs.c: - Disable escaping backslashes in emacs reverse search if 'nobackslashctrl' is enabled. - Fix the buggy behavior of backslashes in emacs reverse search by processing backslashes in a loop. src/cmd/ksh93/tests/pty.sh: - Add regression tests. src/cmd/ksh93/sh.1: - Fix a minor documentation error (^C is the usual interrupt character, not ^?). Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-02-17 14:29:12 +00:00
2021-02-17:
- Emacs mode fixes:
1. Erasing a backslash while doing a reverse search (^R) no longer deletes
extra characters.
2. The backslash now escapes a subsequent interrupt (^C) as documented.
Fix subshell scoping of changes in shared command substitution A ${ shared-state command substitution; } (internally called subshare) is documented to share its state with the parent shell environment, so all changes made within the command substitution survive outside of it. However, when it is run within a virtual/non-forked subshell, variables that are not already local to that subshell will leak out of it into the grandparent state. Reproducer: $ ksh -c '( v=${ bug=BAD; } ); echo "$bug"' BAD If the variable pre-exists in the subshell, the bug does not occur: $ ksh -c '( bug=BAD1; v=${ bug=BAD2; } ); echo "$bug"' (empty line, as expected) The problem is that the sh_assignok() function, which is responsible for variable scoping in virtual subshells, does not ever bother to create a virtual subshell scope for a subshare. That is an error if a subshare's parent (or higher-up ancestor) environment is a virtual subshell, because a scope needs to be created in that parent environment if none exists. To make this bugfix possible, first we need to get something out of the way. nv_restore() temporarily sets the subshell's pointer to the preesnt working directory, shpwd, to null. This causes sh_assignok() to assume that the subshell is a subshare (because subshares don't store their own PWD) and refuse to create a scope. However, nv_restore() sets it to null for a different purpose: to temporarily disable scoping for *all* virtual subshells, making restoring possible. This is a good illustration of why it's often not a good idea to use the same variable for unrelated purposes. src/cmd/ksh93/sh/subshell.c: - Add a global static subshell_noscope flag variable to replace the misuse of sh.shpwd described above. - sh_assignok(): . Check subshell_noscope instead of shpwd to see if scope creation is disabled. This makes it possible to distinguish between restoring scope and handling subshares. . If the current environment is a subshare that is in a virtual subshell, create a scope in the parent subshell. This is done by temporarily making the parent virtual subshell the current subshell (by setting the global subshell_data pointer to it) and calling sh_assignok() again, recursively. - nv_restore(): To disable subshell scope creation while restoring, set subshell_noscope instead of saving and unsetting sh.shpwd. src/cmd/ksh93/tests/subshell.sh: - Add tests. I like tests. Tests are good. Fixes: https://github.com/ksh93/ksh/issues/143
2021-02-17 15:33:48 +00:00
- Fixed a longstanding bug with shared-state command substitutions of the form
${ command; }. If these were executed in a subshell, changes made within
could survive not only the command substitution but also the parent subshell.
Add new 'nobackslashctrl' shell option; other minor editor tweaks The following emacs editor 'feature' kept making me want to go back to bash. I forget a backslash escape in a command somewhere. So I go back to insert it. I type the \, then want to go forward. My right arrow key, instead of moving the cursor, then replaces my backslash with garbage. Why? The backslash escapes the following control character at the editor level and inserts it literally. The vi editor has a variant of this which is much less harmful. It only works in insert mode and the backslash only escapes the next kill or erase character. In both editors, this feature is completely redundant with the 'stty lnext' character which is ^V by default -- and works better as well because it also escapes ^C, ^J (linefeed) and ^M (Return). [In fact, you could even issue 'stty lnext \\' and get a much more consistent version of this feature on any shell. You have to type two backslashes to enter one, but it won't kill your cursor keys.] If it were up to me alone, I'd simply remove this misfeature from both editors. However, it is long-standing documented behaviour. It's in the 1995 book. Plus, POSIX specifies the vi variant of it. So, this adds a shell option instead. It was quite trivial to do. Now I can 'set --nobackslashctrl' in my ~/.kshrc. What a relief! Note: To keep .kshrc compatibile with older ksh versions, use: command set --nobackslashctrl 2>/dev/null src/cmd/ksh93/include/shell.h, src/cmd/ksh93/data/options.c: - Add new SH_NOBACKSLCTRL/"nobackslashctrl" long-form option. The "no" prefix shows it to the user as "backslashctrl" which is on by default. This avoids unexpectedly changing historic behaviour. src/cmd/ksh93/edit/emacs.c: ed_emacsread(), src/cmd/ksh93/edit/vi.c: getline(): - Only set the flag for special backslash handling if SH_NOBACKSLCTRL is off. src/cmd/ksh93/sh.1, src/cmd/ksh93/data/builtins.c: - Document the new option (as "backslashctrl", on by default). Other minor tweaks: src/cmd/ksh93/edit/edit.c: - ed_setup(): Add fallback #error if no tput method is set. This should never be triggered; it's to catch future editing mistakes. - escape(): cntl('\t') is nonsense as '\t' is already a control character, so change this to just '\t'. - xcommands(): Let's enable the ^X^D command for debugging information on non-release builds. src/cmd/ksh93/features/cmds: - The tput feature tests assumed a functioning terminal in $TERM. However, for all we know we might be compiling with no tty and TERM=dumb. The tput commands won't work then. So set TERM=ansi to use a standard default.
2021-02-16 00:07:03 +00:00
2021-02-15:
Fix BUG_CASEEMPT: empty 'case' list was syntax error 'case x in esac' should be syntactically correct, but was an error: $ ksh -c 'case x in esac' ksh: syntax error at line 1: `case' unmatched Inserting a newline was a workaround: $ ksh -c $'case x in\nesac' (no output) The problem was that the 'esac' reserved word was not being recognised if it immediately followed the 'in' reserved word. src/cmd/ksh93/sh/lex.c: sh_lex(): - Do not turn off recognition of reserved words after 'in' if we're in a 'case' construct; only do this for 'for' and 'select'. src/cmd/ksh93/tests/case.sh: - Add seven regression test for correct recognition of 'esac'. Only two failed on ksh93. The rest is to catch future bugs. Fixes: https://github.com/ksh93/ksh/issues/177
2021-02-16 06:50:12 +00:00
- Fixed a regression introduced by ksh93 (was not in ksh88): an empty 'case'
list on a single line ('case x in esac') was a syntax error.
Fix the Alt+D and Alt+H keyboard shortcuts in emacs mode (#178) This commit fixes the functionality of Alt+D and Alt+H in emacs mode. These keyboard shortcuts are intended to work on whole words, but after commit 13c3fb21 their functionality was reduced to deleting only singular letters: $ Test word <Alt+H> # This should delete 'word', not just 'd'. $ Foo <Alt+B> <Alt+D> # This should delete 'Foo', not just 'F'. Man page entries for reference: M-d Delete current word. M-^H (Meta-backspace) Delete previous word. M-h Delete previous word. src/cmd/ksh93/edit/emacs.c: - 'count' cannot be overridden when handling Alt+D or Alt+H, so add the total number of repetitions to count (the number of repetitions can't be negative). - If 'count' is a negative number, set it to one before adding the number of repetitions.
2021-02-16 01:47:15 +00:00
- Fixed a bug in the emacs built-in editor, introduced on 2020-09-17, that
made the Meta-D and Meta-H keys delete single characters instead of words.
Add new 'nobackslashctrl' shell option; other minor editor tweaks The following emacs editor 'feature' kept making me want to go back to bash. I forget a backslash escape in a command somewhere. So I go back to insert it. I type the \, then want to go forward. My right arrow key, instead of moving the cursor, then replaces my backslash with garbage. Why? The backslash escapes the following control character at the editor level and inserts it literally. The vi editor has a variant of this which is much less harmful. It only works in insert mode and the backslash only escapes the next kill or erase character. In both editors, this feature is completely redundant with the 'stty lnext' character which is ^V by default -- and works better as well because it also escapes ^C, ^J (linefeed) and ^M (Return). [In fact, you could even issue 'stty lnext \\' and get a much more consistent version of this feature on any shell. You have to type two backslashes to enter one, but it won't kill your cursor keys.] If it were up to me alone, I'd simply remove this misfeature from both editors. However, it is long-standing documented behaviour. It's in the 1995 book. Plus, POSIX specifies the vi variant of it. So, this adds a shell option instead. It was quite trivial to do. Now I can 'set --nobackslashctrl' in my ~/.kshrc. What a relief! Note: To keep .kshrc compatibile with older ksh versions, use: command set --nobackslashctrl 2>/dev/null src/cmd/ksh93/include/shell.h, src/cmd/ksh93/data/options.c: - Add new SH_NOBACKSLCTRL/"nobackslashctrl" long-form option. The "no" prefix shows it to the user as "backslashctrl" which is on by default. This avoids unexpectedly changing historic behaviour. src/cmd/ksh93/edit/emacs.c: ed_emacsread(), src/cmd/ksh93/edit/vi.c: getline(): - Only set the flag for special backslash handling if SH_NOBACKSLCTRL is off. src/cmd/ksh93/sh.1, src/cmd/ksh93/data/builtins.c: - Document the new option (as "backslashctrl", on by default). Other minor tweaks: src/cmd/ksh93/edit/edit.c: - ed_setup(): Add fallback #error if no tput method is set. This should never be triggered; it's to catch future editing mistakes. - escape(): cntl('\t') is nonsense as '\t' is already a control character, so change this to just '\t'. - xcommands(): Let's enable the ^X^D command for debugging information on non-release builds. src/cmd/ksh93/features/cmds: - The tput feature tests assumed a functioning terminal in $TERM. However, for all we know we might be compiling with no tty and TERM=dumb. The tput commands won't work then. So set TERM=ansi to use a standard default.
2021-02-16 00:07:03 +00:00
- A new 'backslashctrl' shell option has been added. It is on by default.
Turning it off (set +o backslashctrl or set --nobackslashctrl) disables the
special escaping behaviour of the backslash character in the emacs and vi
built-in editors. Particularly in the emacs editor, this makes it much easier
to go back, insert a forgotten backslash into a command, and then continue
editing without having your next cursor key replace your backslash with
garbage. Note that Ctrl+V (or whatever other character was set using
'stty lnext') always escapes all control characters in either editing mode.
add NEWS entry for e2d54b71
2021-02-14 19:39:01 +00:00
2021-02-14:
Various minor capitalization and typo fixes (#371) This commit fixes various minor typos, punctuation errors and corrects the capitalization of many names.
2021-12-12 21:40:38 +00:00
- Due to a deficiency in some UNIX variants, the 'sleep' built-in command could
add NEWS entry for e2d54b71
2021-02-14 19:39:01 +00:00
occasionally sleep for slightly less than the time specified. It now performs
an additional check against the system clock to make sure it sleeps at least
the given amount of time. Thanks to Lev Kujawski for adding this feature.
Fix bugs related to --posix shell option (re: 921bbcae, f45a0f16) This fixes the following: 1. 'set --posix' now works as an equivalent of 'set -o posix'. 2. The posix option turns off braceexpand and turns on letoctal. Any attempt to override that in a single command such as 'set -o posix +o letoctal' was quietly ignored. This now works as long as the overriding option follows the posix option in the command. 3. The --default option to 'set' now stops the 'posix' option, if set or unset in the same 'set' command, from changing other options. This allows the command output by 'set +o' to correctly restore the current options. src/cmd/ksh93/data/builtins.c: - To make 'set --posix' work, we must explicitly list it in sh_set[] as a supported option so that AST optget(3) recognises it and won't override it with its own default --posix option, which converts the optget(3) string to at POSIX getopt(3) string. This means it will appear as a separate entry in --man output, whether we want it to or not. So we might as well use it as an example to document how --optionname == -o optionname, replacing the original documentation that was part of the '-o' description. src/cmd/ksh93/sh/args.c: sh_argopts(): - Add handling for explitit --posix option in data/builtins.c. - Move SH_POSIX syncing SH_BRACEEXPAND and SH_LETOCTAL from sh_applyopts() into the option parsing loop here. This fixes the bug that letoctal was ignored in 'set -o posix +o letoctal'. - Remember if --default was used in a flag, and do not sync options with SH_POSIX if the flag is set. This makes 'set +o' work. src/cmd/ksh93/include/argnod.h, src/cmd/ksh93/data/msg.c, src/cmd/ksh93/sh/args.c: sh_printopts(): - Do not potentially translate the 'on' and 'off' labels in 'set -o' output. No other shell does, and some scripts parse these. src/cmd/ksh93/sh/init.c: sh_init(): - Turn on SH_LETOCTAL early along with SH_POSIX if the shell was invoked as sh; this makes 'sh -o' and 'sh +o' show expected options (not that anyone does this, but correctness is good). src/cmd/ksh93/include/defs.h, src/cmd/ksh93/include/shell.h: - The state flags were in defs.h and most (but not all) of the shell options were in shell.h. Gather all the shell state and option flag definitions into one place in shell.h for clarity. - Remove unused SH_NOPROFILE and SH_XARGS option flags. src/cmd/ksh93/tests/options.sh: - Add tests for these bugs. src/lib/libast/misc/optget.c: styles[]: - Edit default optget(3) option self-documentation for clarity. Several changed files: - Some SHOPT_PFSH fixes to avoid compiling dead code.
2021-02-14 23:51:19 +00:00
- A few bugs were fixed that 93u+m introduced along with the new '-o posix'
shell option on 2020-09-01:
1. 'set --posix' now works as the expected equivalent of 'set -o posix'.
2. As of 2020-09-18, the posix option turns off braceexpand and turns on
letoctal. Any attempt to override that in a single command such as
'set -o posix +o letoctal' was quietly ignored. This now works as long
as the overriding option follows the posix option on the command line.
3. The --default option to 'set' now stops the 'posix' option, if set or
unset in the same 'set' command, from changing other options. This allows
the command output by 'set +o' to correctly restore the current options.
Fix ${ comsub; } killing job control Another longstanding whopper of a bug in basic ksh93 functionality: run a ${ shared-state; } command substitution twice and job control promptly loses track of all your running jobs. New jobs are tracked again until you run another two shared-state command substitutions. This is in at least 93t+, 93u-, 93u+, 93v- and ksh2020. $ sleep 300 & [1] 56883 $ jobs # OK [1] + Running sleep 300 & $ v=${ echo hi1; } $ jobs # OK [1] + Running sleep 300 & $ v=${ echo hi2; } $ jobs # Nothing! $ fg ksh: fg: no such job src/cmd/ksh93/sh/subshell.c: sh_subshell(): - The current environment number shp->curenv (a.k.a. sh.curenv) was not being restored if the virtual subshell we're leaving is of the shared-state command substitution variety as it was wrongly considered to be part of the environment that didn't need restoring. This caused it to be out of sync with shp->jobenv (a.k.a. sh.jobenv) which did get restored from savedcurenv. Restore both from savedcurenv at the same time for any subshell. (How these numbers are used exactly remains to be discovered.) src/cmd/ksh93/tests/jobs.sh: - Added, with a test for this bug to start it off. There is no other test script where job control fits, and a lot more related fixes are anticipated: https://github.com/ksh93/ksh/issues/119
2021-02-11 13:41:10 +00:00
2021-02-11:
- Fixed a bug that caused ksh to lose track of all running background jobs if
a shared-state command substitution of the form v=${ cmd; } was used twice.
Fix most of job control (-m/-o monitor) in scripts If I haven't missed anything, this should make the non-interactive aspects of job control in scripts work as expected, except for the "<command unknown>" issue in the output of 'bg', 'fg' and 'jobs' (which is not such a high priority as those commands are really designed for interactive use). Plus, I believe I now finally understand what these three are for: * The job.jobcontrol variable is set to nonzero by job_init() in jobs.c if, and only if, the shell is interactive *and* managed to get control of the terminal. Therefore, any changing of terminal settings (tcsetpgrp(3), tty_set()) should only be done if job.jobcontrol is nonzero. This commit changes several checks for sh_isoption(SH_INTERACTIVE) to checks for job.jobcontrol for better consistency with this. * The state flag, sh_isstate(SH_MONITOR), determines whether the bits of job control that are relevant for both scripts and interactive shells are active, which is mostly making sure that a background job gets its own process group (setpgid(3)). * The shell option, sh_isoption(SH_MONITOR), is just that. When the user turns it on or off, the state flag is synched with it. It should usually not be directly checked for, as the state may be temporarily turned off without turning off the option. Prior discussion: https://www.mail-archive.com/austin-group-l@opengroup.org/msg06456.html src/cmd/ksh93/bltins/typeset.c, src/cmd/ksh93/sh/args.c: - Move synching the SH_MONITOR state flag with the SH_MONITOR shell option from b_set() (the 'set' builtin) to sh_applyopts() which is indirectly called from b_set() and is also used when parsing the shell invocation command line. This ensures -m is properly enabled in both scenarios. src/cmd/ksh93/sh/jobs.c: - job_init(): Do not refuse to initialise job control on non-interactive shells. Instead, skip everything that should only be done on interactive shells (i.e., everything to do with the terminal). This function is now even more of a mess than it was before, so refactoring may be desirabe at some point. - job_close(), job_set(), job_reset(), job_wait(): Do not reset the terminal process group (tcsetpgrp()) if job.jobcontrol isn't on. src/cmd/ksh93/sh/xec.c: - sh_exec(): TFORK: For SIGINT handling, check the SH_MONITOR state flag, not the shell option. - sh_exec(): TFORK: Do not turn off the SH_MONITOR state flag in forked children. The non-interactive part of job control should stay active. Instead, turn off the SH_INTERACTIVE state flag so we don't get interactive shell behaviour (i.e. job control noise on the terminal) in forked subshells. - _sh_fork(), sh_ntfork(): Do not reset the terminal process group (tcsetpgrp()) if job.jobcontrol isn't on. Do not turn off the SH_MONITOR state flag in forked children. src/cmd/ksh93/sh/subshell.c: sh_subfork(): - Do not turn off the monitor option and state in forked subshells. The non-interactive part of job control should stay active. src/cmd/ksh93/bltins/misc.c: b_bg(): - Check isstate(SH_MONITOR) instead of sh_isoption(SH_MONITOR) && job.jobcontrol before throwing a 'no job control' error. This fixes a minor bug: fg, bg and disown could quietly fail. src/cmd/ksh93/tests/jobs.sh: - Add tests for 'fg' with job control IDs (%%, %1) in scripts. - Add test checking that a background job launched from a subsell with job control enabled correctly becomes the leader of its own process group. Makes progress on: https://github.com/ksh93/ksh/issues/119
2021-02-11 22:05:00 +00:00
- Job control (the -m/-o monitor option) has been fixed for scripts. Background
jobs are now correctly assigned their own process group when run from
subshells (except command substitutions). The 'fg' command now also works for
scripts as it does on other shells, though 'wait' should be preferred.
Make redirections like {varname}>file work with brace expansion off This is some nonsense: redirections that store a file descriptor greater than 9 in a variable, like {var}<&2 and the like, stopped working if brace expansion was turned off. '{var}' is not a brace expansion as it doesn't contain ',' or '..'; something like 'echo {var}' is always output unexpanded. And redirections and brace expansion are two completely unrelated things. It wasn't documented that these redirections require the -B/braceexpand option, either. src/cmd/ksh93/sh/lex.c: sh_lex(): - Remove incorrect check for braceexpand option before processing redirections of this form. src/cmd/ksh93/COMPATIBILITY: - Insert a brief item mentioning this. src/cmd/ksh93/sh.1: - Correction: these redirections do not yield a file descriptor > 10, but > 9, a.k.a. >= 10. - Add a brief example showing how these redirections can be used. src/cmd/ksh93/tests/io.sh: - Add a quick regression test.
2021-02-05 03:42:10 +00:00
2021-02-05:
- Fixed a longstanding bug that caused redirections that store a file
descriptor > 10 in a variable, such as {var}>file, to stop working if
brace expansion (the -B or -o braceexpand option) was turned off. (Note
that '{var}' is not a brace expansion as it does not contain ',' or '..'.)
Add missing NEWS entries (re: a410bc48, 6f3b23e6); update README.md
2021-02-04 17:31:22 +00:00
2021-02-04:
- Fixed ksh crashing if an autoloaded function tried to autoload itself.
ksh now errors out gracefully with an "autoload loop" error message.
- Fixed crash on trying a very long nonexistent command.
Correctly block invalid values for arrays of an enum type This fixes part of https://github.com/ksh93/ksh/issues/87: Scalar arrays (-a) and associative arrays (-A) of a type created by 'enum' did not consistently block values not specified by the enum type, yielding corrupted results. An expansion of type "${array[@]}" yielded random numbers instead of values for associative arrays of a type created by 'enum'. This does not yet fix another problem: ${array[@]} does not yield all values for associative enum arrays. src/cmd/ksh93/bltins/enum.c: put_enum(): - Always throw an error if the value is not in the list of possible values for an enum type. Remove incorrect check for the NV_NOFREE flag. Whatever that was meant to accomplish, I've no idea. src/cmd/ksh93/sh/array.c: nv_arraysettype(): - Instead of sh_eval()ing a shell assignment, use nv_putval() directly. Also use the stack (see src/lib/libast/man/stk.3) instead of malloc to save the value; it's faster and will be auto-freed at some point. This shortens the function and makes it faster by not entering into a whole new shell context -- which also fixes another problem: the error message from put_enum() didn't cause the shell to exit for indexed enum arrays. src/cmd/ksh93/sh/name.c: nv_setlist(): - Apply a patch from David Korn that correctly sets the data type for associative arrays, fixing the ${array[@]} expansion yielding random numbers. Thanks to @JohnoKing for the pointer. https://github.com/ksh93/ksh/issues/87#issuecomment-662613887 https://www.mail-archive.com/ast-developers@lists.research.att.com/msg00697.html src/cmd/ksh93/tests/enum.sh: - Add tests checking that invalid values are correctly blocked for indexed and associative arrays of an enum type. Makes progress on: https://github.com/ksh93/ksh/issues/87
2021-02-01 16:19:04 +00:00
2021-02-01:
typeset: fix short integer restriction (#166) This commit corrects how shortint was being applied to various possible typeset variables in error. The short integer option modifier 'typeset -s' should only be able to be applied if the the variable is also an integer. Several issues were resolved with this fix: - 'typeset -s': created a short integer having an invalid base of zero. 'typeset -s foo' created 'typeset -s -i 0 foo=0' and now will result in an empty string. - 'typeset -sL': previously resulted in a segmentation fault. The following are the various incorrect 'typeset' instances that have been fixed: $ 'export foo; typeset -s foo; readonly foo; typeset -p foo' (before) typeset -x -r -s -i 0 foo=0 ( after) typeset -x -r foo $ 'typeset -sL foo=1*2; typeset -p foo' (before) Segmentation fault (core dumped) ( after) typeset -L 3 foo='1*2' $ 'typeset -sR foo=1*2; typeset -p foo' (before) typeset -s -i foo=2 ( after) typeset -R 3 foo='1*2' $ 'typeset -sZ foo=1*2; typeset -p foo' (before) typeset -F 0 foo=2 ( after) typeset -Z 3 -R 3 foo='1*2' src/cmd/ksh93/bltins/typeset.c: b_typeset(): - Add conditional check within the 's' option to only apply NV_SHORT as well as remove any NV_LONG flag if NV_INTEGER flag was set. - Relocate shortint conditional logic to the 'i' option. src/cmd/ksh93/tests/attributes.sh: - Adjust regression tests for '-s' and add '-si' check.
2021-02-01 23:35:18 +00:00
- Fixed a bug in 'typeset': the '-s' modifier option for short integer will
now only be applied if the integer option '-i' is also present, avoiding
inconsistent results and a crash.
Correctly block invalid values for arrays of an enum type This fixes part of https://github.com/ksh93/ksh/issues/87: Scalar arrays (-a) and associative arrays (-A) of a type created by 'enum' did not consistently block values not specified by the enum type, yielding corrupted results. An expansion of type "${array[@]}" yielded random numbers instead of values for associative arrays of a type created by 'enum'. This does not yet fix another problem: ${array[@]} does not yield all values for associative enum arrays. src/cmd/ksh93/bltins/enum.c: put_enum(): - Always throw an error if the value is not in the list of possible values for an enum type. Remove incorrect check for the NV_NOFREE flag. Whatever that was meant to accomplish, I've no idea. src/cmd/ksh93/sh/array.c: nv_arraysettype(): - Instead of sh_eval()ing a shell assignment, use nv_putval() directly. Also use the stack (see src/lib/libast/man/stk.3) instead of malloc to save the value; it's faster and will be auto-freed at some point. This shortens the function and makes it faster by not entering into a whole new shell context -- which also fixes another problem: the error message from put_enum() didn't cause the shell to exit for indexed enum arrays. src/cmd/ksh93/sh/name.c: nv_setlist(): - Apply a patch from David Korn that correctly sets the data type for associative arrays, fixing the ${array[@]} expansion yielding random numbers. Thanks to @JohnoKing for the pointer. https://github.com/ksh93/ksh/issues/87#issuecomment-662613887 https://www.mail-archive.com/ast-developers@lists.research.att.com/msg00697.html src/cmd/ksh93/tests/enum.sh: - Add tests checking that invalid values are correctly blocked for indexed and associative arrays of an enum type. Makes progress on: https://github.com/ksh93/ksh/issues/87
2021-02-01 16:19:04 +00:00
- Fixed: scalar arrays (-a) and associative arrays (-A) of a type created by
'enum' allowed values not specified by the enum type, corrupting results.
- Fixed: the "${array[@]}" expansion for associative arrays of a type created
by 'enum' expanded to random numbers instead of the array's values.
command -x: fix efficiency; always run external cmd (re: acf84e96) This commit fixes 'command -x' to adapt to OS limitations with regards to data alignment in the arguments list. A feature test is added that detects if the OS aligns the argument on 32-bit or 64-bit boundaries or not at all, allowing 'command -x' to avoid E2BIG errors while maximising efficiency. Also, as of now, 'command -x' is a way to bypass built-ins and run/query an external command. Built-ins do not limit the length of their argument list, so '-x' never made sense to use for them. And because '-x' hangs on Linux and macOS on every ksh93 release version to date (see acf84e96), few use it, so there is little reason not to make this change. Finally, this fixes a longstanding bug that caused the minimum exit status of 'command -x' to be 1 if a command with many arguments was divided into several command invocations. This is done by replacing broken flaggery with a new SH_XARG state flag bit. src/cmd/ksh93/features/externs: - Add new C feature test detecting byte alignment in args list. The test writes a #define ARG_ALIGN_BYTES with the amount of bytes the OS aligns arguments to, or zero for no alignment. src/cmd/ksh93/include/defs.h: - Add new SH_XARG state bit indicating 'command -x' is active. src/cmd/ksh93/sh/path.c: path_xargs(): - Leave extra 2k in the args buffer instead of 1k, just to be sure; some commands add large environment variables these days. - Fix a bug in subtracting the length of existing arguments and environment variables. 'size -= strlen(cp)-1;' subtracts one less than the size of cp, which makes no sense; what is necessary is to substract the length plus one to account for the terminating zero byte, i.e.: 'size -= strlen(cp)+1'. - Use the ARG_ALIGN_BYTES feature test result to match the OS's data alignment requirements. - path_spawn(): E2BIG: Change to checking SH_XARG state bit. src/cmd/ksh93/bltins/whence.c: b_command(): - Allow combining -x with -p, -v and -V with the expected results by setting P_FLAG to act like 'whence -p'. E.g., as of now, command -xv printf is equivalent to whence -p printf but note that 'whence' has no equivalent of 'command -pvx printf' which searches $(getconf PATH) for a command. - When -x will run a command, now set the new SH_XARG state flag. src/cmd/ksh93/sh/xec.c: sh_exec(): - Change to using the new SH_XARG state bit. - Skip the check for built-ins if SH_XARG is active, so that 'command -x' now always runs an external command. src/lib/libcmd/date.c, src/lib/libcmd/uname.c: - These path-bound builtins sometimes need to run the external system command by the same name, but they did that by hardcoding an unportable direct path. Now that 'command -x' runs an external command, change this to using 'command -px' to guarantee using the known-good external system utility in the default PATH. - In date.c, fix the format string passed to 'command -px date' when setting the date; it was only compatible with BSD systems. Use the POSIX variant on non-BSD systems.
2021-01-30 05:51:22 +00:00
2021-01-30:
- The -x option to the 'command' built-in now causes it to bypass built-ins
so that it always runs/queries an external command. See 'command --man'.
- Fixed a bug in 'command -x' that caused the minimum exit status to be 1 if
a command with many arguments was divided into several command invocations.
- The 2020-08-16 fix is improved with a compile-time feature test that
Correctly block invalid values for arrays of an enum type This fixes part of https://github.com/ksh93/ksh/issues/87: Scalar arrays (-a) and associative arrays (-A) of a type created by 'enum' did not consistently block values not specified by the enum type, yielding corrupted results. An expansion of type "${array[@]}" yielded random numbers instead of values for associative arrays of a type created by 'enum'. This does not yet fix another problem: ${array[@]} does not yield all values for associative enum arrays. src/cmd/ksh93/bltins/enum.c: put_enum(): - Always throw an error if the value is not in the list of possible values for an enum type. Remove incorrect check for the NV_NOFREE flag. Whatever that was meant to accomplish, I've no idea. src/cmd/ksh93/sh/array.c: nv_arraysettype(): - Instead of sh_eval()ing a shell assignment, use nv_putval() directly. Also use the stack (see src/lib/libast/man/stk.3) instead of malloc to save the value; it's faster and will be auto-freed at some point. This shortens the function and makes it faster by not entering into a whole new shell context -- which also fixes another problem: the error message from put_enum() didn't cause the shell to exit for indexed enum arrays. src/cmd/ksh93/sh/name.c: nv_setlist(): - Apply a patch from David Korn that correctly sets the data type for associative arrays, fixing the ${array[@]} expansion yielding random numbers. Thanks to @JohnoKing for the pointer. https://github.com/ksh93/ksh/issues/87#issuecomment-662613887 https://www.mail-archive.com/ast-developers@lists.research.att.com/msg00697.html src/cmd/ksh93/tests/enum.sh: - Add tests checking that invalid values are correctly blocked for indexed and associative arrays of an enum type. Makes progress on: https://github.com/ksh93/ksh/issues/87
2021-02-01 16:19:04 +00:00
detects if the OS requires extra bytes per argument in the arguments list,
command -x: fix efficiency; always run external cmd (re: acf84e96) This commit fixes 'command -x' to adapt to OS limitations with regards to data alignment in the arguments list. A feature test is added that detects if the OS aligns the argument on 32-bit or 64-bit boundaries or not at all, allowing 'command -x' to avoid E2BIG errors while maximising efficiency. Also, as of now, 'command -x' is a way to bypass built-ins and run/query an external command. Built-ins do not limit the length of their argument list, so '-x' never made sense to use for them. And because '-x' hangs on Linux and macOS on every ksh93 release version to date (see acf84e96), few use it, so there is little reason not to make this change. Finally, this fixes a longstanding bug that caused the minimum exit status of 'command -x' to be 1 if a command with many arguments was divided into several command invocations. This is done by replacing broken flaggery with a new SH_XARG state flag bit. src/cmd/ksh93/features/externs: - Add new C feature test detecting byte alignment in args list. The test writes a #define ARG_ALIGN_BYTES with the amount of bytes the OS aligns arguments to, or zero for no alignment. src/cmd/ksh93/include/defs.h: - Add new SH_XARG state bit indicating 'command -x' is active. src/cmd/ksh93/sh/path.c: path_xargs(): - Leave extra 2k in the args buffer instead of 1k, just to be sure; some commands add large environment variables these days. - Fix a bug in subtracting the length of existing arguments and environment variables. 'size -= strlen(cp)-1;' subtracts one less than the size of cp, which makes no sense; what is necessary is to substract the length plus one to account for the terminating zero byte, i.e.: 'size -= strlen(cp)+1'. - Use the ARG_ALIGN_BYTES feature test result to match the OS's data alignment requirements. - path_spawn(): E2BIG: Change to checking SH_XARG state bit. src/cmd/ksh93/bltins/whence.c: b_command(): - Allow combining -x with -p, -v and -V with the expected results by setting P_FLAG to act like 'whence -p'. E.g., as of now, command -xv printf is equivalent to whence -p printf but note that 'whence' has no equivalent of 'command -pvx printf' which searches $(getconf PATH) for a command. - When -x will run a command, now set the new SH_XARG state flag. src/cmd/ksh93/sh/xec.c: sh_exec(): - Change to using the new SH_XARG state bit. - Skip the check for built-ins if SH_XARG is active, so that 'command -x' now always runs an external command. src/lib/libcmd/date.c, src/lib/libcmd/uname.c: - These path-bound builtins sometimes need to run the external system command by the same name, but they did that by hardcoding an unportable direct path. Now that 'command -x' runs an external command, change this to using 'command -px' to guarantee using the known-good external system utility in the default PATH. - In date.c, fix the format string passed to 'command -px date' when setting the date; it was only compatible with BSD systems. Use the POSIX variant on non-BSD systems.
2021-01-30 05:51:22 +00:00
maximising the efficiency of 'command -x' for the system it runs on.
Fix field splitting bug triggered by DEBUG trap An unquoted variable expansion evaluated in a DEBUG trap action caused IFS field splitting to be deactivated in code executed after the trap action. Thanks to Koichi Nakashima for the reproducer: | v='' | trap ': $v' DEBUG | A="a b c" | set -- $A | printf '%s\n' "$@" | | Expected | | a | b | c | | Actual | | a b c src/cmd/ksh93/sh/fault.c: sh_trap(): - Remove incorrect save/restore of sh.ifstable, the internal state table for field splitting. This reverts three lines added in ksh 93t+ 2009-11-30. Analysis: As an expansion is split into fields (macro.c, lines 2367-2471), sh.ifstable is modified. If that happens within a DEBUG trap, any modifications in ifstable are undone by the restoring memccpy, leaving an inconsistent state. src/cmd/ksh93/COMPATIBILITY: - Document the DEBUG trap fixes, particularly the incorrect inheritance by subshells and functions that some scripts may now rely on because this bug is so longstanding. (re: 2a835a2d) src/cmd/ksh93/tests/basic.sh: - Add relevant tests. Resolves: https://github.com/ksh93/ksh/issues/155 TODO: add a -T (-o functrace) option as in bash, which should allow subshells and ksh-style functions to inherit DEBUG traps. P.S.: The very handy multishell repo allows us to use 'git blame' to trace the origin of the recently fixed DEBUG trap bugs. The off-by-one error causing various bugs, reverted in 2a835a2d, was introduced in ksh 93t 2008-07-25: https://github.com/multishell/ksh93/commit/8e947ccf (fault.c, line 321) The incorrect check causing the exit status bug, reverted in d00b4b39, was introduced in ksh 93t 2008-11-04: https://github.com/multishell/ksh93/commit/b1ade268 (fault.c, line 459) The ifstable save/restore causing the field splitting bug, reverted in this commit, was introduced in ksh 93t+ 2009-11-30: https://github.com/multishell/ksh93/commit/53d9f009 (fault.c, lines 440, 444, 482) So all the bugs reported in #155 were fixed by simply reverting these specific changes. I think that they are some experiments that the developers simply forgot to remove. I've suspected such a thing multiple times before. ksh93 was developed by researchers who were genius innovators, but incredibly sloppy maintainers.
2021-01-24 15:46:46 +00:00
2021-01-24:
typeset: Correct numeric attribute change for floating points (#163) This commit resolves the following incorrect variable assignments: $ unset a; typeset -uF a=2; typeset -p a typeset -X a=0x1.0000000000p+1 $ unset a; typeset -Fu a=2; typeset -p a typeset -X a=0x1.0000000000p+1 $ unset a; typeset -ulF a=2; typeset -p a typeset -l -X a=0x1.0000000000p+1 $ unset a; typeset -Ful a=2; typeset -p a typeset -l -X a=0x1.0000000000p+1 $ unset a; typeset -Eu a=2; typeset -p a typeset -E -X a=2 $ unset a; typeset -Eul a=2; typeset -p a typeset -l -E -X a=2 src/cmd/ksh93/bltins/typeset.c: - If the unsigned option (-u) was provided in conjunction with a floating point (-F) then due to a flag collision with NV_UNSIGN and NV_HEXFLOAT both having the value of NV_LTOU caused the floating point to become a hexadecimal floating point (-X) in error. Also, if a -E option flag was followed with a -u option then the resulting variable would be both a scientific notation and a hexadecimal floating point at the same time. src/cmd/ksh93/tests/attributes.sh: - Add regression tests. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-01-24 22:45:08 +00:00
- Fixed a bug in 'typeset': combining the -u option with -F or -E caused the
variable to become a hexadecimal floating point in error.
Fix field splitting bug triggered by DEBUG trap An unquoted variable expansion evaluated in a DEBUG trap action caused IFS field splitting to be deactivated in code executed after the trap action. Thanks to Koichi Nakashima for the reproducer: | v='' | trap ': $v' DEBUG | A="a b c" | set -- $A | printf '%s\n' "$@" | | Expected | | a | b | c | | Actual | | a b c src/cmd/ksh93/sh/fault.c: sh_trap(): - Remove incorrect save/restore of sh.ifstable, the internal state table for field splitting. This reverts three lines added in ksh 93t+ 2009-11-30. Analysis: As an expansion is split into fields (macro.c, lines 2367-2471), sh.ifstable is modified. If that happens within a DEBUG trap, any modifications in ifstable are undone by the restoring memccpy, leaving an inconsistent state. src/cmd/ksh93/COMPATIBILITY: - Document the DEBUG trap fixes, particularly the incorrect inheritance by subshells and functions that some scripts may now rely on because this bug is so longstanding. (re: 2a835a2d) src/cmd/ksh93/tests/basic.sh: - Add relevant tests. Resolves: https://github.com/ksh93/ksh/issues/155 TODO: add a -T (-o functrace) option as in bash, which should allow subshells and ksh-style functions to inherit DEBUG traps. P.S.: The very handy multishell repo allows us to use 'git blame' to trace the origin of the recently fixed DEBUG trap bugs. The off-by-one error causing various bugs, reverted in 2a835a2d, was introduced in ksh 93t 2008-07-25: https://github.com/multishell/ksh93/commit/8e947ccf (fault.c, line 321) The incorrect check causing the exit status bug, reverted in d00b4b39, was introduced in ksh 93t 2008-11-04: https://github.com/multishell/ksh93/commit/b1ade268 (fault.c, line 459) The ifstable save/restore causing the field splitting bug, reverted in this commit, was introduced in ksh 93t+ 2009-11-30: https://github.com/multishell/ksh93/commit/53d9f009 (fault.c, lines 440, 444, 482) So all the bugs reported in #155 were fixed by simply reverting these specific changes. I think that they are some experiments that the developers simply forgot to remove. I've suspected such a thing multiple times before. ksh93 was developed by researchers who were genius innovators, but incredibly sloppy maintainers.
2021-01-24 15:46:46 +00:00
- Fixed: an unquoted variable expansion evaluated in a DEBUG trap action caused
IFS field splitting to be deactivated in code executed after the trap action.
This bug was introduced in ksh 93t+ 2009-11-30.
Fix restoring DEBUG trap upon exiting virtual subshell This trap failed to be restored correctly when being trapped in a subshell, causing corruption or a crash when restoring the parent shell environment's trap upon leaving the subshell. Thanks to Koichi Nakashima for the report and reproducer. src/cmd/ksh93/sh/fault.c: sh_sigreset(): - Fix an off-by-one error in the loop that restores the pseudosignal traps. src/cmd/ksh93/tests/basic.sh: - Test overwriting the main shell trap in a subshell for all pseudosignals. Makes progress on: https://github.com/ksh93/ksh/issues/155
2021-01-24 00:49:36 +00:00
2021-01-23:
- Fixed: when the DEBUG trap was redefined in a subshell, the DEBUG trap in
the parent environment was corrupted or the shell crashed.
Add regress test for redirection in DEBUG trap action (re: 2a835a2d) Turns out the previous commit also fixed the bug that disables the DEBUG trap if a redirection is used in a DEBUG trap action -- in other words, that's the same bug. src/cmd/ksh93/tests/basic.sh: - Add test from the reproducer in the bug report. Makes progress on: https://github.com/ksh93/ksh/issues/155
2021-01-24 03:51:00 +00:00
When a redirection was used in a DEBUG trap action, the trap was disabled.
Fix field splitting bug triggered by DEBUG trap An unquoted variable expansion evaluated in a DEBUG trap action caused IFS field splitting to be deactivated in code executed after the trap action. Thanks to Koichi Nakashima for the reproducer: | v='' | trap ': $v' DEBUG | A="a b c" | set -- $A | printf '%s\n' "$@" | | Expected | | a | b | c | | Actual | | a b c src/cmd/ksh93/sh/fault.c: sh_trap(): - Remove incorrect save/restore of sh.ifstable, the internal state table for field splitting. This reverts three lines added in ksh 93t+ 2009-11-30. Analysis: As an expansion is split into fields (macro.c, lines 2367-2471), sh.ifstable is modified. If that happens within a DEBUG trap, any modifications in ifstable are undone by the restoring memccpy, leaving an inconsistent state. src/cmd/ksh93/COMPATIBILITY: - Document the DEBUG trap fixes, particularly the incorrect inheritance by subshells and functions that some scripts may now rely on because this bug is so longstanding. (re: 2a835a2d) src/cmd/ksh93/tests/basic.sh: - Add relevant tests. Resolves: https://github.com/ksh93/ksh/issues/155 TODO: add a -T (-o functrace) option as in bash, which should allow subshells and ksh-style functions to inherit DEBUG traps. P.S.: The very handy multishell repo allows us to use 'git blame' to trace the origin of the recently fixed DEBUG trap bugs. The off-by-one error causing various bugs, reverted in 2a835a2d, was introduced in ksh 93t 2008-07-25: https://github.com/multishell/ksh93/commit/8e947ccf (fault.c, line 321) The incorrect check causing the exit status bug, reverted in d00b4b39, was introduced in ksh 93t 2008-11-04: https://github.com/multishell/ksh93/commit/b1ade268 (fault.c, line 459) The ifstable save/restore causing the field splitting bug, reverted in this commit, was introduced in ksh 93t+ 2009-11-30: https://github.com/multishell/ksh93/commit/53d9f009 (fault.c, lines 440, 444, 482) So all the bugs reported in #155 were fixed by simply reverting these specific changes. I think that they are some experiments that the developers simply forgot to remove. I've suspected such a thing multiple times before. ksh93 was developed by researchers who were genius innovators, but incredibly sloppy maintainers.
2021-01-24 15:46:46 +00:00
DEBUG traps were also incorrectly inherited by subshells and ksh functions.
All this was caused by a bug introduced in ksh 93t 2008-07-25.
Fix restoring DEBUG trap upon exiting virtual subshell This trap failed to be restored correctly when being trapped in a subshell, causing corruption or a crash when restoring the parent shell environment's trap upon leaving the subshell. Thanks to Koichi Nakashima for the report and reproducer. src/cmd/ksh93/sh/fault.c: sh_sigreset(): - Fix an off-by-one error in the loop that restores the pseudosignal traps. src/cmd/ksh93/tests/basic.sh: - Test overwriting the main shell trap in a subshell for all pseudosignals. Makes progress on: https://github.com/ksh93/ksh/issues/155
2021-01-24 00:49:36 +00:00
Build system: make SHOPT_* editable again; allow indenting Mamfiles The build system is adapted to make SHOPT_* compile-time options editable without nmake. We can now easily change ksh's compile-time options by editing src/cmd/ksh93/SHOPT.sh. The bin/package script is adapted to turn these into compile flags. This resolves the most important drawback of not using nmake. Also, mamake now has support for indented Mam (Make Abstract Machine) code. Only one type of block (make...done) is supported in Mamfiles, so they are easy to indent automatically. A script to (re)do this is included. Since nmake is not going to be restored (it has too many problems that no one is interested in fixing), this at least makes mamake significantly easier to work with. The Makefiles are deleted. They may still be handy for reference to understand the Mamfiles, but they haven't actually matched the Mamfiles for a while -- and you can still look in the git history. Deleting them requires some adaptations to bin/package and mamake.c because, even though they do not use those files, they still looked for them to decide whether to build code in a directory. Finally, this commit incorporates some #pragmas for clang to suppress annoying warnings about the coding style used in this historic code base. (gcc does not complain so much.) src/cmd/ksh93/SHOPT.sh: - Added. bin/package, src/cmd/INIT/package.sh: - cd into our own directory in case we were run from another dir. - $makefiles: only look for Mamfiles. - Add ksh compile-options via KSH_SHOPTFLAGS. Include SHOPT.sh. - make_recurse(): Do not write a missing Makefile. - finalize environment: Look for Mamfiles instead of Makefiles. src/cmd/INIT/mamake.c: - Tell clang to suppress annoying warnings about coding style. - Update version string and self-documentation. - input(): Add support for indented Mam code by skipping initial whitespace on each input line. - files[]: Instead of looking for various of Makefiles to decide where to build, only look for Mamfiles. src/Makefile, src/cmd/INIT/Makefile, src/cmd/Makefile, src/cmd/builtin/Makefile, src/cmd/ksh93/Makefile, src/lib/Makefile, src/lib/libast/Makefile, src/lib/libcmd/Makefile, src/lib/libdll/Makefile, src/lib/libsum/Makefile: - Removed. src/Mamfile, src/cmd/INIT/Mamfile, src/cmd/Mamfile, src/cmd/builtin/Mamfile, src/cmd/ksh93/Mamfile, src/lib/Mamfile, src/lib/libast/Mamfile, src/lib/libcmd/Mamfile, src/lib/libdll/Mamfile, src/lib/libsum/Mamfile: - Indent the code with tabs. - In ksh93/Mamfile, add ${KSH_SHOPT_FLAGS} to every $CC command. - In ksh93/Mamfile, add "prev SHOPT.sh" for every *.o file so they are rebuilt whenever SHOPT.sh changes. bin/Mamfile_indent: - Added, in case someone wants to re-indent a Mamfile. src/cmd/INIT/proto.c, src/cmd/INIT/ratz.c, src/cmd/INIT/release.c, src/lib/libast/features/common, src/lib/libast/include/ast.h: - Tell clang to suppress annoying warnings about coding style that it disapproves of (mainly concerning the use of parentheses). src/cmd/INIT/cc.darwin, src/cmd/INIT/cc.freebsd, src/cmd/INIT/cc.openbsd: - Remove now-redundant clang warning suppression flags. Resolves: https://github.com/ksh93/ksh/issues/60
2021-01-22 23:23:14 +00:00
2021-01-22:
- Compile-time shell options can now be edited in src/cmd/ksh93/SHOPT.sh
before building.
Fix side effect to exit status of DEBUG trap in comsub This fixes the following: trap ':' DEBUG r=$(exit 123) echo $? # Expected 123, but actually 0. Thanks to Koichi Nakashima for the report and reproducer. src/cmd/ksh93/sh/fault.c: sh_trap(): - Restore the saved current exit status (exitval) for all traps. Do not except the DEBUG trap from doing that. I've no idea why this exception was made, but it's not correct. src/cmd/ksh93/tests/basic.sh: - Add tests. Makes progress on: https://github.com/ksh93/ksh/issues/155
2021-01-20 17:40:09 +00:00
2021-01-20:
- Fixed: executing a DEBUG trap in a command substitution had side effects
on the exit status ($?) of non-trap commands.
Fix field splitting bug triggered by DEBUG trap An unquoted variable expansion evaluated in a DEBUG trap action caused IFS field splitting to be deactivated in code executed after the trap action. Thanks to Koichi Nakashima for the reproducer: | v='' | trap ': $v' DEBUG | A="a b c" | set -- $A | printf '%s\n' "$@" | | Expected | | a | b | c | | Actual | | a b c src/cmd/ksh93/sh/fault.c: sh_trap(): - Remove incorrect save/restore of sh.ifstable, the internal state table for field splitting. This reverts three lines added in ksh 93t+ 2009-11-30. Analysis: As an expansion is split into fields (macro.c, lines 2367-2471), sh.ifstable is modified. If that happens within a DEBUG trap, any modifications in ifstable are undone by the restoring memccpy, leaving an inconsistent state. src/cmd/ksh93/COMPATIBILITY: - Document the DEBUG trap fixes, particularly the incorrect inheritance by subshells and functions that some scripts may now rely on because this bug is so longstanding. (re: 2a835a2d) src/cmd/ksh93/tests/basic.sh: - Add relevant tests. Resolves: https://github.com/ksh93/ksh/issues/155 TODO: add a -T (-o functrace) option as in bash, which should allow subshells and ksh-style functions to inherit DEBUG traps. P.S.: The very handy multishell repo allows us to use 'git blame' to trace the origin of the recently fixed DEBUG trap bugs. The off-by-one error causing various bugs, reverted in 2a835a2d, was introduced in ksh 93t 2008-07-25: https://github.com/multishell/ksh93/commit/8e947ccf (fault.c, line 321) The incorrect check causing the exit status bug, reverted in d00b4b39, was introduced in ksh 93t 2008-11-04: https://github.com/multishell/ksh93/commit/b1ade268 (fault.c, line 459) The ifstable save/restore causing the field splitting bug, reverted in this commit, was introduced in ksh 93t+ 2009-11-30: https://github.com/multishell/ksh93/commit/53d9f009 (fault.c, lines 440, 444, 482) So all the bugs reported in #155 were fixed by simply reverting these specific changes. I think that they are some experiments that the developers simply forgot to remove. I've suspected such a thing multiple times before. ksh93 was developed by researchers who were genius innovators, but incredibly sloppy maintainers.
2021-01-24 15:46:46 +00:00
This bug was introduced in ksh 93t 2008-11-04.
Fix side effect to exit status of DEBUG trap in comsub This fixes the following: trap ':' DEBUG r=$(exit 123) echo $? # Expected 123, but actually 0. Thanks to Koichi Nakashima for the report and reproducer. src/cmd/ksh93/sh/fault.c: sh_trap(): - Restore the saved current exit status (exitval) for all traps. Do not except the DEBUG trap from doing that. I've no idea why this exception was made, but it's not correct. src/cmd/ksh93/tests/basic.sh: - Add tests. Makes progress on: https://github.com/ksh93/ksh/issues/155
2021-01-20 17:40:09 +00:00
typeset: add error msgs for incompatible options; improve usage msg This adds informative error messages if incompatible options are given. It also documents the exclusive -m, -n and -T options on separate usage lines, as was already done with -f. The usage message for incompatible options now looks something like this: | $ ksh -c 'typeset -L10 -F -f -i foo' | ksh: typeset: -i/-F/-E/-X cannot be used with -L/-R/-Z | ksh: typeset: -f cannot be used with other options | Usage: typeset [-bflmnprstuxACHS] [-a[type]] [-i[base]] [-E[n]] | [-F[n]] [-L[n]] [-M[mapping]] [-R[n]] [-X[n]] | [-h string] [-T[tname]] [-Z[n]] [name[=value]...] | Or: typeset -f [name...] | Or: typeset -m [name=name...] | Or: typeset -n [name=name...] | Or: typeset -T [tname[=(type definition)]...] | Help: typeset [ --help | --man ] 2>&1 (see also the previous commit, e21a053e) Unfortunately the first "Usage" line has some redundancies with the "Or:" lines showing separate usages. It doesn't seem to be possible to avoid this; it's a flaw in how libast generates everything (usage, help, manual) from one huge getopt(3) string. I still think the three added "Or:" lines are an improvement as it wasn't previously shown that these options need to be used on their own. src/cmd/ksh93/bltins/typeset.c: b_typeset(): - Instead of only showing a generic usage message, add an informative error message if incompatible options were given. - Conflicting options detection was failing because NV_LJUST and NV_EXPNOTE have the same bitmask value. Use a new 'isadjust' flag for -L/-R/-Z to remember if one of these was set. - Detect conflict between -L/-R/-Z and a float option, not just -i. src/cmd/ksh93/include/name.h, src/cmd/ksh93/data/msg.c: - Add the two new error messages for incompatible options. src/cmd/ksh93/data/builtins.c: sh_opttypeset[]: - Add a space after 'float' in in "[+float?\btypeset -lE\b]" as this makes 'float' appear on its own line, improving formatting. - Show -m, -n, -T on separate usage lines like -f, as none of these can be combined with other options. - Remove "cannot be combined with other options" from -m and -n descriptions, as that should now be clear from the separate usage lines -- and even if not, the error message is now informative. src/cmd/ksh93/sh.1, src/cmd/ksh93/COMPATIBILITY: - Update. src/cmd/ksh93/tests/types.sh: - Remove obsolete test: 'typeset -RF' is no longer accepted. (It crashed in 93u+, so this is not an incompatibility...) Resolves: https://github.com/ksh93/ksh/issues/48
2021-01-21 00:24:13 +00:00
- The typeset builtin command now gives an informative error message if an
incompatible combination of options is given.
Fix crash on subshell exit if PWD is inaccessible (re: dd9bc229) This commit also further mitigates the problems with restoring an inaccessible or nonexistent PWD on exiting a virtual subshell. Harald van Dijk writes: > On a build of ksh with -fsanitize=undefined to help diagnose > problems: > > $ mkdir deleted > $ cd deleted > $ rmdir ../deleted > $ ksh -c '(cd /; (cd /)); :' > /home/harald/ksh/src/cmd/ksh93/sh/subshell.c:561:22: runtime > error: null pointer passed as argument 1, which is declared to > never be null > Segmentation fault (core dumped) > > Note that it segfaults the same with default compilation flags, > but it does not print out the useful extra message. The code > assumes that pwd is non-null and passes it to strcmp without > checking, but it will be null if the current directory cannot be > determined, for instance because it has been deleted. src/cmd/ksh93/sh/subshell.c: sh_subshell(): - Avoid the null pointer dereference reported above. src/cmd/ksh93/bltins/cd_pwd.c: b_cd(): - Fork a virtual subshell even on systems with fchdir(2) if the present working directory tests as inaccessible on invoking 'cd'; it may no longer exist and fchdir would fail to get a handle. (For the test we have to opendir(3) the full path to the PWD and not ".", as the latter may succeed even if the PWD is gone.) src/cmd/ksh93/data/builtins.c: - Update 'cd' version string. Fixes: https://github.com/ksh93/ksh/issues/153 Related: https://github.com/ksh93/ksh/issues/141
2021-01-19 18:47:41 +00:00
2021-01-19:
- Fixed a crash when using 'cd' in a virtual/non-forking subshell in a
situation where the current working directory cannot be determined.
Fix crash on long PS1 prompt (Solaris patch 195-17824699) Original report and info: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01677.html https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01679.html Patch pulled in from: https://raw.githubusercontent.com/oracle/solaris-userland/master/components/ksh93/patches/195-17824699.patch src/cmd/ksh93/edit/edit.c: ed_setup(): - Prevent the ed_setup() function from writing past ep->e_prompt, which is set to the local char prompt[PRSIZE] variable in ed_emacsread(). src/cmd/ksh93/include/edit.h: - Increase maximum prompt size, PRSIZE, to 256.
2021-01-08 22:15:48 +00:00
2021-01-08:
- Fixed a crash on exceeding the maximum size of the $PS1 prompt.
The maximum size is also increased from 160 to 256 bytes.
Fix segfault while updating ${.sh.match} The SHOPT_2DMATCH code block in sh_setmatch() modifies the 'ap' pointer, which is initialised as nv_arrayptr(SH_MATCHNOD). This caused a (rarely occurring) segfault in the following line near the end of the function: ap->nelem -= x; as this line assumed that 'ap' still had the initial value. src/cmd/ksh93/sh/init.c: sh_setmatch(): - On init, save ap in ap_save and use ap_save instead of ap where it should be pointing to SH_MATCHNOD. This also allows removing two redundant nv_arrayptr(SH_MATCHNOD) calls, slightly increasing the efficiency of this function.
2021-01-07 17:34:47 +00:00
2021-01-07:
- Fixed a crash that could occur while ksh updated ${.sh.match}.
Implement hash tables for virtual subshells (re: 102868f8, 9d428f8f) The forking fix implemented in 102868f8 and 9d428f8f, which stops the main shell's hash table from being cleared if PATH is changed in a subshell, can cause a significant performance penalty for certain scripts that do something like ( PATH=... command foo ) in a subshell, especially if done repeatedly. This is because the hash table is cleared (and hence a subshell forks) even for temporary PATH assignments preceding commands. It also just plain doesn't work. For instance: $ hash -r; (ls) >/dev/null; hash ls=/bin/ls Simply running an external command in a subshell caches the path in the hash table that is shared with a main shell. To remedy this, we would have to fork the subshell before forking any external command. And that would be an unacceptable performance regression. Virtual subshells do not need to fork when changing PATH if they get their own hash tables. This commit adds these. The code for alias subshell trees (which was removed in ec888867 because they were broken and unneeded) provided the beginning of a template for their implementation. src/cmd/ksh93/sh/subshell.c: - struct subshell: Add strack pointer to subshell hash table. - Add sh_subtracktree(): return pointer to subshell hash table. - sh_subfuntree(): Refactor a bit for legibility. - sh_subshell(): Add code for cleaning up subshell hash table. src/cmd/ksh93/sh/name.c: - nv_putval(): Remove code to fork a subshell upon resetting PATH. - nv_rehash(): When in a subshell, invalidate a hash table entry for a subshell by creating the subshell scope if needed, then giving that entry the NV_NOALIAS attribute to invalidate it. src/cmd/ksh93/sh/path.c: path_search(): - To set a tracked alias/hash table entry, use sh_subtracktree() and pass the HASH_NOSCOPE flag to nv_search() so that any new entries are added to the current subshell table (if any) and do not influence any parent scopes. src/cmd/ksh93/bltins/typeset.c: b_alias(): - b_alias(): For hash table entries, use sh_subtracktree() instead of forking a subshell. Keep forking for normal aliases. - setall(): To set a tracked alias/hash table entry, pass the HASH_NOSCOPE flag to nv_search() so that any new entries are added to the current subshell table (if any) and do not influence any parent scopes. src/cmd/ksh93/sh/init.c: put_restricted(): - Update code for clearing the hash table (when changing $PATH) to use sh_subtracktree(). src/cmd/ksh93/bltins/cd_pwd.c: - When invalidating path name bindings to relative paths, use the subshell hash tree if applicable by calling sh_subtracktree(). - rehash(): Call nv_rehash() instead of _nv_unset()ting the hash table entry; this is needed to work correctly in subshells. src/cmd/ksh93/tests/leaks.sh: - Add leak tests for various PATH-related operations in the main shell and in a virtual subshell. - Several pre-existing memory leaks are exposed by the new tests (I've confirmed these in 93u+). The tests are disabled and marked TODO for now, as these bugs have not yet been fixed. src/cmd/ksh93/tests/subshell.sh: - Update. Resolves: https://github.com/ksh93/ksh/issues/66
2021-01-07 21:18:33 +00:00
- Any changes to the hash table (a.k.a. "tracked aliases", i.e. cached $PATH
searches) in a subshell now no longer affect the parent shell's hash table.
'cd': properly ignore $CDPATH if initial component is '.' or '..' @stephane-chazelas writes: > Per POSIX[*], cd should skip the $CDPATH processing if the first > component of the directory given to cd is . or ... > > Yet, with ksh93u+m 2021-01-03 at least, while that's OK with .., > it's not with . with or without the posix option: > > $ CDPATH=/ ./ksh -o posix -c 'cd -P ./etc && pwd' > /etc > /etc > > It seems to be a regression introduced with ksh93u+ as I can't > reproduce it with ksh93u or any version prior to that. I can also > reproduce in u+, v- and the ksh2020 from the Ubuntu 20.04 > package. src/cmd/ksh93/bltins/cd_pwd.c: b_cd(): - Skip $CDPATH processing not only if the path is absolute, but also if the initial path component is '.' or '..' (in the latter case the $CDPATH processing was done but appeared to be a no-op). src/cmd/ksh93/tests/builtins.sh: - Add regression test. [*] https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/utilities/cd.html Fixes: https://github.com/ksh93/ksh/issues/151
2021-01-05 04:55:07 +00:00
2021-01-05:
- Fixed a bug in 'cd' that caused 'cd ./foo' to search for 'foo' in $CDPATH.
Invocation: fix infinite loop on 'ksh +s' When starting ksh +s, it gets stuck in an infinite loop continually trying to parse its own binary as a shell script and rejecting it: $ arch/linux.i386-64/bin/ksh +s arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error] arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error] arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error] arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error] arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error] [...] $ echo 'echo "this is stdin"' | arch/linux.i386-64/bin/ksh +s arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error] (no loop, but still ksh trying to parse itself) src/cmd/ksh93/sh/init.c: sh_init(): - When forcing on the '-s' option upon finding no command arguments, also update sh.offoptions, a.k.a. shp->offoptions. This avoids the inconsistent state causing this problem. In main.c, there is: if(sh_isoption(SH_SFLAG)) fdin = 0; else (code to open $0 as a file) This was entering the else block because sh_isoption(SH_SFLAG) was returning 0, and $0 is set to the ksh binary as it is supposed to when no other script is provided. When I looked for why sh_isoption was returning 0, I found main.c's for(i=0; i<elementsof(shp->offoptions.v); i++) shp->options.v[i] &= ~shp->offoptions.v[i]; Before this loop, shp->offoptions tracks which options were explicitly disabled by the user on the command line. The effect of this loop is to make "explicitly disabled" take precedence over "implicitly enabled". My patch removes the registration of the +s option. Fixes: https://github.com/ksh93/ksh/issues/150 Co-authored-by: Martijn Dekker <martijn@inlv.org>
2021-01-03 23:54:36 +00:00
2021-01-03:
- The invocation
$ ksh +s
caused an infinite loop and corrupted ~/.sh_history. This is now fixed so
that the '-s' option is automatically turned on if there are no non-option
command arguments, as documented in Bolsky & Korn (1995), p. 261.
Fix floating point numerics having precision of 0 with assignments (#149) Issuing typeset floating point numerics having a precision of 0 failed as the precision/size was being overwritten with the string length of the value, e.g. 'typeset -F0 x=5.67' would result in 'typeset -F 4 x=5.6700' as len('5.67') is 4. src/cmd/ksh93/include/nval.h: - Created a symbolic name of NV_FLTSIZEZERO to respresent a float having a precision/size of 0. NV_FLTSIZEZERO needs to be a negative value. src/cmd/ksh93/bltins/typeset.c: - In b_typeset(), added code to set tdata.argnum to NV_FLTSIZEZERO for E, F, X options. - In setall(), adjusted code to allow for tp->argnum to be negative. src/cmd/ksh93/sh/name.c: nv_newattr(): - Adjusted option value only change code to handle NV_FLTSIZEZERO as well as changed to directly setting np->nvsize instead of using nv_setsize(np,size) as nv_setsize might contain conflicting and/or redundant code. - Added missing conditional check of '!(newatts&NV_INTEGER)' to constrain the size==0 code block to justified strings as NV_LJUST, NV_RJUST, or NV_ZFILL are only valid for strings if NV_INTEGER is not set. This code block was mistakenly setting the precision/size value to the length of the value of an assignment for floats whereas it should only be performing auto assignment length for justified strings.
2020-11-26 13:50:30 +00:00
2020-10-22:
- Fixed: 'typeset -F0', 'typeset -E0', and 'typeset -X0' floating point
numerics having a precision of 0 with variable assignment.
'typeset -F0 x; x=4.56' worked but not 'typeset -F0 x=4.56'.
Improved 'typeset -xu'/'typeset -xl' fix (re: fdb9781e) (#147) 'typeset -xu' and 'typeset -xl' would export the variable but fail to change case in the value as the check between old and new attributes did not provide the necesssary insight for lower or upper case transcoding due to the lower or upper case attribute being set within typeset.c prior to calling name.c nv_newattr function. Previous rhbz#1188377 patch added a conditional check for size==-1 which in effect caused the nv_newattr export code block return optimization to never be executed as one cannot set any attributes using the readonly builtin. By altering the size==-1 check to !trans the export only optimization can run. Also, the rhbz#1188377 patch altered new_attr function by setting the new size to oldsize if run by the readonly builtin. The result of setting size==oldsize allowed the succeeding if statement to run more frequently and if size was a non-zero value resulted in nv_setsize resetting the value to what it already was. Investigation yielded that size was always 0 coming from the readonly builtin. src/cmd/ksh93/bltins/typeset.c: - Remove the setting of tdata.argnum to -1 as it is not needed due to existing name.c nv_newattr() logic. src/cmd/ksh93/sh/name.c: nv_newattr(): - Corrected the export only check optimization by using !trans instead of using size==-1. - Removed previous condition check to set size=oldsize if coming from the readonly builtin. nv_newattr already had existing logic to prevent changing the size via nv_setsize as size is always 0 when coming from readonly builtin.
2020-11-26 13:30:24 +00:00
2020-10-21:
- Fixed: More concisely correct the exporting of uppercase and lowercase
variables when only the export and change case attributes were applied.
This fix improves upon the previous 2020-09-30 modifications.
Mitigate PWD race condition in non-forking subshells Virtual/non-forking subshells that change the present working directory (PWD) with 'cd' suffer from a serious race condition. The PWD is changed within the same process. This means it may not be possible to change back to the original PWD when exiting the subshell, as some other process may destroy the PWD or modify its permissions in the meantime. ksh did not handle this error condition at all, so, after exiting a subshell that invoked 'cd', it could silently end up running the script's following command(s) in the wrong directory. Which might be 'rm -rf *'. So, ouch. The proper and obvious fix is never to allow a virtual subshell to change the PWD, as it can never be guaranteed you can return to a previous directory. If the PWD is changed in a child process, there is no need to restore it in the parent process, and this whole problem is avoided. So subshells really should always fork on encountering a 'cd' command. But forking is slow. It is not uncommon for scripts to 'cd' in a subshell that is run repeatedly in a loop. There is also the issue of custom builtins that can be added to ksh via shared libraries. In the standard shell language, 'cd' is the only command that changes the PWD, so we could just make that command fork the subshell it is run from. But there's no telling what a custom builtin might do. So this commit implements a compromise that will not affect performance unless there is the pathological condition of a PWD that has been rendered inaccessible in some way: 1. When entering a virtual subshell, if the parent shell's PWD proves inaccessible upon saving it, the subshell will now fork into a separate process, avoiding the unrestorable PWD problem. 2. If some attack renders the parent shell's PWD unrestorable *after* ksh enters a virtual subshell, ksh will now error out when exiting it. There is nothing else left to do then. Continuing would mean running arbitrary commands in the wrong PWD. src/cmd/ksh93/sh/subshell.c: - Put all the code/variables only needed for fchdir() behind '#if _lib_fchdir'. This makes it clearer what's what. (I don't know if there is still any system out there without fchdir(3); I haven't found any. The chdir(3) fallback version may be removed later as there is no way to make it remotely secure.) - Fix the attempt to use the O_PATH mode for open(2) as a fallback for nonexistent O_SEARCH on Linux. Define _GNU_SOURCE on Linux, or <fcntl.h> (which is included indirectly) won't define O_PATH. - Fix use of O_SEARCH. The code was simply wrong, repeating an open(".",O_RDONLY) instead. Since a nonexistent O_SEARCH is now redefined as either O_PATH or O_RDONLY, we can simply open(".",O_SEARCH) and be done with it. - Fix fatal error handling. Introduce fatal error condition for failure to fchdir(3) back to the parent's PWD; rename 'duped' to 'fatalerror' and use it for error numbers; save and restore errno on fatal error so the message will report the cause. (We must call errormsg() near the end of sh_subshell() to avoid crashes.) - If open(".",O_SEARCH) was not able get a file descriptor to our PWD on entry, then call sh_subfork() immediately before running the subshell commands. (Forking earlier causes a crash.) - When restoring the PWD, if fchdir(3) fails, do *not* fall back to chdir(3). We already know the PWD is inaccessible, so if chdir(3) "succeeds" then, it's very likely to be a substitute injected by an attacker. src/cmd/ksh93/bltins/cd_pwd.c: - If we don't have fchdir(3), then sh_subshell() must fall back to chdir(2) to restore the PWD. That is highly vulnerable, as a well-timed rename would allow an attacker to usurp the PWD. We can't do anything about that if some custom builtin changes the PWD, but we can at least make 'cd' always fork a subshell, which slows down ksh but removes the need for the parent shell ever to restore the PWD. (There is certainly no popular system where this is relevant and there might not be any such current system.) This commit adds no regression test because a portable regression test is not really doable. Different kernels, external /bin/pwd utilities, etc. all have quite different behaviour under the pathological condition of an inaccessible PWD, so both the before-fix and the after-fix behaviour differs. See link below. Resolves: https://github.com/ksh93/ksh/issues/141 Thanks to Stéphane Chazelas for the bug report.
2020-10-06 21:13:41 +00:00
2020-10-06:
- The security of virtual/non-forking subshells that locally change the present
working directory (PWD) using 'cd' has been improved in two ways.
1. On entering a subshell, if the parent shell's PWD proves inaccessible upon
saving it, the subshell will now fork into a separate process so the
parent process never changes its PWD, avoiding the need to restore it.
2. If some attack renders the parent shell's PWD unrestorable *after* ksh
enters a virtual subshell, ksh will now error out on exiting it, as
continuing would mean running arbitrary commands in the wrong PWD.
Hopefully this is an acceptable compromise between performance and security.
The proper fix would be to always fork a subshell when changing the working
directory within it, but the resulting slowdown would likely be unpopular.
Fix 'typeset -xu', 'typeset -xl' (rhbz#1188377) 'typeset -xu' and 'typeset -xl' would export the variable but fail to change case in the value under certain conditions. Original patch: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-xufix.patch This applies the patch essentially without change and adds a regression test based on the reproducer provided in the RH bug. Unfortunately there is no description of how the patch works and it's a little obscure to me. As far as I can figure out, the cause of the problem was that nv_newattr() erroneously processed a nonexistent size option-argument such as what can be given to options like typeset -F, e.g. typeset -F3 for 3 digits after the dot. A nonexistent size argument is represented by the value of -1.
2020-09-29 22:34:02 +00:00
2020-09-30:
- Fixed: 'typeset -xu' and 'typeset -xl' (export + change case) failed to
change the case of a variable's value in certain conditions.
Fix pipefail with (errexit or ERR trap) regression ksh 93u+ introduced a regression in the combination of the 'set -o pipefail' and 'set -e'/'set -o errexit' options: $ ksh93 -o errexit -o pipefail -c \ '(exit 3) | true; echo "still here despite $? status"' still here despite 3 status The bug is in how the the huge sh_exec() function in xec.c handles the 'echeck' flag. Near the end of sh_exec(), this flag triggers a sh_chktrap() call to check whether to trigger any traps, including the ERR trap -- and that same function also handles the errexit option, which is basically the same as 'trap "exit" ERR'. We can learn more easily how sh_exec() works by inserting debug warnings in all its 'switch(type&COMMSK)' cases, like: case TCOM: errormsg(SH_DICT,ERROR_warn(0),"[DEBUG] TCOM"); ... and same for all the others. With that done, the output of a very simple dummy pipeline looks as follows: $ arch/*/bin/ksh -c 'true | true | true' arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TFIL arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TFORK arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TFORK arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TSETIO arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TCOM arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TCOM arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TCOM So, it looks like sh_exec() handles this pipeline as follows: TFIL |_____TFORK | |_____TCOM |_____TFORK | |_____TCOM |_____TSETIO |_____TCOM Each time a pipeline like command1 | command2 | ... is executed, sh_exec() is invoked with type TFIL; this then recursively invokes sh_exec() to handle the individual elements. The last element of the pipe triggers a sh_exec() run with type TSETIO; since it is run in the current shell environment, it is effectively treated as a command with an input redirection. All the previous elements are of type TFORK instead, because they are executed asynchronously in separate, forked subshell processes. Finally, the TFORK or TSETIO code then recursively calls sh_exec() again with type TCOM to actually execute the commands. When reading the code, we find that the 'echeck' flag is set as part of the TSETIO code. This makes sense of why only an error in the last element of the pipe triggers the errexit/ERR trap action. So that's the bug: the flag is set in the wrong place. This can be fixed by setting that flag in the TFIL handling code instead, as this is what calls everything else and collects all the exit statuses. So the sh_chktrap() call is now executed after handling the entire pipeline, at the TFIL recursion level. This also allows getting rid of the special-casing in the buggy TSETIO version. The SH_ERREXIT state is restored at the end of each sh_exec() call, so since we're now doing this at a lower recursion level, it will already have been restored. src/cmd/ksh93/sh/xec.c: sh_exec(): - Fix the bug as per the above. src/cmd/ksh93/tests/options.sh: - Add tests for errexit and ERR trap combined with pipefail. src/cmd/ksh93/tests/basic.sh: - Tweak a couple of tests that reported a trap wasn't triggered even if it was actually triggered more than once. Fixes: https://github.com/ksh93/ksh/issues/121 Thanks to Stéphane Chazelas for the bug report.
2020-09-30 14:13:34 +00:00
- A ksh 93u+ regression was fixed in the combination of ERR trap handling and
the 'pipefail' option. A pipeline now triggers the ERR trap correctly again
if the 'pipefail' option is active and any of the pipeline elements return a
nonzero exit status. Similarly, if both the 'errexit' and 'pipefail' options
are active, ksh now correctly exits if any pipeline element returns nonzero.
Fix $LINENO corruption when autoloading functions Autoloading a function caused the calling script's $LINENO to be off by the number of lines in the function definition file. In addition, while running autoloaded functions, errors/warnings were reported with wrong line numbers. src/cmd/ksh93/sh/path.c: - Save $LINENO (shp->inlineno) before autoloading a function, reset it to 1 so that the correct line number offset is remembered for the function definition, and restore it after. src/cmd/ksh93/tests/variables.sh: - Add regression test for $LINENO, directly and in error messages, within and outside a non-autoloaded and an autoloaded function. Fixes: https://github.com/ksh93/ksh/issues/116
2020-10-01 04:13:00 +00:00
- Autoloading a function no longer causes the calling script's $LINENO to be
off by the number of lines in the function definition file that was loaded.
This also corrects line numbers in warnings and error messages.
Fix signal/trap behaviour in ksh functions (rhbz#1454804) Prior discussion: https://bugzilla.redhat.com/1454804 On 2017-05-23 13:33:25 UTC, Paulo Andrade wrote: > In previous ksh versions, when exiting the scope of a ksh > (not posix) function, it would restore the trap table of > the "calling context" and if the reason the function exited > was a signal, it would call sh_fault() passing as argument > the signal value. > Newer ksh checks it, but calls kill(getpid(), signal_number) > after restoring the trap table, but only calls for SIGINT and > SIGQUIT. [...] > The old way appears to have been more appropriate, but there > must be a reason to only pass SIGINT and SIGQUIT as it is an > explicit patch. The last paragraph is where I differ. This would not be the first example of outright breakage that appeared to be added deliberately and that 93u+m has fixed or removed, see e.g. 8477d2ce ('printf %H' had code that deleted all multibyte characters), cefe087d, or 781f0a39. Sometimes it seems the developers added a little experiment and then forgot all about it, so it became a misfeature. In this instance, the correct pre-2012 ksh behaviour is still explicitly documented in (k)sh.1: "A trap condition that is not caught or ignored by the function causes the function to terminate and the condition to be passed on to the caller". Meaning, if there is no function-local trap, the signal defaults to the parent scope. There is no language that limits this to SIGINT and SIGQUIT only. It also makes no sense at all to do so -- signals such as SIGPIPE, SIGTERM, or SIGSEGV need to be caught by default and to do otherwise results in misbehaviour by default. src/cmd/ksh93/sh/xec.c: sh_funscope(): - When resending a signal after restoring the global traps state, remove the spurious check that limits this to SIGINT and SIGQUIT. - Replace it with a check for nsig!=0, as that means there were parent trap states to restore. Otherwise 'kill' may be called with an invalid signal argument, causing a crash on macOS. src/cmd/ksh93/tests/signal.sh: - Update a test to check that a function-local SIGTERM trap is triggered correctly when signalled from another process. - Complete the tests for 3aee10d7; this bug needed fixing before we could test that previous fix in a ksh function scope. - Add a test for triggering global traps from ksh functions, testing multiple POSIX-standard signals.
2020-09-29 00:28:08 +00:00
2020-09-28:
- While executing a ksh-style function, ksh 93u+ ignored all signals for which
the function had not set a local trap, except for SIGINT and SIGQUIT. This
was contrary to the manual, which states that a "trap condition that is not
caught or ignored by the function causes the function to terminate and the
condition to be passed on to the caller". This has now been fixed in 93u+m to
match the documentation, so that e.g. global traps work as expected again.
Fix lexing of 'case' in do...done in a $(comsub) (rhbz#1241013) The following caused a spurious syntax error: $ x=$(for i in 1; do case $i in word) true;; esac; done) -ksh: syntax error: `;;' unexpected Prior discussion: https://bugzilla.redhat.com/1241013 Original patch, backported from 93v- beta, applied without change: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-parserfix.patch
2020-09-27 19:26:09 +00:00
2020-09-27:
NEWS: fix typo (re: bd283959)
2020-09-28 02:47:53 +00:00
- The shell's lexical analysis of a 'case' statement within a do...done block
Fix lexing of 'case' in do...done in a $(comsub) (rhbz#1241013) The following caused a spurious syntax error: $ x=$(for i in 1; do case $i in word) true;; esac; done) -ksh: syntax error: `;;' unexpected Prior discussion: https://bugzilla.redhat.com/1241013 Original patch, backported from 93v- beta, applied without change: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-parserfix.patch
2020-09-27 19:26:09 +00:00
within a command substitution of the form $(...) has been fixed so that code
like the following no longer throws a spurious syntax error:
x=$(for i in 1; do case $i in word) true;; esac; done)
Previously, this required a leading parenthesis before 'word', although the
syntax error claimed that the ';;' was unexpected.
whence -f: ignore functions (#137) According to 'whence --man', 'whence -f' should ignore functions: -f Do not check for functions. Right now this is only accomplished partially. As of commit a329c22d 'whence -f' avoids any output when encountering a function (in ksh93u+ 'whence -f' has incorrect output). The return value is still wrong though: $ foo() { true; } $ whence -f foo; echo $? 0 This commit fixes the return value and makes 'type -f' error out when given a function (like in Bash). src/cmd/ksh93/bltins/whence.c: - If -f was passed, set 'cp' to NULL since functions should be ignored (as documented). - Simplify return value by avoiding bitwise logic. src/cmd/ksh93/tests/builtins.sh: - Add regression tests for 'whence -f' and 'type -f'. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2020-09-26 18:26:18 +00:00
2020-09-26:
- 'whence -f' now completely ignores the existence of functions, as documented.
Avoid importing env vars with invalid names (rhbz#1147645) This imports a new version of the code to import environment variable values that was sent to Red Hat from upstream in 2014. It avoids importing environment variables whose names are not valid in the shell language, as it would be impossible to change or unset them. However, they stay in the environment to be passed to child processes. Prior discussion: https://bugzilla.redhat.com/1147645 Original patch: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-oldenvinit.patch src/cmd/ksh93/sh/init.c: - env_init(): Import new, simplified code to import environment variable name/value pairs. Instead of doing the heavy lifting itself, this version uses nv_open(), passing the NV_IDENT flag to reject and skip invalid names. - Get rid of gotos and a static var by splitting off the code to import attributes into a new env_import_attributes() function. This is a better way to avoid importing attributes when initialising the shell in POSIX mode (re: 00d43960 - Remove an nv_mapchar() call that was based on some unclear flaggery which was also removed by upstream as sent to Red Hat. I don't know what that did, if anything; looks like it might have had something to do with typeset -u/-l, but those particular attributes have never been successfully inherited through the environment. (Maybe that's another bug, or maybe I just don't care as inheriting attributes is a misfeature anyway; we have to put up with it because legacy scripts might use it. Maybe someone can prove it's an unacceptable security risk to import attributes like readonly from an environment variable that is inherently vulnerable to manipulation. That would be nice, as a CVE ID would give us a solid reason to get rid of this nonsense.) - Remove an 'else cp += 2;' that was very clearly a no-op; 'cp' is immediately overwritten on the next loop iteration and not used past the loop. src/cmd/ksh93/tests/variables.sh: - Test.
2020-09-26 18:28:55 +00:00
- ksh now does not import environment variables whose names are not valid in
the shell language, as it would be impossible to change or unset them.
However, they stay in the environment to be passed to child processes.
Fix argv rewrite on invoking hashbangless script (rhbz#1047506) The fixargs() function is invoked when ksh needs to run a script without a #!/hashbang/path. Instead of letting the kernel invoke a shell, ksh exfile()s the script itself from sh_main(). In the forked child, it calls fixargs() to set the argument list in the environment to the args of the new script, so that 'ps' and /proc/PID/cmdline show the expected output. But fixargs() is broken because, on systems other than HP-UX (on which ksh uses pstat(2)), ksh simply inserts a terminating zero. The arguments list is not a zero-terminated C string. Unix systems expect the entire arguments buffer to be zeroed out, otherwise 'ps' and /proc/*/cmdline will have fragments of previous command lines in the output. The Red Hat patch for this bug is: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-argvfix.patch However, that fix is incomplete because 'command_len' was also hardcoded to be limited to 64 characters (!), which still gave invalid 'ps' output if the erased command line was longer. src/cmd/ksh93/sh/main.c: fixargs(): - Remove CMD_LENGTH macro which was defined as 64. - Remove code that limited the erasure of the arguments buffer to CMD_LENGTH characters. That code also had quite a dodgy strdup() call -- it copies arguments to the heap, but they are never freed (or even used), so it's a memory leak. Also, none of this is ever done if the length is calculated using pstat(2) on HP-UX, which is a clear indication that it's unnecessary. (I think this code block must have been some experiment they forgot to remove. One reason why I think so is that a 64 byte arguments limit never made sense, even in the 1980s when they wrote ksh on 80-column CRT displays. Another indication of this is that fixing it didn't require adding anything; the code to do the right thing was already there, it was just being overridden.) - Zero out the full arguments length as in the Red Hat patch. src/cmd/ksh93/tests/basic.sh: - Add test. It's sort of involved because 'ps' is one of the least portable commands in practice, in spite of standardisation.
2020-09-25 06:19:43 +00:00
2020-09-25:
whence -v/-a: report path to autoloadable functions Since at least 1999, whence -v on pdksh (and its successor mksh) reports the path where an autoloadable function may be found: $ mkdir ~/fun; FPATH=~/fun $ echo 'myfn() { echo hi; }' >~/fun/myfn $ whence -v myfn myfn is a undefined (autoload from /home/user/fun/myfn) function Whereas ksh93 only reports, rather uselessly: myfn is an undefined function As of this commit, whence -v/-a on ksh 93u+m does the same as pdksh, but with correct grammar: myfn is an undefined function (autoload from /home/user/fun/myfn) This may be a small violation of my own "no new features" policy for 93u+m, but I couldn't resist. This omission has been annoying me, and it's just embarrassing to lack a pdksh feature :) src/cmd/ksh93/include/path.h, src/cmd/ksh93/data/msg.c: - Add e_autoloadfrom[] = " (autoload from %s)" message. src/cmd/ksh93/bltins/whence.c: whence(): - Report the path (if any) when reporting an undefined function. This needs to be done in two places: 1. When a function has been explicitly marked undefined with 'autoload', we need to do a quick path_search() loop to find the path. (These undefined functions take precedence over regular commands, so are reported first.) 2. When a function is not explicitly autoloaded but merely available in $FPATH, that path search was already done, so all we need to do is report it. (These are reported last.) Note that the output remains as on 93u+ if no function definition file is found on $FPATH. This is also like pdksh/mksh. src/cmd/ksh93/data/builtins.c: - Bump 'whence' version date. The inline docs never detailed very exactly what 'whence -v' reports, so no need for further edits. src/cmd/ksh93/tests/path.sh: - Regress-test the new whence behaviour plus actual autoloading, including the command override behaviour of autoloaded functions.
2020-09-25 13:39:08 +00:00
- whence -v/-a now reports the path to the file that an "undefined" (i.e.
autoloadable) function will be loaded from when invoked, if found in $FPATH.
Fix argv rewrite on invoking hashbangless script (rhbz#1047506) The fixargs() function is invoked when ksh needs to run a script without a #!/hashbang/path. Instead of letting the kernel invoke a shell, ksh exfile()s the script itself from sh_main(). In the forked child, it calls fixargs() to set the argument list in the environment to the args of the new script, so that 'ps' and /proc/PID/cmdline show the expected output. But fixargs() is broken because, on systems other than HP-UX (on which ksh uses pstat(2)), ksh simply inserts a terminating zero. The arguments list is not a zero-terminated C string. Unix systems expect the entire arguments buffer to be zeroed out, otherwise 'ps' and /proc/*/cmdline will have fragments of previous command lines in the output. The Red Hat patch for this bug is: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-argvfix.patch However, that fix is incomplete because 'command_len' was also hardcoded to be limited to 64 characters (!), which still gave invalid 'ps' output if the erased command line was longer. src/cmd/ksh93/sh/main.c: fixargs(): - Remove CMD_LENGTH macro which was defined as 64. - Remove code that limited the erasure of the arguments buffer to CMD_LENGTH characters. That code also had quite a dodgy strdup() call -- it copies arguments to the heap, but they are never freed (or even used), so it's a memory leak. Also, none of this is ever done if the length is calculated using pstat(2) on HP-UX, which is a clear indication that it's unnecessary. (I think this code block must have been some experiment they forgot to remove. One reason why I think so is that a 64 byte arguments limit never made sense, even in the 1980s when they wrote ksh on 80-column CRT displays. Another indication of this is that fixing it didn't require adding anything; the code to do the right thing was already there, it was just being overridden.) - Zero out the full arguments length as in the Red Hat patch. src/cmd/ksh93/tests/basic.sh: - Add test. It's sort of involved because 'ps' is one of the least portable commands in practice, in spite of standardisation.
2020-09-25 06:19:43 +00:00
- When ksh invoked a shell script that does not have a leading
#!/hashbang/path, 'ps' and /proc/<PID>/cmdline showed corrupted output if
the new script's command line was shorter than that of the invoking script.
This has been fixed by wiping the arguments buffer correctly.
Allow turning off brace expansion in comsubs (rhbz#1078698) There was no check for the -B/braceexpand option before calling path_expand() to process brace expansion, making it impossible to turn off brace expansion within command substitutions. Normally the lexer flags brace expansion so that this code is not reached, but shell code within command substitutions is handled differently. Red Hat patches this by adding this check to the function itself: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20140301-fikspand.patch But I think it's more logical to patch it at the point of decision. src/cmd/ksh93/sh/macro.c: endfield(): - Decide to call either path_generate() or path_expand() based on the state of the SH_BRACEEXPAND shell option. - Fix '#if SHOPT_BRACEPAT' preprocessor check that previously hardcoded this decision at compile time. src/cmd/ksh93/tests/options.sh: - Add tests.
2020-09-24 05:36:44 +00:00
2020-09-24:
- An omission made it impossible to turn off brace expansion within command
substitutions (`...`, $(...) or ${ ...; }) as the code for parsing these
did not check the -B/braceexpand option. This check has now been added.
Fix crash in `backtick comsubs` with job control on (rhbz#825520) This imports another fix from Red Hat/Fedora. Original patch: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-crash.patch src/cmd/ksh93/include/jobs.h, src/cmd/ksh93/sh/jobs.c, src/cmd/ksh93/sh/subshell.c, src/cmd/ksh93/sh/xec.c: - Import the Red Hat fix with these differences: - Rename the 'hack1_waitall' variable to 'bktick_waitall' and add a comment describing what it's for. - Remove unused 'pipefail' variable. src/cmd/ksh93/tests/basic.sh: - Regression test from reproducer given in the Red Hat bug report. - Add special handling to SIGKILL it, as it might freeze hard.
2020-09-22 23:56:09 +00:00
2020-09-23:
- Fixed a crash that could occur when running a pipeline containing
backtick-style command substitutions with job control enabled.
Fix typeset -l/-u crash on special vars (rhbz#1083713) When using typeset -l or -u on a variable that cannot be changed when the shell is in restricted mode, ksh crashed. This fixed is inspired by this Red Hat fix, which is incomplete: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-tpstl.patch The crash was caused by the nv_shell() function. It walks though a discipline function tree to get the pointer to the interpreter associated with it. Evidently, the problem is that some pointer in that walk is not set correctly for all special variables. Thing is, ksh only has one shell language interpreter, and only one global data structure (called 'sh') to keep its main state[*]. Yet, the code is full of 'shp' pointers to that structure. Most (not all) functions pass that pointer around to each other, accessing that struct indirectly, ostensibly to account for the non-existent possibility that there might be more than one interpreter state. The "why" of that is an interesting cause for speculation that I may get to sometime. For now, it is enough to know that, in the code as it is, it matters not one iota what pointer to the shell interpreter state is used; they all point to the same thing (unless it's broken, as in this bug). So, rather than fixing nv_shell() and/or associated pointer assignments, this commit simply removes it, and replaces it with calls to sh_getinterp(), which always returns a pointer to sh (see init.c, where that function is defined as literally 'return &sh'). [*] Defined in shell.h, with the _SH_PRIVATE part in defs.h src/cmd/ksh93/include/defs.h, src/cmd/ksh93/sh/name.c: - Remove nv_shell(). src/cmd/ksh93/sh/init.c: - In all the discipline functions for special variables, initialise shp using sh_getinterp() instead of nv_shell(). src/cmd/ksh93/tests/variables.sh: - Add regression test for typeset -l/-u on all special variables.
2020-09-23 17:15:45 +00:00
- Fixed a crash that occurred when using 'typeset -u' or 'typeset -l' on a
special variable such as PATH, ENV or SHELL.
update NEWS, SH_RELEASE (re: 970069a6)
2020-09-21 23:42:15 +00:00
2020-09-21:
- A bug was fixed that caused command substitutions embedded in here-documents
to lose the output of the commands they ran. This bug occurred when ksh was
compiled with the SHOPT_SPAWN compile-time option.
Fix v=$(<file) for closed FD 0,1,2 (rhbz#1066589) var=$(< file) now reads the file even if the standard inout, standard output and/or standard error file descriptors are closed. Original patch: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-filecomsubst.patch src/cmd/ksh93/sh/io.c: sh_redirect(): - When processing the '<' redirector as part of $(< ...), i.e. if flag==3, make sure the FD of the file to read is > 2 by calling sh_iomovefd(). Unlike the RedHat patch, this checks for flag==3 to avoid unnecessary sh_iomovefd() calls for normal redirections, as there was no bug with those. src/cmd/ksh93/tests/io.sh: - Add test.
2020-09-22 00:45:59 +00:00
- Bugfix: var=$(< file) now reads the file even if the standard inout, standard
output and/or standard error file descriptors are closed.
Multiple 'whence' and path search fixes Hopefully this doesn't introduce new bugs, but it does fix at least the following: 1. When whence -v/-a found an "undefined" (i.e. autoloadable) function in $FPATH, it actually loaded the function as a side effect of reporting on its existence (!). Now it only reports. 2. 'whence' will now canonicalise paths properly. Examples: $ whence ///usr/lib/../bin//./env /usr/bin/env $ (cd /; whence -v dev/../usr/bin//./env) dev/../usr/bin//./env is /usr/bin/env 3. 'whence' no longer prefixes a spurious double slash when doing something like 'cd / && whence bin/echo'. On Cygwin, an initial double slash denotes a network server, so this was not just a cosmetic problem. 4. 'whence -a' now reports a "tracked alias" (a.k.a. hash table entry, i.e. cached $PATH search) even if an actual alias by the same name exists. This needed fixing because in fact the hash table entry continues to be used when bypassing the alias. Aliases and "tracked aliases" are not remotely the same thing; confusing nomenclature is not a reason to report wrong results. 5. When using 'hash' or 'alias -t' on a command that is also a builtin to force caching a $PATH search for the external command, 'whence -a' double-reported the path: $ hash printf; whence -a printf printf is a shell builtin printf is /usr/bin/printf printf is a tracked alias for /usr/bin/printf This is now fixed so that the second output line is gone. Plus, if there were multiple versions of the command on $PATH, the tracked alias was reported at the end, which is the wrong order. This is also fixed. src/cmd/ksh93/bltins/whence.c: whence(): - Refactor the do...while loop that handles whence -v/-a for path searches in such a way that the code actually makes sense and stops looking like higher esotericism. Just doing this fixed #2, #4 and #5 above (the latter two before I even noticed them). For instance, the path_fullname() call to canonicalise paths was already there; it was just never used. - Remove broken 'notrack' flaggery for deciding whether to report a hash table entry a.k.a. "tracked alias"; instead, check the hash table (shp->track_tree). src/cmd/ksh93/sh/path.c: - path_search(): Re #3: When prefixing the PWD, first check if we're in '/' and if so, don't prefix it; otherwise, adding the next slash causes an initial double slash. (Since '/' is the only valid single-character absolute path, all we need to do is check if the second character pwd[1] is non-null.) - path_search(): Re #1: Stop autoloading when called by 'whence': * The 'flag==2' check to avoid autoloading a function was broken. The flag value is 2 on the first whence() loop iteration, but 3 on subsequent ones. Change to 'flag >= 2'. * However, this only fixes it if the function file does not have the x permission bit, as executable files are handled by path_absolute() which unconditionally autoloads functions! So, pass on our flag parameter when callling path_absolute(). - path_absolute(): Re #1: Add flag parameter. Do not autoload functions if flag >= 2. src/cmd/ksh93/include/path.h, src/cmd/ksh93/bltins/typeset.c, src/cmd/ksh93/sh/main.c, src/cmd/ksh93/sh/xec.c: - Re #1: Update path_absolute() calls, adding a 0 flag parameter. src/cmd/ksh93/include/name.h: - Remove now-unused pathcomp member from union Value. It was introduced in 99065353 to allow examining the value of a tracked alias. This commit uses nv_getval() instead. src/cmd/ksh93/tests/builtins.sh, src/cmd/ksh93/tests/path.sh: - Add and tweak various related tests. Fixes: https://github.com/ksh93/ksh/issues/84
2020-09-20 05:56:09 +00:00
2020-09-20:
- Bugfix: when whence -v/-a found an "undefined" (i.e. autoloadable) function
in $FPATH, it actually loaded the function as a side effect of reporting on
its existence. Now it only reports, as documented.
- 'whence' will now canonicalise paths properly, resolving '.' and '..'
elements in paths given to it. It also no longer prefixes a spurious
double slash when doing something like 'cd / && whence bin/echo'.
-o posix: inverse-sync braceexpand; properly sync letoctal {Brace,expansion} is potentially incompatible with POSIX scripts, because in POSIX those are simple literal strings with no special meaning. So the POSIX option should really turn that off. As of b301d417, the 'posix' option was also forcing 'letoctal' behaviour on, without actually setting that option. I've since found that to be a botch; 'let' may recognise octals without that option being set, and that looks like a bug. So as of this commit, the '-o posix' option actually toggles both of these options off/on and on/of, respectively. 'set +o posix' toggles them inversely. However, it is now possible to control both options (and their associated behaviour) independently in between 'set -o posix' and 'set +o posix'. Much better. src/cmd/ksh93/sh/main.c: sh_main(): - If SH_POSIX was set on init, turn on SH_LETOCTAL by default instead of SH_BRACEEXPAND. src/cmd/ksh93/sh/args.c: sh_applyopts(): - Turn off SH_BRACEEXPAND and turn on SH_LETOCTAL when SH_POSIX is turned on (but not if it was already on). - Turn on SH_BRACEEXPAND and turn off SH_LETOCTAL when SH_POSIX is turned off (but not if it was already off). src/cmd/ksh93/sh/arith.c: arith(): - Revert to pre-b301d417 and only check SH_LETOCTAL option when deciding whether 'let' should skip initial zeros. src/cmd/ksh93/tests/options.sh: - Update $- test to allow '-o posix' to switch B = braceexpand. src/cmd/ksh93/sh.1: - Update. - Edit for clarity.
2020-09-18 18:32:34 +00:00
2020-09-18:
- Setting the 'posix' option now turns off the 'braceexpand' option, as brace
expansion is not specified by POSIX and potentially incompatible with sh
scripts. In addition, 'set -o posix' now turns on the 'letoctal' option
instead of controlling that behaviour directly. 'set +o posix' does the
reverse of these.
emacs, vi: Support repeat parameters to VT220 keys (re: f2a3f4e3) In the vi and emacs line editors, repeat count parameters can now also be used for the arrow keys and the forward-delete key. E.g., in emacs mode, <ESC> 7 <left-arrow> will now move the cursor seven positions to the left. In vi control mode, this would be entered as: 7 <left-arrow>. src/cmd/ksh93/edit/emacs.c: - ed_emacsread(): Upon getting ^[ (ESC), save current repeat count in a new variable; restore and reset it upon the next character. - escape(): Minor bugfix: when processing a ^[[x sequence where 'x' is a character other than '~' (which would be DEL), also reinsert the final character into the buffer so scripts can detect them. src/cmd/ksh93/edit/vi.c: - cntlmode(): Do not reset the repeat count if the command is '[', the character following ESC in VT220 escape sequences. - mvcursor(): * Do not use getcount() to get the character following '[', as that was parsing repetition parameters in the wrong place. There wouldn't be any, so this would reset the repeat count. * After that, no more need for the special-casing of ^[[3~ (DEL) introduced in f2a3f4e3. Move it to within the 'switch' block. * When handling left and right arrows and Home and End keys, do not modify cursor directly but ed_ungetchar() the corresponding traditional command keys as with the rest. Otherwise a repeat count parameter would now wrongly survive those keys. src/cmd/ksh93/sh.1: - Document control character notation used for vi mode docs. - Since vi control mode beeps and aborts on ESC except if a subsequent [ is already in the input buffer upon receiving ESC, document that VT220 escape sequences only preserve repeat counts when entered into the input buffer all at once. - Don't skip the initial ESC in the documentation of the VT220 escape sequences. In control mode, skipping the initial ESC still works as before, but that is now undocumented, as it's really nothing more than an artefact of VT220 escape processing. - Move the two long paragraphs on '-o viraw' and canonical (i.e. line-based) input processing from the vi editor introduction to the options section under 'viraw'. It is much too arcane for the intro, and besides, ksh 93u+ (and hence also 93u+m) has SHOPT_VIRAW enabled by default, so the shell is compiled to force this option on at all times, making it even less relevant for most users.
2020-09-17 15:28:03 +00:00
2020-09-17:
- In the vi and emacs line editors, repeat count parameters can now also be
used for the arrow keys and the forward-delete key. E.g., in emacs mode,
<ESC> 7 <left-arrow> will now move the cursor seven positions to the left.
In vi control mode, this would be entered as: 7 <left-arrow>.
data/signals.c: fix empty SIGINT/SIGPIPE messages src/cmd/ksh93/data/signals.c includes two checks for the JOBS identifier; if it is not defined then the interactive shell's background job signal messages for SIGINT and SIGPIPE are empty. The cause was that the "jobs.h" header, which defines that ID, was not #included in signals.c. This commit adds that #include. (ksh 93u+, ksh 93v- and ksh2020 all have this bug as well.) Before: $ sleep 30 & [1] 86430 $ kill -s INT "$!" [1] + sleep 30 & $ After: $ sleep 30 & [1] 86445 $ kill -s INT "$!" [1] + Interrupt sleep 30 & $
2020-09-18 01:03:39 +00:00
- When a background job on an interactive shell received SIGINT or SIGPIPE, the
job termination message was empty. It now shows "Interrupt" or "Broken Pipe".
A few job control (-m, -o monitor) fixes (rhbz#960034) This patch from Red Hat fixes the following: 1. ksh was ignoring the -m (-o monitor) option when specified on the invocation command line. 2. Scripts did not properly terminate their background processes on Ctrl+C if the -m option was turned off. Reproducer: xterm & read junk When run as a script without turning on -m, pressing Ctrl+C should terminate the xterm, and now does. 3. Scripts no longer attempt to set the terminal foreground process group ID, as only interactive shells should be doing that. This makes some progress on https://github.com/ksh93/ksh/issues/119 but we're a long way from fixing all of that. src/cmd/ksh93/sh/main.c: exfile(): - On non-interactive shells, do not turn off the monitor option. Instead, if it was turned on, turn on the SH_MONITOR state flag. src/cmd/ksh93/edit/edit.c: ed_getchar(): - On Ctrl+C, issue SIGINT to the current process group using killpg(2) instead of going via sh_fault(), which handles a signal only for the current shell process. src/cmd/ksh93/sh/jobs.c: job_reap(), job_reset(), src/cmd/ksh93/sh/xec.c: sh_exec(): - Only attempt to set the terminal foreground process group ID using tcsetpgrp(3) if the shell is interactive. Original patch: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-kshmfix.patch This was applied to Red Hat's ksh 93u+ on 8 July 2013.
2020-09-18 02:42:27 +00:00
- The -m (-o monitor) option is no longer ignored when specified on the shell
invocation command line.
- A script that is interrupted with Ctrl+C now terminates its background jobs
as expected, unless the -m (-o monitor) option was turned on.
sfio: correct floating decimal point scaling of fractions (#131) _sfcvt(), "convert a floating point value to ASCII", did not adjust for negative decimal place movement as what happens with leading zeroes. This caused ksh's 'printf %f' formatter to fail to round floating point values correctly. src/lib/libast/sfio/sfcvt.c: - Removed constraint of <1e-8 for doubles by matching what was done for long doubles having <.1. - Corrected a condition when the next power of 10 occurred and that new 1 digit was being overwritten by a 0. src/cmd/ksh93/tests/math.sh: - Validate that typeset -E/F formatting matches that of their equivalent printf formatting options as well as checking for correct float scaling of the fractional parts.
2020-09-14 10:15:05 +00:00
2020-09-14:
- Corrected rounding of floating point values by ksh's printf %f formatting
operator. Fix contributed by @hyenias.
Handle forward-delete key in emacs and vi editors On every modern system, the forward-delete key on PC/Mac keyboards generates the VT220 sequence ESC [ 3 ~. Every other shell with an editor handles this now, ksh93 seems to be the last not to. src/cmd/ksh93/edit/emacs.c: escape(): - Handle the ^[[3 as part of normal escape processing, then read an extra character to check for the final '~'. If detected, insert an ERASECHAR key event. src/cmd/ksh93/edit/vi.c: mvcursor(): - Replace the ^[[3~ sequence by an 'x' command. We have to special-case its processing, because vi mode parses numbers as repetition operators. The escape sequence contains a number, making it incompatible with normal command handling. This means number repetitions don't work with the forward-delete key. If that annoys anyone enough to fix it, a patch would be welcome. For now, it will do to make the forward-delete key stop exhibiting bizarre behaviour (beep + change case + move forward). src/cmd/ksh93/sh.1 - Copy-edit emacs documentation for VT220-style sequences; map them to their actual key, otherwise it's meaningless to the reader. - Document the new forward-delete key behaviour for emacs mode. - Leave the forward-delete key for vi mode undocumented for now, as repetitions don't work, so it doesn't really match the vi canon. (OTOH, it doesn't work in vim, either...)
2020-09-14 22:49:05 +00:00
- The forward-delete key now works as expected in emacs and vi editing modes.
Fix 'command' expansion bug and POSIX compliance The 'command' name can now result from an expansion, e.g.: c=command; "$c" ls set -- command ls; "$@" both work now. This fixes BUG_CMDEXPAN. If -o posix is on, 'command' now disables not only the "special" but also the "declaration" properties of builtin commands that it invokes. This is because POSIX specifies 'command' as a simple regular builtin, and any command name following 'command' is just an argument to the 'command' command, so there is nothing that allows any further arguments (such as assignment-arguments) to be treated specially by the parser. So, if and only if -o posix is on: a. Arguments that start with a variable name followed by '=' are always treated as regular words subject to normal shell syntax. b. Since assignment-arguments are not processed as assignments before the command itself, 'command' can now stop the shell from exiting (as required by the standard) if a command that it invokes (such as 'export') tries to modify a readonly variable. This fixes BUG_CMDSPEXIT. Most of 'command' is integrated in the parser and parse tree executer, so that is where it needed fixing. src/cmd/ksh93/sh/parse.c: simple(): - If the posix option is on, do not skip past SYSCOMMAND so that any declaration builtin commands that are arguments to 'command' are not detected and thus not treated specially at parsetime. src/cmd/ksh93/sh/xec.c: sh_exec(): - When detecting SYSCOMMAND in order to skip past it, not only compare the Namval_t pointer 'np' to SYSCOMMAND, but also handle the case where that pointer is NULL, as when the command name results from an expansion. In that case, search the function tree shp->fun_tree for the name and see if that yields the SYSCOMMAND pointer. fun_tree is initialised with a dtview to bltin_tree, so searching fun_tree instead allows for overriding 'command' with a shell function (which the POSIX standard requires us to allow). src/cmd/ksh93/sh.1, src/cmd/ksh93/data/builtins.c: - Update documentation to match these changes. - Various related edits and improvements. src/cmd/ksh93/tests/builtins.sh: - Check that 'command' works if resulting from an expansion. - Check that 'command' can be overridden by a shell function.
2020-09-11 07:33:29 +00:00
2020-09-11:
- The 'command' regular builtin utility (which runs a simple command, removing
special properties) has been made fully POSIX compliant.
1. The 'command' name can now result from an expansion (fixing BUG_CMDEXPAN),
e.g. 'c=command; "$c" ls' and 'set -- command ls; "$@"' now work.
2. If and only if the POSIX mode (the new -o posix shell option) is active,
then the 'command' utility now disables not only "special" but also
"declaration" properties of builtin commands that it invokes, meaning:
a. arguments that start with a variable name followed by '=' are
always treated as regular words subject to normal shell syntax;
b. 'command' can now stop the shell from exiting if a command that it
invokes tries to modify a readonly variable (fixing BUG_CMDSPEXIT).
Reinstate 'r' and 'history' as preset aliases for interactive ksh Following a community discussion, it became clear that 'r' is particularly problematic as a regular builtin, as the name can and does conflict with at least one legit external command by that name. There was a consensus against removing it altogether and letting users set the alias in their login scripts. However, aliases are easier to bypass, remove or rename than builtins are. My compromise is to reinstate 'r' as a preset alias on interactive shells only, along with 'history', as was done in 17f81ebe before they were converted to builtins in 03224ae3. So this reintroduces the notion of predefined aliases to ksh 93u+m, but only for interactive shells that are not initialised in POSIX mode. src/cmd/ksh93/Makefile, src/cmd/ksh93/Mamfile, src/cmd/ksh93/include/shtable.h, src/cmd/ksh93/data/aliases.c: - Restore aliases.c containing shtab_aliases[], a table specifying the preset aliases. src/cmd/ksh93/include/shtable.h, src/cmd/ksh93/sh/init.c: - Rename inittree() to sh_inittree() and make it extern, because we need to use it in main.c (sh_main()). src/cmd/ksh93/sh/main.c: sh_main(): - Init preset aliases from shtab_aliases[] only if the shell is interactive and not in POSIX mode. src/cmd/ksh93/bltins/typeset.c, src/cmd/ksh93/tests/alias.sh: - unall(): When unsetting an alias, pass on the NV_NOFREE attribute to nv_delete() to avoid an erroneous attempt to free a preset alias from read-only memory. See: 5d50f825 src/cmd/ksh93/data/builtins.c: - Remove "history" and "r" entries from shtab_builtins[]. - Revert changes to inline fc/hist docs in sh_opthist[]. src/cmd/ksh93/bltins/hist.c: b_hist(): - Remove handling for 'history' and 'r' as builtins. src/cmd/ksh93/sh.1: - Update accordingly. Resolves: https://github.com/ksh93/ksh/issues/125
2020-09-11 18:51:35 +00:00
- The 'history' (== 'hist -l') and 'r' (== 'hist -s') interactive shell
history commands have reverted to preset aliases and are now only loaded if
the shell is interactive and not initialised in POSIX mode. This avoids
unneeded conflicts with external commands by these names, particularly 'r'.
Fix BUG_LOOPRET2 and related return/exit misbehaviour The 'exit' and 'return' commands without an argument failed to pass down the exit status of the last-run command when incorporated in a block with redirection, &&/|| list, 'case' statement, or 'while', 'until' or 'for' loop. src/cmd/ksh93/bltins/cflow.c: - Use $?, which is sh.savexit a.k.a. shp->savexit, as the default exit status value if there is no argument, instead of shp->oldexit. This fixes the default exit status behaviour to match POSIX and other shells. src/cmd/ksh93/include/defs.h, src/cmd/ksh93/include/shell.h: - Remove now-unused sh.oldexit (a.k.a. shp->oldexit) private struct member. It appeared to fulfill the same function as sh.savexit, but in a slightly broken way. - Move the savexit/$? declaration from the _SH_PRIVATE part of the struct definition to the public API part. Since $? uses this, it's clearly a publicly exposed value already, and this is generally the one to use. (If anything, it's exitval that should have been private.) This declares savexit right next to exitval, rewriting the comments to clarify the difference between them. src/cmd/ksh93/sh/fault.c, src/cmd/ksh93/sh/subshell.c, src/cmd/ksh93/sh/xec.c: - Remove assignments to shp->oldexit. src/cmd/ksh93/tests/basic.sh: - Add thorough regression tests for the default exit status behaviour of 'return' and 'exit' in various lexical contexts. - Verify that 'for' and 'case' without any command, as well as a lone redirection, still correctly reset the exit status to 0. Fixes: #117
2020-09-09 18:02:20 +00:00
2020-09-09:
- Fixed BUG_LOOPRET2 and related bugs. The 'exit' and 'return' commands without
an argument now correctly default to passing down the exit status of the
last-run command. Tests like the following, in which the last-run command is
'false', now correctly output 1 instead of 0:
fn() { return || true; }; false; fn; echo "$?"
fn() { while return; do true; done; }; false; fn; echo "$?"
fn() { for i in 1; do return; done; }; false; fn; echo "$?"
fn() { case 1 in 1) return ;; esac; }; false; fn; echo "$?"
fn() { { return; } 2>&1; }; false; fn; echo "$?"
This fixes erroneous syntax errors in parameter expansions such as ${var:-wor)d} or ${var+w(ord}. The parentheses now correctly lose their normal grammatical meaning within the braces. Fix by Eric Scrivner (@etscrivner) from July 2018 backported from ksh2020. This fix complies with POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02 src/cmd/ksh93/sh/lex.c: sh_lex(): - Set the ST_QUOTE state when analysing a modifier with parameter expansions using operators ':', '-', '+', '='. This state causes subsequent characters (including parentheses) to be considered quoted, suppressing their normal grammatical meaning. src/cmd/ksh93/sh/macro.c: varsub(): - Same for skipping the expansion. Fixes: https://github.com/ksh93/ksh/issues/126 Prior discussion: https://github.com/att/ast/issues/475
2020-09-05 14:20:22 +00:00
2020-09-05:
- Fixed erroneous syntax errors in parameter expansions such as ${var:-wor)d}
or ${var+w(ord}. The parentheses now correctly lose their normal grammatical
meaning within the braces. Fix by Eric Scrivner backported from ksh2020.
update NEWS, SH_RELEASE (re: 6575903d)
2020-09-04 03:29:52 +00:00
2020-09-04:
- Fixed a bug that caused a syntax error to be thrown if the special parameter
expansions ${!} and ${$} (including braces) were used within a here-document.
Bug reported by @Saikiran-m on GitHub.
Remove SHOPT_BASH; keep &> redir operator, '-o posix' option On 16 June there was a call for volunteers to fix the bash compatibility mode; it has never successfully compiled in 93u+. Since no one showed up, it is now removed due to lack of interest. A couple of things are kept, which are now globally enabled: 1. The &>file redirection shorthand (for >file 2>&1). As a matter of fact, ksh93 already supported this natively, but only while running rc/profile/login scripts, and it issued a warning. This makse it globally available and removes the warning, bringing ksh93 in line with mksh, bash and zsh. 2. The '-o posix' standard compliance option. It is now enabled on startup if ksh is invoked as 'sh' or if the POSIXLY_CORRECT variable exists in the environment. To begin with, it disables the aforementioned &> redirection shorthand. Further compliance tweaks will be added in subsequent commits. The differences will be fairly minimal as ksh93 is mostly compliant already. In all changed files, code was removed that was compiled (more precisely, failed to compile/link) if the SHOPT_BASH preprocessor identifier was defined. Below are other changes worth mentioning: src/cmd/ksh93/sh/bash.c, src/cmd/ksh93/data/bash_pre_rc.sh: - Removed. src/cmd/ksh93/data/lexstates.c, src/cmd/ksh93/include/shlex.h, src/cmd/ksh93/sh/lex.c: - Globally enable &> redirection operator if SH_POSIX not active. - Remove warning that was issued when &> was used in rc scripts. src/cmd/ksh93/data/options.c, src/cmd/ksh93/include/defs.h, src/cmd/ksh93/sh/args.c: - Keep SH_POSIX option (-o posix). - Replace SH_TYPE_BASH shell type by SH_TYPE_POSIX. src/cmd/ksh93/sh/init.c: - sh_type(): Return SH_TYPE_POSIX shell type if ksh was invoked as sh (or rsh, restricted sh). - sh_init(): Enable posix option if the SH_TYPE_POSIX shell type was detected, or if the CONFORMANCE ast config variable was set to "standard" (which libast sets on init if POSIXLY_CORRECT exists in the environment). src/cmd/ksh93/tests/options.sh, src/cmd/ksh93/tests/io.sh: - Replace regression tests for &> and move to io.sh. Since &> is now for general use, no longer test in an rc script, and don't check that a warning is issued. Closes: #9 Progresses: #20
2020-09-01 05:19:19 +00:00
2020-09-01:
- The bash-style '&>file' redirection shorthand (for '>file 2>&1') is now
always recognised and not only when running rc/profile init scripts. It no
longer issues a warning. This brings ksh93 in line with mksh, bash and zsh.
- A long-form shell option '-o posix' has been added, which implements a
mode for better compatibility with the POSIX standard. It is automatically
turned on if ksh is invoked under the name 'sh'.
For now, it:
* disables the &> redirection shorthand
-o posix: always recognise octals in "let" builtin Though the "let" builtin is not itself a POSIX standard command, it processes standard shell arithmetic, so it should recognise octals by leading zeros as POSIX requires if the 'posix' option is on. This overrides the setting of the 'letoctal' option. Note that none of this applies to the ((...)) arithmetic command, which has always recognised leading-octal zeros and does not listen to 'letoctal'. So setting the posix mode makes this consistent. src/cmd/ksh93/sh/arith.c: - When running the 'let' builtin, test that both SH_LETOCTAL and SH_POSIX are off before stripping leading zeros to disable octal number recognition. - Cosmetic: fix spurious newline. src/cmd/ksh93/sh.1: - Document the change. src/cmd/ksh93/tests/shtests: - Make sure to disable posix mode by default for regression tests.
2020-09-01 06:16:51 +00:00
* causes the 'let' arithmetic command to recognise octal numbers by
leading zeros regardless of the setting of the 'letoctal' option
-o posix: allow invoked programs to inherit FDs > 2 If there are file descriptors > 2 opened with 'exec' or 'redirect', ksh93 has always closed them when invoking another pogram. This is contrary to POSIX which states: Utilities other than the special built-ins […] shall be invoked in a separate environment that consists of the following. The initial value of these objects shall be the same as that for the parent shell, except as noted below. * Open files inherited on invocation of the shell, open files controlled by the exec special built-in plus any modifications, and additions specified by any redirections to the utility * […] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12 src/cmd/ksh93/sh/io.c: sh_redirect(): - When flag==2, do not close FDs > 2 if POSIX mode is active. src/cmd/ksh93/tests/io.sh: - Regress-test inheriting FD 7 with and without POSIX mode. src/cmd/ksh93/sh.1: - Update.
2020-09-01 07:11:27 +00:00
* causes file descriptors > 2 to be left open when invoking another program
Revert <> redir FD except in posix mode (re: eeee77ed, 60516872) eeee77ed implemented a POSIX compliance fix that caused a potential incompatibility with existing ksh scripts; it made the (rarely used) read/write redirection operator, <>, default to file descriptor 0 (standard input) as POSIX specified, instead of 1 (standard output) which is traditional ksh93 behaviour. So ksh scripts needed to change all <> to 1<> to override the new default. This commit reverts that change, except in the new posix mode. src/cmd/ksh93/sh/lex.c: - Make FD for <> default to 0 in POSIX mode, 1 otherwise. src/cmd/ksh93/tests/io.sh: - Revert <> regression test changes from 60516872; we no longer need 1<> instead of <> in ksh code.
2020-09-01 07:36:28 +00:00
* makes the <> redirection operator default to stdin instead of stdout
(this keeps the 2020-05-13 BUG_REDIRIO fix for the POSIX mode while
restoring traditional ksh93 behaviour for backwards compatibility)
-o posix: disable '[ -t ]' == '[ -t 1 ]' hack On ksh93, 'test -t' is equivalent to 'test -t 1' (and of course "[ -t ]" is equivalent to "[ -t 1 ]"). This is purely for compatibility with ancient Bourne shell breakage. No other shell supports this. ksh93 should probably keep it for backwards compatibility, but it should definitely be disabled in POSIX mode as it is a violation of the standard; 'test -t' is an instance of 'test "$string"', which tests if the string is empty, so it should test if the string '-t' is empty (quod non). This also replaces the fix for 'test -t 1' in a command substitution with a better one that avoids forking (re: cafe33f0). src/cmd/ksh93/sh/parse.c: - qscan(): If the posix option is active, disable the parser-based hack that converts a simple "[ -t ]" to "[ -t 1 ]". src/cmd/ksh93/bltins/test.c: - e3(): If the posix option is active, disable the part of the compatibility hack that was used for compound expressions that end in '-t', e.g. "[ -t 2 -o -t ]". - test_unop(): Remove the forking fix for "[ -t 1 ]". src/cmd/ksh93/edit/edit.c: - tty_check(): This function is used by "[ -t 1 ]" and in other contexts as well, so a fix here is more comprehensive. Forking here would cause a segfault, but we don't actually need to. This adds a fix that simply returns false if we're in a virtual subshell that is also a command substitution. Since command substitutions always fork upon redirecting standard output within them (making them no longer virtual), it is safe to do this. src/cmd/ksh93/tests/bracket.sh - Add comprehensive regression tests for test/[/[[ -t variants in command substitutions, in simple and compound expressions, with and without redirecting stdout to /dev/tty within the comsub. - Add tests verifying that -o posix disables the old hack. - Tweak other tests, including one that globally disabled xtrace.
2020-09-01 19:24:44 +00:00
* disables a noncompliant 'test -t' == 'test -t 1' compatibility hack
-o posix: don't import/export variable attributes thru environment When exporting variables, ksh exports their attributes (such as 'integer' or 'readonly') in a magic environment variable called "A__z" (string defined in e_envmarker[] in data/msg.c). Child shells recognise that variable and restore the attributes. This little-known feature is risky; the environment cannot necessarily be trusted and that A__z variable is easy to manipulate before or between ksh invocations, so you can cause a script's variables to be of the wrong type, or readonly. Backwards compatibility requires keeping it, at least for now. But it should be disabled in the posix mode, as it violates POSIX. To do this, we have to solve a catch-22 in init.c. We must parse options to know whether to turn on posix mode; it may be specified as '-o posix' on the command line. The option parsing loop depends on an initialised environment[*], while environment initialisation (i.e., importing attributes) should depend on the posix option. The catch-22 can be solved because initialising just the values before option parsing is enough to avoid regressions. Importing the attributes can be delayed until after option parsing. That involves basically splitting env_init() into two parts while keeping a local static state variable between them. src/cmd/ksh93/sh/init.c: - env_init(): * Split the function in two stages based on a new 'import_attributes' parameter. Import values in the first stage; import attributes from A__z in the second (if ever). Make the 'next' variable static as it keeps a state needed for the attributes import stage. * Single point of truth, greppability: don't hardcode "A__z" in separate character comparisons, but use e_envmarker[]. * Fix an indentation error. - sh_init(): When initialising the environment (env_init), don't import the attributes from A__z yet; parse options first, then import attributes only if posix option is not set. src/cmd/ksh93/sh/name.c: - sh_envgen(): Don't export variable attributes to A__z if the posix option is set. src/cmd/ksh93/tests/attributes.sh: - Check that variable attributes aren't imported or exported if the POSIX option is set. src/cmd/ksh93/sh.1: - Update. This was the last item on the TODO list for -o posix for now. Closes: #20 [*] If environment initialisation is delayed until after option parsing, bin/shtests shows various regressions, including: restricted mode breaks; the locale is not initialised properly so that multibyte variable names break; $SHLVL breaks.
2020-09-05 08:33:50 +00:00
* disables passing an exported variable's attributes (such as integer or
readonly) to a new ksh process through the environment
Remove SHOPT_BASH; keep &> redir operator, '-o posix' option On 16 June there was a call for volunteers to fix the bash compatibility mode; it has never successfully compiled in 93u+. Since no one showed up, it is now removed due to lack of interest. A couple of things are kept, which are now globally enabled: 1. The &>file redirection shorthand (for >file 2>&1). As a matter of fact, ksh93 already supported this natively, but only while running rc/profile/login scripts, and it issued a warning. This makse it globally available and removes the warning, bringing ksh93 in line with mksh, bash and zsh. 2. The '-o posix' standard compliance option. It is now enabled on startup if ksh is invoked as 'sh' or if the POSIXLY_CORRECT variable exists in the environment. To begin with, it disables the aforementioned &> redirection shorthand. Further compliance tweaks will be added in subsequent commits. The differences will be fairly minimal as ksh93 is mostly compliant already. In all changed files, code was removed that was compiled (more precisely, failed to compile/link) if the SHOPT_BASH preprocessor identifier was defined. Below are other changes worth mentioning: src/cmd/ksh93/sh/bash.c, src/cmd/ksh93/data/bash_pre_rc.sh: - Removed. src/cmd/ksh93/data/lexstates.c, src/cmd/ksh93/include/shlex.h, src/cmd/ksh93/sh/lex.c: - Globally enable &> redirection operator if SH_POSIX not active. - Remove warning that was issued when &> was used in rc scripts. src/cmd/ksh93/data/options.c, src/cmd/ksh93/include/defs.h, src/cmd/ksh93/sh/args.c: - Keep SH_POSIX option (-o posix). - Replace SH_TYPE_BASH shell type by SH_TYPE_POSIX. src/cmd/ksh93/sh/init.c: - sh_type(): Return SH_TYPE_POSIX shell type if ksh was invoked as sh (or rsh, restricted sh). - sh_init(): Enable posix option if the SH_TYPE_POSIX shell type was detected, or if the CONFORMANCE ast config variable was set to "standard" (which libast sets on init if POSIXLY_CORRECT exists in the environment). src/cmd/ksh93/tests/options.sh, src/cmd/ksh93/tests/io.sh: - Replace regression tests for &> and move to io.sh. Since &> is now for general use, no longer test in an rc script, and don't check that a warning is issued. Closes: #9 Progresses: #20
2020-09-01 05:19:19 +00:00
Speed up 'read', fixing macOS hang (take 2) This fixes a hanging bug that could occur on macOS when using the 'read' command to read from a FIFO and encountering end-of-file without a final newline character. It also makes the 'read' command perform 15-25% faster on macOS and Linux. The previous version (ff385e5a) failed on SunOS/Solaris/Illumos because those systems apparently don't (fully) support the POSIX standard recv(2) syscall with MSG_PEEK[*], which is the feature that iffe detects under the 'socket_peek' identifier. On Illumos, using that methods causes a compilation failure (unknown identifier MSG_PEEK); on Solaris 11.4, that method causes multiple regressions in tests/io.sh, suggesting the method compiles but doesn't work at all. Instead, SunOS/Solaris/Illumos requires the method using ioctl(2)+I_PEEK and select(2). No other system that ksh currently builds on requires this method, so it is now only used on SunOS/Solaris/Illumos. So far, this version of sfpkrd() has been tested to work correctly on Linux, macOS, FreeBSD, NetBSD, OpenBSD, HP-UX, Solaris, and OmniOS (an Illumos distribution). It still fails to peek on Cygwin, but in the exact same way it failed before, so that's no loss. To test, run the 'io' test set: bin/shtests -p io src/lib/libast/sfio/sfpkrd.c: sfpkrd(): - Remove long-obsolete Mac OS X and Solaris bug workarounds. - Remove methods that are no longer needed. On systems with a POSIX compliant recv(2), the only thing that is required to avoid regressions is the code that was conditional upon the socket_peek feature test, which tests for the correct functioning of the recv(2) syscall. This has now been made mandatory for non-SunOS/Solaris/Illumos systems (using an #error directive if it is not detected), with the other methods removed. The result performs 15-25% faster on macOS and Linux while passing all the regression tests. On macOS, avoiding the select(2) method fixes the hanging bug. On SunOS/Solaris/Illumos (the '__sun' identifier), the method using ioctl(2)+I_PEEK and select(2) (iffe feature IDs: stream_peek and lib_select) is preserved. Resolves: https://github.com/ksh93/ksh/issues/118 (again) [*] https://pubs.opengroup.org/onlinepubs/9699919799/functions/recv.html
2020-08-19 20:58:46 +00:00
2020-08-19:
- Sped up the 'read' command on most systems by 15-25%. Fixed a hanging bug
on reading from a FIFO that could occur on macOS.
Fix 'command -p' lookup if hash table entry exists (re: c9ccee86) If a command's path was previously added to the hash table as a 'tracked alias', then the hash table entry was used, bypassing the default utility path search activated by 'command -p'. 'command -p' activates a SH_DEFPATH shell state. The bug was caused by a failure to check for this state before using the hash table. This check needs to be added in four places. src/cmd/ksh93/sh/path.c, src/cmd/ksh93/sh/xec.c: - path_search(), path_spawn(), sh_exec(), sh_ntfork(): Only consult the hash table, which is shp->track_tree, if the SH_DEFPATH shell state is not active. src/cmd/ksh93/tests/path.sh: - Add regress tests checking that 'command -p' and 'command -p -v' still search in the default path if a hash table entry exists for the command searched.
2020-08-17 19:23:39 +00:00
2020-08-17:
- 'command -p' incorrectly used the hash table entry (a.k.a. tracked alias)
for a command if its path was previously hashed. It has now been fixed so
it never consults the hash table.
Fix 'command -x' on macOS, Linux, Solaris 'command -x' (basically builtin xargs for 'command') worked for long argument lists on *BSD and HP-UX, but not on macOS and Linux, where it reliably entered into an infinite loop. The problem was that it assumed that every byte of the environment space can be used for arguments, without accounting for alignment that some OSs do. MacOS seems to be the most wasteful one: it aligns on 16-byte boundaries and requires some extra bytes per argument as well. src/cmd/ksh93/sh/path.c: - path_xargs(): When calculating how much space to subtract per argument, add 16 extra bytes to the length of each argument, then align the result on 16-byte boundaries. The extra 16 bytes is more than even macOS needs, but hopefully it is future-proof. - path_spawn(): If path_xargs() does fail, do not enter a retry loop (which always becomes an infinite loop if the argument list exceeds OS limitations), but abort with an error message.
2020-08-16 08:08:07 +00:00
2020-08-16:
- Fixed 'command -x' on macOS, Linux and Solaris by accounting for a 16-byte
argument alignment. If execution does fail, it now aborts with an internal
error message instead of entering an infinite retry loop.
Fix leak and crash upon defining functions in subshells A memory leak occurred upon leaving a virtual subshell if a function was defined within it. If this was done more than 32766 (= 2^15-2 = the 'short' max value - 1) times, the shell crashed. Discussion and reproducer: https://github.com/ksh93/ksh/issues/114 src/cmd/ksh93/sh/subshell.c: table_unset(): - A subshell-defined function was never freed because a broken check for autoloaded functions (which must not be freed[*]). It looked for an initial '/' in the canonical path of the script file that defined the function, but that path is also stored for regular functions. Now use a check that executes nv_search() in fpathdict, the same method used in _nv_unset() in name.c for a regular function unset. src/cmd/ksh93/bltins/misc.c: b_dot_cmd(): - Fix an additional memory leak introduced in bd88cc7f, that caused POSIX functions (which are run with b_dot_cmd() like dot scripts) to leak extra. This fix avoids both the crash fixed there and the memory leak by introducing a 'tofree' variable remembering the filename to free. Thanks to Johnothan King for the patch. src/lib/libast/include/stk.h, src/lib/libast/misc/stk.c, src/lib/libast/man/stk.3, src/lib/libast/man/stak.3: - Make the stack more resilient by extending the stack reference counter 'stkref' from (signed) short to unsigned int. On modern systems with 32-bit ints, this extends the maximum number of elements on a stack from 2^15-1==32767 to 2^32-1==4294967295. The ref counter can never be negative, so there is no reason for signedness. sizeof(int) is defined as the size of a single CPU word, so this should not affect performance at all. On a 16-bit system (not that ksh still compiles there), this doubles the max number of entries to 2^16-1=65535. src/cmd/ksh93/tests/leaks.sh: - Add leak regression tests for ksh functions, POSIX functions, dot scripts run with '.', and dot scripts run with 'source'. src/cmd/ksh93/tests/path.sh: - Add an output builtin with a redirect to an autoloaded function so that a crash[*] is triggered if the check for an autoloaded function is ever removed from table_unset(), as was done in ksh 93v- (which crashed). [*] Freeing autoloaded functions after leaving a virtual subshell causes a crashing bug: https://github.com/att/ast/issues/803 Co-authored-by: Johnothan King <johnothanking@protonmail.com> Fixes: https://github.com/ksh93/ksh/issues/114
2020-08-13 22:29:27 +00:00
2020-08-13:
- Fixed memory leaks and a crashing bug that occurred when defining and
running functions in subshells.
Fix crash upon running many subshells (#113) Co-authored-by: Martijn Dekker <martijn@inlv.org> An intermittent crash occurred after running many thousands of virtual/non-forked subshells. One reproducer is a crash in the shbench fibonacci.ksh test, as documented here: https://github.com/ksh-community/shbench/blob/f3d9e134/bench/fibonacci.ksh#L4-L10 The apparent cause was the signed and insufficiently large 'short' data type of 'curenv' and related variables which wrapped around to a negative number when overflowing. These IDs are necessary for the 'wait' builtin to obtain the exit status from a background job. This fix is inspired by a patch based on ksh 93v-: https://build.opensuse.org/package/view_file/shells/ksh/ksh93-longenv.dif?expand=1 https://src.fedoraproject.org/rpms/ksh/blob/f24/f/ksh-20130628-longer.patch However, we change the type to 'unsigned int' instead of 'long'. On all remotely modern systems, ints are 32-bit values, and using this type avoids a performance degradation on 32-bit sytems. Making them unsigned prevents an overflow to negative values. src/cmd/ksh93/include/defs.h, src/cmd/ksh93/include/jobs.h, src/cmd/ksh93/include/nval.h, src/cmd/ksh93/include/shell.h: - Change the types of the static global 'subenv' and the subshell structure members 'curenv', 'jobenv', 'subenv', 'p_env' and 'subshell' to one consistent type, unsigned int. src/cmd/ksh93/sh/jobs.c, src/cmd/ksh93/sh/macro.c: src/cmd/ksh93/sh/name.c: src/cmd/ksh93/sh/nvtype.c, src/cmd/ksh93/sh/subshell.c: - Updates to match new variable types. src/cmd/ksh93/tests/subshell.sh: - Show wrong exit status in message on failure of 'wait' builtin.
2020-08-12 17:50:59 +00:00
2020-08-11:
- Fixed an intermittent crash upon running a large number of subshells.
printf: Fix HTML and URI encoding (%H, %#H) This applies a number of fixes to the printf formatting directives %H and %#H (as well as their equivalents %(html)q and %(url)q): 1. Both formatters have been made multibyte/UTF-8 aware, and no longer delete multibyte characters. Invalid UTF-8 byte sequences are rendered as ASCII question marks. 2. %H no longer wrongly encodes spaces as non-breaking spaces (&nbsp;) and instead correctly encodes the UTF-8 non-breaking space as such. 3. %H now converts the single quote (') to '%#39;' instead of '&apos;' which is not a valid entity in all HTML versions. 4. %#H failed to encode some reserved characters (e.g. '?') while encoding some unreserved ones (e.g. '~'). It now percent-encodes all characters except those 'unreserved' as per RFC3986 (ASCII alphanumeric plus -._~). Prior discussion: https://groups.google.com/d/msgid/korn-shell/ce8d1467-4a6d-883b-45ad-fc3c7b90e681%40inlv.org src/cmd/ksh93/include/defs.h: src/cmd/ksh93/sh/string.c: - defs.h: If compiling without SHOPT_MULTIBYTE, redefine the mbwide() macro (which tests if we're in a multibyte locale) as 0. This lets the compiler optimiser do the work that would otherwise require a lot of tedious '#if SHOPT_MULTIBYTE' directives. - string.c: Remove some now-unneeded '#if SHOPT_MULTIBYTE' stuff. - defs.h, string.c: Rename is_invisible() to sh_isprint(), invert the boolean return value, and make it an extern for use in fmthtml() -- see below. If compiling without SHOPT_MULTIBYTE, simply #define sh_isprint() as equivalent to isprint(3). - defs.h: Add URI_RFC3986_UNRESERVED macro for fmthtml() containing the characters "unreserved" for purposes of URI percent-encoding. src/cmd/ksh93/bltins/print.c: fmthtml(): - Remove kludge that skipped all multibyte characters (!). - Complete rewrite to implement fixes described above. - Don't bother with '#if SHOPT_MULTIBYTE' directives (see above). src/cmd/ksh93/data/builtins.c: - sh_optprintf[]: %H: Add single quote to encoded chars doc. - Edit credits and bump version date. src/cmd/ksh93/tests/builtins.sh: - Update and tweak old regression tests. - Add a number of new tests for UTF-8 HTML and URI encoding, which are only run when running tests in a UTF-8 locale (shtests -u).
2020-08-10 21:15:53 +00:00
2020-08-10:
- A number of fixes have been applied to the printf formatting directives
%H and %#H (as well as the undocumented equivalents %(html)q and %(url)q):
1. Both formatters have been made multibyte/UTF-8 aware, and no longer
delete multibyte characters. Invalid UTF-8 byte sequences are rendered
as ASCII question marks.
2. %H no longer wrongly changes spaces to non-breaking spaces (&nbsp;).
3. %H now converts the single quote (') to '%#39;' instead of '&apos;'
which is not a valid entity in all HTML versions.
4. %#H failed to encode some reserved characters (e.g. '?') while encoding
some unreserved ones (e.g. '~'). It now percent-encodes all characters
except those 'unreserved' as per RFC3986 (ASCII alphanumeric plus -._~).
Fix crash, take three (re: e805c7d9, 33858689) The current fix appears to be only partially successful in eliminating the intermittent crash, and also breaks '-o notify' during the 60-second $TMOUT grace period. This replaces it. The root cause appears to be that the state of job control becomes somehow inconsistent when running external commands in a command substitution expanded from the $PS1 prompt. The job_unpost() or (sometimes) the job_list() function intermittently crash. These are called if the SH_TTYWAIT state is active: https://github.com/ksh93/ksh/blob/88e8fa67/src/cmd/ksh93/sh/jobs.c#L463-L469 Temporarily deactivating the SSH_TTYWAIT state while expanding PS{1..4} prompts appears to fix the problem reliably. It is quite possible that this fix merely masks a bug in the job control system, but testing has shown that it stops ksh crashing without side effects, so I'm calling it good for now. Thanks to Marc Wilson for many hours of persistent testing. src/cmd/ksh93/sh/jobs.c: - Revert changes made in 33858689 and e805c7d9. src/cmd/ksh93/sh/io.c: io_prompt(): - Save SH_TTYWAIT state and turn it off while expanding prompts. Resolves: https://github.com/ksh93/ksh/issues/103 Resolves: https://github.com/ksh93/ksh/issues/112
2020-08-11 00:02:31 +00:00
- Fixed a crash that occurred intermittently after running an external
command from a command substitution expanded from the $PS1 shell prompt.
Skip '.' and '..' when globbing patterns like .* There are convincing arguments why including '.' and '..' in the result of pathname expansion is actively harmful. See: https://www.austingroupbugs.net/view.php?id=1228 https://github.com/ksh93/ksh/issues/58#issuecomment-653716846 pdksh, mksh and zsh already skip these special traversal names in all cases. This commit makes ksh act like these shells. Since passing '.' and especially '..' as arguments to commands like 'chmod -R' and 'cp -r' may cause harm, this change seems likely to fix more legacy scripts than it breaks. I'm unaware of anyone ever having come up with a concrete use case for the old behaviour. This change also fixes the bug that '.' and '..' failed to be ignored as documented if FIGNORE is set. src/lib/libast/misc/glob.c: glob_dir(): - Explicitly skip any matching '.' and '..' in all cases. src/cmd/ksh93/tests/glob.sh: - Add test_glob() tests for '*' and '.*'. src/cmd/ksh93/sh.1: File Name Generation: - Update to match new behaviour. Resolves: https://github.com/ksh93/ksh/issues/58
2020-08-09 23:02:23 +00:00
2020-08-09:
- File name generation (a.k.a. pathname expansion, a.k.a. globbing) now
never matches the special navigational names '.' (current directory) and
'..' (parent directory). This change makes a pattern like .* useful; it
now matches all hidden files (dotfiles) in the current directory, without
the harmful inclusion of '.' and '..'.
redirect: check args before executing redirections (re: 7b82c338) The 'redirect' builtin command did not error out before executing any valid redirections. For example, 'redirect ls >foo.txt' issued an "incorrect syntax" error, but still created 'foo.txt' and left standard output permanently redirected to it. src/cmd/ksh93/sh/xec.c: sh_exec(): - If we have redirections (io != NULL), and the command is SYSREDIR, then check for arguments and error out if there are any, before calling sh_redirect() to execute redirections. (Note, the other check for arguments in b_exec() in bltins/misc.c must be kept, as that applies if there are no redirections.) src/cmd/ksh93/sh/io.c: sh_redirect(): - Edit comments to better explain what the flag values do. src/cmd/ksh93/bltins/misc.c: - Add a dummy b_redirect() function declaration "for the dictionary generator" as has historically been done for other builtins that share one C function. I'm not sure what that dictionary generator is supposed to be, but this also improves greppability. src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/sh.1: - Fix misleading "I/O redirection arguments" term. I/O redirections are not arguments at all; no argument parser ever sees them. src/cmd/ksh93/tests/io.sh: - Test both conditions that should make 'redirect' produce an "incorrect syntax" error. - Test that any redirections are not executed if erroneous non-redirection arguments exist. src/cmd/ksh93/tests/builtins.sh: - "... should show usage info on unrecognized options" test: Because 'redirect' now refuses to process redirections on error, the error message was not captured. The fix is to run the builtin in a braces block and add the redirection to the block.
2020-08-08 22:57:57 +00:00
2020-08-08:
- Argument checking in the 'redirect' builtin command (see 2020-06-11) has
been improved to error out before executing redirections. For example, an
error like 'redirect ls >foo.txt' now will not create 'foo.txt' and will
not leave your standard output permanently redirected to it.
Fix crash: do not list job if in 60 sec grace period (re: 33858689) The crash in job_list() or job_unpost() could still occur after the previous patch if a signal was being handled after $TMOUT was exceeded and the 60-second grace period was entered. It *should* work to add a general check for !sh_isstate(SH_GRACE). We know that the SH_GRACE state is set immediately after printing the 60 second grace period warning message: https://github.com/ksh93/ksh/blob/9de65210/src/cmd/ksh93/sh/io.c#L1869-L1870 (and that the crashes occur upon re-evaluating the $PS1 prompt after setting the SH_GRACE state). We know that the SH_GRACE state is not turned off again until either the user enters a line: https://github.com/ksh93/ksh/blob/9de65210/src/cmd/ksh93/sh/main.c#L474 or the shell times out after the grace period: https://github.com/ksh93/ksh/blob/9de65210/src/cmd/ksh93/sh/io.c#L1861 The SH_GRACE state flag is not used or changed in any other context (verified with grep -rn SH_GRACE src/cmd/ksh93). So, logically, this should suffice to make sure the crash stays gone. src/cmd/ksh93/sh/jobs.c: job_reap(): - Do not list jobs when the SH_GRACE state (the 60 second timeout grace period after TMOUT was exceeded) is active. - Keep the previous check for job control just to be sure, and because it makes sense. Fixes: https://github.com/ksh93/ksh/issues/103 (again)
2020-08-07 20:09:01 +00:00
2020-08-06:
Add ${.sh.pid} as an alternative to $BASHPID (#109) This variable is like Bash's $BASHPID, but in virtual subshells it will retain its previous value as virtual subshells don't fork. Both $BASHPID and ${.sh.pid} are different from $$ as the latter is only set to the parent shell's process ID (i.e. it isn't set to the process ID of the current subshell). src/cmd/ksh93/include/defs.h: - Add 'current_pid' for storing the current process ID at a valid memory address. - Change 'ppid' from 'int32_t' to 'pid_t', as the return value from 'getppid' is of the 'pid_t' data type. src/cmd/ksh93/data/variables.c, src/cmd/ksh93/include/variables.h, src/cmd/ksh93/sh/init.c, src/cmd/ksh93/sh/xec.c: - Add the ${.sh.pid} variable as an alternative to $BASHPID. The process ID is stored in a struct before ${.sh.pid} is set as environment variables are pointers that must point to a valid memory address. ${.sh.pid} is updated by the _sh_fork() function, which is called when ksh forks a new process with sh_fork() or sh_ntfork(). src/cmd/ksh93/tests/variables.sh: - Add ${.sh.pid} to the list of special variables and add three regression tests for ${.sh.pid}. src/cmd/ksh93/tests/subshell.sh: - Update the PATH forking regression test to use ${.sh.pid} and remove the TODO note.
2020-08-07 01:53:25 +00:00
- Added the '${.sh.pid}' variable as an alternative to Bash's '$BASHPID'.
This variable is set to the current shell's PID, unlike '$$' (which is
Various minor capitalization and typo fixes (#371) This commit fixes various minor typos, punctuation errors and corrects the capitalization of many names.
2021-12-12 21:40:38 +00:00
set to the main shell's PID). In virtual subshells '${.sh.pid}' is not
Add ${.sh.pid} as an alternative to $BASHPID (#109) This variable is like Bash's $BASHPID, but in virtual subshells it will retain its previous value as virtual subshells don't fork. Both $BASHPID and ${.sh.pid} are different from $$ as the latter is only set to the parent shell's process ID (i.e. it isn't set to the process ID of the current subshell). src/cmd/ksh93/include/defs.h: - Add 'current_pid' for storing the current process ID at a valid memory address. - Change 'ppid' from 'int32_t' to 'pid_t', as the return value from 'getppid' is of the 'pid_t' data type. src/cmd/ksh93/data/variables.c, src/cmd/ksh93/include/variables.h, src/cmd/ksh93/sh/init.c, src/cmd/ksh93/sh/xec.c: - Add the ${.sh.pid} variable as an alternative to $BASHPID. The process ID is stored in a struct before ${.sh.pid} is set as environment variables are pointers that must point to a valid memory address. ${.sh.pid} is updated by the _sh_fork() function, which is called when ksh forks a new process with sh_fork() or sh_ntfork(). src/cmd/ksh93/tests/variables.sh: - Add ${.sh.pid} to the list of special variables and add three regression tests for ${.sh.pid}. src/cmd/ksh93/tests/subshell.sh: - Update the PATH forking regression test to use ${.sh.pid} and remove the TODO note.
2020-08-07 01:53:25 +00:00
changed from its previous value, while in forked subshells '${.sh.pid}'
is set to the subshell's process ID.
Over-shifting in a POSIX function should cause scripts to exit (#106) The required longjmp used to terminate scripts was not being run when over-shifting in a POSIX function with a redirection. This caused scripts to continue after an error in the shift builtin, which is incorrect since shift is a special builtin. The interpreter is sent into an indeterminate state that causes undefined behavior as well: $ cat reproducer.ksh some_func() { shift 10 } for i in a b c d e f; do echo "read $i" [ "$i" != "c" ] && continue some_func 2>&1 echo "$i = c" done $ ksh ./reproducer.ksh read a read b read c /tmp/k[2]: shift: 10: bad number c = c read d /tmp/k[2]: shift: 10: bad number d = c read e /tmp/k[2]: shift: 10: bad number e = c read f /tmp/k[2]: shift: 10: bad number f = c src/cmd/ksh93/sh/xec.c: sh_exec(): - Do the necessary longjmp needed to terminate the script after over-shifting in a POSIX function when the function call has a redirection. src/cmd/ksh93/tests/functions.sh: - Add the over-shifting regression test from ksh93v- 2013-10-10-alpha. Bug report and fix on the old mailing list: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg00732.html
2020-08-05 17:06:16 +00:00
2020-08-05:
Fix unset method in multidimensional arrays (#105) A segfault happens when an array with an unset method is turned into a multidimensional array. Reproducer: function foo { typeset -a a a.unset() { print unset } a[3][6][11][20]=7 } foo src/cmd/ksh93/sh/nvdisc: - Fix the multidimensional array unset method crash by checking if np->nvenv is an array, since multidimensional arrays need to be handled as arrays. This bugfix was backported from ksh93v- 2013-10-10-alpha. src/cmd/ksh93/tests/arrays2.sh: - Add the reproducer as a regression test for the crash with multidimensional arrays. Bug report on the old mailing list: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01195.html
2020-08-05 17:14:30 +00:00
- Fixed a bug in functions that caused ksh to crash when an array with an
unset method was turned into a multidimensional array.
Over-shifting in a POSIX function should cause scripts to exit (#106) The required longjmp used to terminate scripts was not being run when over-shifting in a POSIX function with a redirection. This caused scripts to continue after an error in the shift builtin, which is incorrect since shift is a special builtin. The interpreter is sent into an indeterminate state that causes undefined behavior as well: $ cat reproducer.ksh some_func() { shift 10 } for i in a b c d e f; do echo "read $i" [ "$i" != "c" ] && continue some_func 2>&1 echo "$i = c" done $ ksh ./reproducer.ksh read a read b read c /tmp/k[2]: shift: 10: bad number c = c read d /tmp/k[2]: shift: 10: bad number d = c read e /tmp/k[2]: shift: 10: bad number e = c read f /tmp/k[2]: shift: 10: bad number f = c src/cmd/ksh93/sh/xec.c: sh_exec(): - Do the necessary longjmp needed to terminate the script after over-shifting in a POSIX function when the function call has a redirection. src/cmd/ksh93/tests/functions.sh: - Add the over-shifting regression test from ksh93v- 2013-10-10-alpha. Bug report and fix on the old mailing list: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg00732.html
2020-08-05 17:06:16 +00:00
- Fixed a bug that caused scripts to continue running after over-shifting
in a function when the function call had a redirection.
Fix shellquoting of invalid multibyte char (re: f9d28935, 8c7c60ec) This commit fixes two bugs in the generation of $'...' shellquoted strings: 1. A bug introduced in f9d28935. In UTF-8 locales, a byte that is invalid in UTF-8, e.g. hex byte 86, would be shellquoted as \u[86], which is not the same as the correct quoting, \x86. 2. A bug inherited from 93u+. Single bytes (e.g. hex 11) were always quoted as \x11 and not \x[11], even if a subsequent character was a hexadecimal digit. However, the parser reads past two hexadecimal digits, so we got: $ printf '%q\n' $'\x[11]1' $'\x111' $ printf $'\x111' | od -t x1 0000000 c4 91 0000002 After the bug fix, this works correctly: $ printf '%q\n' $'\x[11]1' $'\x[11]1' $ printf $'\x[11]1' | od -t x1 0000000 11 31 0000002 src/cmd/ksh93/sh/string.c: sh_fmtq(): - Make the multibyte code for $'...' more readable, eliminating the 'isbyte' flag. - When in a multibyte locale, make sure to shellquote both invalid multibyte characters and unprintable ASCII characters as hexadecimal bytes (\xNN). This reinstates 93u+ behaviour. - When quoting bytes, use isxdigit(3) to determine if the next character is a hex digit, and if so, protect the quoted byte with square brackets. src/cmd/ksh93/tests/quoting2.sh: - Move the 'printf %q' shellquoting regression tests here from builtins.sh; they test the shellquoting algorithm, not so much the printf builtin itself. - Add regression tests for these bugs.
2020-08-05 17:22:22 +00:00
- When generating shellquoted strings (such as with 'printf %q'), the
hexadecimal value of a quoted unprintable character was not protected with
square braces, e.g. 0x12 followed by '3' would be quoted as '\x123', which
is a different value. Such strings are now quoted like '\x[12]3' if the
next character is a hexadecimal digit.
Fix creation of extra associative array element '0' (#101) Multidimensional associative arrays are created with an extra array member named '0', which is set to no value. Reproducer: $ typeset -A foo $ typeset -A foo[bar] $ typeset -p foo typeset -A foo=([bar]=([0]='') ) The bugfix prevents nv_setarray from creating the extra '[0]' member when an associative array is empty. This bug was discussed on the old mailing list: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01574.html src/cmd/ksh93/sh/array.c: - Do not allow the creation of an extra array member when an array is empty. src/cmd/ksh93/tests/arrays.sh: - Add a regression test for creating multidimensional associative arrays, but use the output from 'typeset -p' instead of fgrep.
2020-07-31 16:32:09 +00:00
2020-07-31:
- Fixed a bug that caused multidimensional associative arrays to be created
with an extra array member.
Fix expansion of multibyte character after $1 - $9, $?, etc (#102) A multibyte character immediately following an expansion of a single-character name, e.g. $1 through $9, $?, $-, etc. was corrupted when in a UTF-8 locale, e.g.: $ set -- foo; echo "$1テスト" foo?スト Prior discussion: https://www.mail-archive.com/ast-users@lists.research.att.com/msg01060.html https://bugzilla.redhat.com/show_bug.cgi?id=1256495 src/cmd/ksh93/sh/macro.c: - Apply a Red Hat patch by Paulo Andrade that avoids calling fcmbget() if backtracking more than one byte might be required. src/cmd/ksh93/tests/basic.c: - Test "テスト" following expansion of "$1", "$?" and "$#". Co-authored-by: Martijn Dekker <martijn@inlv.org>
2020-08-01 00:12:45 +00:00
- Fixed a bug that caused the expansions of positional parameters $1 - $9,
as well as special parameters such as $? and $-, to corrupt any multibyte
characters immediately following the expansion if a UTF-8 locale is active.
Fix blocked signals after fork(2)ing external command in subshell When the classic fork/exec mechanism was used (via sh_fork()) to run an external command from within a non-forking subshell, SIGINT was blocked until that subshell was exited. If a subsequent loop was run in the subshell, it became uninterruptible, e.g.: $ arch/*/bin/ksh -c '(/usr/bin/true; while :; do :; done); exit' ^C^C^C^C^C src/cmd/ksh93/sh/xec.c: - sh_fork() did not reset the savesig variable in the parent part of the fork when running in a virtual subshell. This had the effect of delaying signal handling until exiting the subshell. There is no reason for that subshell check that I can discern, so this removes it. I've verified that this causes no regression test failures even when ksh is compiled with -DSHOPT_SPAWN=0 which means the classic fork/exec mechanism is always used. Fixes: https://github.com/ksh93/ksh/issues/86
2020-07-30 00:22:11 +00:00
2020-07-29:
- On a ksh compiled to use fork(2) to run external commands, a bug has been
fixed that caused signals (such as SIGINT, Ctrl+C) to be ignored within a
non-forked subshell after running an external command within that subshell.
Add support for multibyte characters to $IFS (#92) Add support for multibyte characters to $IFS This commit fixes BUG_MULTIBIFS, which had two bug reports in the ksh2020 branch. src/cmd/ksh93/sh/macro.c: - Backport Eric Scrivner's fix for multibyte IFS characters (slightly modified for compatibility with C89). Explanation from https://github.com/att/ast/pull/737: Previously, the varsub method used for the macro expansion of $param, ${param}, and ${param op word} would incorrectly expand the internal field separator (IFS) if it was a multibyte character. This was due to truncation based on the incorrect assumption that the IFS would never be larger than a single byte. This change fixes this issue by carefully tracking the number of bytes that should be persisted in the IFS case and ensuring that all bytes are written during expansion and substitution. Bug report: https://github.com/att/ast/issues/13 - Fixed another bug that caused multibyte characters with the same initial byte to be treated as the same character by the IFS. This bug was occurring because the first byte of a multibyte character wasn't being written to the stack when the IFS delimiter had the same initial byte: $ IFS=£ $ v='§' $ set -- $v $ v="${1-}" $ echo "$v" | hd # The first byte should be c2, but it isn't due to the bug 00000000 a7 0a |..| 00000002 Bug report: https://github.com/att/ast/issues/1372 src/cmd/ksh93/tests/variables.sh: - Add (reworked) regression tests from ksh2020 for the multibyte IFS bugs. - Add a regression test for att/ast#1372 based on the reproducer.
2020-07-25 18:46:11 +00:00
2020-07-25:
- Fixed BUG_MULTIBIFS: Multibyte characters can now be used as IFS
delimiters. "$*" was incorrectly joining positional parameters on
the first byte of a multibyte character. This was due to truncation
based on the incorrect assumption the IFS would never be larger
than a single byte.
Fix how unrecognized options are handled in 'sleep' and 'suspend' (#93) When a builtin is given an unrecognized option, the usage information for that builtin should be shown as 'Usage: builtin-name options'. The sleep and suspend builtins were an exception to this. 'suspend' would not show usage information and sleep wouldn't exit on error: $ suspend -e /usr/bin/ksh: suspend: -e: unknown option $ time sleep -e 1 sleep: -e: unknown option real 0m1.00s user 0m0.00s sys 0m0.00s src/cmd/ksh93/bltins/sleep.c: - Show usage information and exit when sleep is given an unknown option. This bugfix was backported from ksh2020: https://github.com/att/ast/pull/1024 src/cmd/ksh93/bltins/trap.c: - Use the normal method of parsing options with optget to fix the suspend builtin's test failure. src/cmd/ksh93/tests/builtins.sh: - Add the ksh2020 regression test for getting the usage information of each builtin. Enable all /opt/ast/bin builtins in a subshell since those should be tested as well (aside from getconf and uname because those builtins fallback to the real commands on error).
2020-07-26 01:18:49 +00:00
- Fixed a bug that caused the sleep builtin to continue after being given
an unrecognized option. 'sleep -: 1' will now show a usage message and
exit instead of sleep for one second.
Fix typeset attributes -a, -A, -l, -u leaking out of subshells If an array or upper/lowercase variable was declared with a null initial value within a virtual/non-forked subshell, like: ( typeset -a foo; ... ) ( typeset -A foo; ... ) ( typeset -l foo; ... ) ( typeset -u foo; ... ) then the type declaration leaked out of the subshell into the parent shell environment, though without any values that may subsequently have been assigned. src/cmd/ksh93/bltins/typeset.c: setall(): - When deciding whether to create a virtual subshell scope for a variable, use sh_assignok(), which was actually designed for the purpose, instead of _nv_unset(). This allows getting rid of a tangled mess of special-casing that never worked quite right. src/cmd/ksh93/tests/arrays.sh: - Add regression tests checking that array declarations don't leak out of virtual subshells. src/cmd/ksh93/tests/attributes.sh: - Add regression tests for combining the 'export' and 'readonly' attributes with every other possible typeset attribute on unset variables. This also includes a subshell leak test for each one. Fixes: https://github.com/ksh93/ksh/issues/88
2020-07-25 15:39:12 +00:00
- Fixed a bug that caused the 'typeset' variable attributes -a, -A, -l, and
-u to leak out of a subshell if they were set without assigning a value.
Fix command substitutions run on the same line as a here-doc (#91) When a command substitution is run on the same line as a here-document, a syntax error occurs due to a regression introduced in ksh93u+ 2011-04-15: true << EOF; true $(true) EOF syntax error at line 1: `<<EOF' here-document not contained within command substitution The regression is caused by an error check that was added to make the following script causes a syntax error (because the here-document isn't completed inside of the command substitution): $(true << EOF) EOF src/cmd/ksh93/sh/lex.c: - Only throw an error when a here-document in a command substitution isn't completed inside of the command substitution. src/cmd/ksh93/tests/heredoc.sh: - Add a regression test for running a command substitution on the same line as a here-document. - Add a missed regression test for using here-documents in command substitutions. This is the original bug that was fixed in ksh93u+ 2011-04-15 (it is why the error message was added), but a regression test for here-documents in command substitutions wasn't added in that version. This bugfix was backported from ksh93v- 2013-10-10-alpha.
2020-07-23 23:03:57 +00:00
2020-07-23:
Fix an infinite loop related to $_ if ksh is /bin/sh (#90) The following explanation is mostly taken from Tomas Klacko's report on the old mailing list (which also contains a C program reproducer) [*]: 1. When ksh starts a binary, it sets its environment variable "_" to "*number*/path/to/binary". Where "number" is the pid of the ksh process. 2. The binary forks and the child executes a suid root shell script which begins with #!/bin/sh. For this bug to occur, ksh must be /bin/sh. 3. The ksh process interpreting the suid shell script leaves the "_" variable as not set (nv_getval(L_ARGNOD) returns NULL) because the "number" from step 1 is not the pid of its parent process. 4-5. Because "_" is not set and the script is suid root, an infinite loop occurs because when the SHELL environment variable contains "/bin/sh" pathshell() returns "/bin/sh". This becomes an infinite loop of /bin/sh /dev/fd/3 executing /bin/sh /dev/fd/3. src/cmd/ksh93/sh/init.c: get_lastarg(): - Disable the check for if the "number" refers to the process id of the parent process. src/cmd/ksh93/sh/main.c: sh_main(): - Prevent an infinite loop when '$_' is not passed in from the environment. Solaris applies this bugfix to their version of ksh: https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/190-17432413.patch [*]: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01680.html
2020-07-24 00:20:26 +00:00
- Fixed an infinite loop that could occur when ksh is the system's /bin/sh.
Fix command substitutions run on the same line as a here-doc (#91) When a command substitution is run on the same line as a here-document, a syntax error occurs due to a regression introduced in ksh93u+ 2011-04-15: true << EOF; true $(true) EOF syntax error at line 1: `<<EOF' here-document not contained within command substitution The regression is caused by an error check that was added to make the following script causes a syntax error (because the here-document isn't completed inside of the command substitution): $(true << EOF) EOF src/cmd/ksh93/sh/lex.c: - Only throw an error when a here-document in a command substitution isn't completed inside of the command substitution. src/cmd/ksh93/tests/heredoc.sh: - Add a regression test for running a command substitution on the same line as a here-document. - Add a missed regression test for using here-documents in command substitutions. This is the original bug that was fixed in ksh93u+ 2011-04-15 (it is why the error message was added), but a regression test for here-documents in command substitutions wasn't added in that version. This bugfix was backported from ksh93v- 2013-10-10-alpha.
2020-07-23 23:03:57 +00:00
- A command substitution that is run on the same line as a here-document
will no longer cause a syntax error.
Fix race conditions running external commands with job control on When ksh is compiled with SHOPT_SPAWN (the default), which uses posix_spawn(3) or vfork(2) (via sh_ntfork()) to launch external commands, at least two race conditions occur when launching external commands while job control is active. See: https://bugs.launchpad.net/ubuntu/+source/ksh/+bug/1887863/comments/3 https://www.mail-archive.com/ast-developers@research.att.com/msg00717.html The basic issue is that this performance optimisation is incompatible with job control, because it uses a spawning mechanism that doesn't copy the parent process' memory pages into the child process, therefore no state that involves memory can be set before exec-ing the external program. This makes it impossible to correctly set the terminal's process group ID in the child process, something that is essential for job control to work. src/cmd/ksh93/sh/xec.c: - Use sh_fork() instead of sh_ntfork() if job control is active. This uses fork(2), which is 30%-ish slower on most sytems, but allows for correctly setting the terminal process group. src/cmd/ksh93/tests/basic.sh: - Add regression test for the race condition reported in #79. src/cmd/INIT/cc.darwin: - Remove hardcoded flag to disable SHOPT_SPAWN on the Mac. It should be safe to use now. Fixes https://github.com/ksh93/ksh/issues/79
2020-07-22 11:47:04 +00:00
2020-07-22:
- Fixed two race conditions when running external commands on
interactive shells with job control active.
whence -a/type -a: report both function and built-in by same name 'whence -a' is documented to list all possible interpretations of a command, but failed to list a built-in command if a shell function by the same name exists or is marked undefined using 'autoload'. src/cmd/ksh93/bltins/whence.c: whence(): - Refactor and separate the code for reporting functions and built-in commands so that both can be reported for one name. src/cmd/ksh93/data/builtins.c: sh_optwhence[]: - Correct 'whence --man' to document that: * 'type' is equivalent to 'whence -v' * '-a' output is like '-v' src/cmd/ksh93/tests/builtins.sh: - Test 'whence -a' with these combinations: * a function, built-in and external command * an undefined/autoload function, built-in and external command Fixes https://github.com/ksh93/ksh/issues/83
2020-07-20 18:49:32 +00:00
2020-07-20:
- If a shell function and a built-in command by the same name exist,
'whence -a' and 'type -a' now report both.
Fix subshell file descriptor leak A file descriptor (at least 3, can't reproduce for 4 and up) opened with 'exec' or 'redirect' in a virtual/non-forked subshell survived that subshell after exiting it: $ ksh -c '(redirect 3>&1); echo bug >&3' bug src/cmd/ksh93/sh/io.c: - Apply a patch from OpenSUSE (ksh93-redirectleak.dif). Source: https://build.opensuse.org/package/show/openSUSE:Leap:42.3:Update/ksh src/cmd/ksh93/tests/io.sh: - Add regression test. Thanks to Marc Wilson for flagging this up.
2020-07-21 03:02:47 +00:00
- Fixed a bug that caused file descriptors opened with 'redirect' or 'exec'
to survive a subshell environment after exiting it.
Fix two crashes related to kshdb (#82) This commit fixes two different crashes related to kshdb: - When redirect is given an invalid file descriptor, a segfault no longer occurs. Reproducer: $ ksh -c 'redirect 9>&200000000000' - Fix a crash due to free(3) being used on an invalid pointer. This can be reproduced with kshdb (commands from att/ast#582): $ git clone https://github.com/rocky/kshdb.git $ cd kshdb $ ksh autogen.sh $ echo "print hi there" > $HOME/.kshdbrc $ ./kshdb -L . test/example/dbg-test1.sh src/cmd/ksh93/bltins/misc.c: b_dot_cmd(): - The string pointed to by shp->st.filename must be able to be freed from memory with free(3), so duplicate the string with strdup(3). src/cmd/ksh93/sh/io.c: sh_redirect(): - Show an error message when a file descriptor is invalid to fix a memory fault.
2020-07-19 22:42:12 +00:00
2020-07-19:
Fix a large number of typos and other problems (#110) Most of these fixes are for typos and extra whitespace at the end of lines. These are the notable changes: - Fixed a compatibility issue with how asterisks are displayed using certain fonts. Bug report: https://github.com/att/ast/issues/764 - Fixed a bug in the man page that caused searches for the '|' character to fail. Bug report: https://github.com/att/ast/issues/871 - Removed a duplicate description of 'set -B' from the man page. Bug report: https://github.com/att/ast/issues/789 - Added documentation for options missing from the ksh man page (applies to 'hist -N', 'sleep -s', 'whence -q' and many of ulimit's options). Bug reports: https://github.com/att/ast/issues/948 https://github.com/att/ast/issues/503#issuecomment-386649715 https://github.com/att/ast/issues/507#issuecomment-507924608 - Applied the following ksh2020 documentation fixes: https://github.com/att/ast/pull/351 https://github.com/att/ast/pull/352 - Fixed a minor GCC -Wformat warning in procopen.c by changing a sentinel to NULL.
2020-08-06 23:50:11 +00:00
- Fixed a crash that occurred in the '.' command when using kshdb.
Fix two crashes related to kshdb (#82) This commit fixes two different crashes related to kshdb: - When redirect is given an invalid file descriptor, a segfault no longer occurs. Reproducer: $ ksh -c 'redirect 9>&200000000000' - Fix a crash due to free(3) being used on an invalid pointer. This can be reproduced with kshdb (commands from att/ast#582): $ git clone https://github.com/rocky/kshdb.git $ cd kshdb $ ksh autogen.sh $ echo "print hi there" > $HOME/.kshdbrc $ ./kshdb -L . test/example/dbg-test1.sh src/cmd/ksh93/bltins/misc.c: b_dot_cmd(): - The string pointed to by shp->st.filename must be able to be freed from memory with free(3), so duplicate the string with strdup(3). src/cmd/ksh93/sh/io.c: sh_redirect(): - Show an error message when a file descriptor is invalid to fix a memory fault.
2020-07-19 22:42:12 +00:00
Fix a large number of typos and other problems (#110) Most of these fixes are for typos and extra whitespace at the end of lines. These are the notable changes: - Fixed a compatibility issue with how asterisks are displayed using certain fonts. Bug report: https://github.com/att/ast/issues/764 - Fixed a bug in the man page that caused searches for the '|' character to fail. Bug report: https://github.com/att/ast/issues/871 - Removed a duplicate description of 'set -B' from the man page. Bug report: https://github.com/att/ast/issues/789 - Added documentation for options missing from the ksh man page (applies to 'hist -N', 'sleep -s', 'whence -q' and many of ulimit's options). Bug reports: https://github.com/att/ast/issues/948 https://github.com/att/ast/issues/503#issuecomment-386649715 https://github.com/att/ast/issues/507#issuecomment-507924608 - Applied the following ksh2020 documentation fixes: https://github.com/att/ast/pull/351 https://github.com/att/ast/pull/352 - Fixed a minor GCC -Wformat warning in procopen.c by changing a sentinel to NULL.
2020-08-06 23:50:11 +00:00
- Fixed a crash that occurred when attempting to use redirection with an
Fix two crashes related to kshdb (#82) This commit fixes two different crashes related to kshdb: - When redirect is given an invalid file descriptor, a segfault no longer occurs. Reproducer: $ ksh -c 'redirect 9>&200000000000' - Fix a crash due to free(3) being used on an invalid pointer. This can be reproduced with kshdb (commands from att/ast#582): $ git clone https://github.com/rocky/kshdb.git $ cd kshdb $ ksh autogen.sh $ echo "print hi there" > $HOME/.kshdbrc $ ./kshdb -L . test/example/dbg-test1.sh src/cmd/ksh93/bltins/misc.c: b_dot_cmd(): - The string pointed to by shp->st.filename must be able to be freed from memory with free(3), so duplicate the string with strdup(3). src/cmd/ksh93/sh/io.c: sh_redirect(): - Show an error message when a file descriptor is invalid to fix a memory fault.
2020-07-19 22:42:12 +00:00
invalid file descriptor.
Make the 'history' and 'r' commands builtins (#76) With this change no more preset aliases exist, so the preset alias tables can be safely removed. All ksh commands can now be used without 'unalias -a' removing them, even in interactive shells. Additionally, the history and r commands are no longer limited to being used in interactive shells. src/cmd/ksh93/bltins/hist.c: - Implement the history and r commands as builtins. Also guarantee lflag is set to one by avoiding 'lflag++'. src/cmd/ksh93/Makefile, src/cmd/ksh93/Mamfile, src/cmd/ksh93/sh/main.c, src/cmd/ksh93/sh/init.c, src/cmd/ksh93/data/aliases.c: - Remove the table of predefined aliases because the last few have been removed. During init the alias tree is now initialized the same way as the function tree. src/cmd/ksh93/bltins/typeset.c: - Remove the bugfix for unsetting predefined aliases because it is now a no-op. Aliases are no longer able to have the NV_NOFREE attribute. src/cmd/ksh93/tests/alias.sh: - Remove the regression test for unsetting predefined aliases since those no longer exist. src/cmd/ksh93/data/builtins.c: - Update sh_opthist[] for 'hist --man', etc. src/cmd/ksh93/sh.1: - Remove the list of preset aliases since those no longer exist. - Document history and r as builtins instead of preset aliases. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2020-07-16 17:56:49 +00:00
2020-07-16:
- The 'history' and 'r' default aliases have been made regular built-ins,
leaving zero default aliases.
Fix three bugs in the sleep builtin (#77) This commit backports the main changes to sh_delay from ksh93v- and ksh2020, which fixes the following bugs: - Microsecond amounts of less than one millisecond are no longer ignored. The following loop will now take a minimum of one second to complete: for ((i = 0; i != 10000; i++)) do sleep PT100U done - 'sleep 30' no longer adds an extra 30 milliseconds to the total amount of time to sleep. This bug is hard to notice since 30 milliseconds can be considered within the margin of error. The only reason why longer delays weren't affected is because the old code masked the bug when the interval is greater than 30 seconds: else if(n > 30) { sleep(n); t -= n; } This caused 'sleep -s' to break with intervals greater than 30 seconds, so an actual fix is used instead of a workaround. - 'sleep -s' now functions correctly with intervals of more than 30 seconds as the new code doesn't need the old workaround. This is done by handling '-s' in sh_delay. src/cmd/ksh93/bltins/sleep.c: - Remove the replacement for sleep(3) from the sleep builtin. - Replace the old sh_delay function with the newer one from ksh2020. The new function uses tvsleep, which uses nanosleep(3) internally. src/cmd/ksh93/include/shell.h, src/cmd/ksh93/edit/edit.c, src/cmd/ksh93/sh/jobs.c, src/cmd/ksh93/sh/xec.c, src/cmd/ksh93/shell.3: - Update sh_delay documentation and usage since the function now requires two arguments. src/cmd/ksh93/tests/builtins.sh: - Add a regression test for 'sleep -s' when the interval is greater than 30 seconds. The other bugs can't be tested for in a feasible manner across all systems: https://github.com/ksh93/ksh/pull/72#issuecomment-657215616
2020-07-17 04:00:28 +00:00
- Fixed a bug that caused 'sleep -s' to have no effect with intervals longer
than 30 seconds.
- The accuracy of the sleep builtin has been improved. It no longer ignores
microseconds and doesn't add extra milliseconds when the interval is less
than 31 seconds.
Convert default typeset aliases to regular builtins This converts the 'autoload', 'compound', 'float', 'functions', 'integer' and 'nameref' default aliases into regular built-in commands, so that 'unalias -a' does not remove them. Shell functions can now use these names, which improves compatibility with POSIX shell scripts. src/cmd/ksh93/data/aliases.c: - Remove default typeset aliases. src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/include/builtins.h: - Add corresponding built-in command declarations. Typeset-style commands are now defined by a pointer range, SYSTYPESET .. SYSTYPESET_END. A couple need their own IDs (SYSCOMPOUND, SYSNAMEREF) for special-casing in sh/xec.c. - Update 'typeset --man'. src/cmd/ksh93/bltins/typeset.c: b_typeset(): - Recognise the new builtin commands by argv[0]. Implement them by inserting the corresponding 'typeset' options into the argument list before parsing options. This may seem like a bit of a hack, but it is simpler, shorter, more future-proof and less error-prone than manually copying and adapting all the complex flaggery from the option parsing loop. src/cmd/ksh93/sh/parse.c, src/cmd/ksh93/sh/xec.c: - Recognise typeset-style commands by SYSTYPESET .. SYSTYPESET_END pointer range. - Special-case 'compound' (SYSCOMPOUND) and 'nameref' (SYSNAMEREF) along with recognising the corresponding 'typeset' options. src/cmd/ksh93/sh.1: - Update to document the new built-ins. - Since not all declaration commands are special built-ins now, identify declaration commands using a double-dagger "\(dd" character (which renders as '=' in ASCII) and disassociate their definition from that of special built-ins. src/cmd/ksh93/tests/variables.sh: - Adapt a regression test as there is no more 'integer' alias.
2020-07-15 18:52:01 +00:00
2020-07-15:
- The 'autoload', 'compound', 'float', 'functions', 'integer' and 'nameref'
default aliases have been converted into regular built-in commands, so
that 'unalias -a' does not remove them. Shell functions can now use
these names, which improves compatibility with POSIX shell scripts.
Handle the escape sequence for the End key (#75) Many terminals (xterm being one example) give the Home and End keys the escape sequences '^[[H' and '^[[F'. The first sequence is handled in both editing modes by moving the cursor to start of line, but ksh ignored the second sequence. src/cmd/ksh93/edit/emacs.c, src/cmd/ksh93/edit/vi.c: - Add case labels for '^[[F' so that in both editing modes the End key moves the cursor to the end of the line.
2020-07-15 22:38:44 +00:00
- The End key escape sequence '^[[F' is now handled in the emacs and vi editing
modes. The End key moves the cursor to the end of the line (in contrast to
the Home key doing the opposite).
Restore 'set -b'/'set -o notify' functionality (#74) 'set -b' had no effect; it should cause the shell to notify job state changes immediately instead of waiting for the next prompt. This fixes a regression that was introduced in ksh93t 2008-07-25. The bugfix is from: https://github.com/att/ast/pull/1089 src/cmd/ksh93/sh/jobs.c: - Save the tty wait state and avoid changing it if TTYWAIT was already on to avoid breaking 'set -b'. The last 'sh_offstate' is inside of an '#if' directive because it is only required when ksh is compiled with SHOPT_COSHELL enabled. src/cmd/ksh93/tests/pty.sh: - Add a regression test for 'set -b' in interactive shells.
2020-07-14 21:00:28 +00:00
2020-07-14:
- Fixed a bug that caused 'set -b' to have no effect.
Millisecond precision for 'times' builtin (re: 65d363fd, 5c677a4c) Now that we have an iffe feature test for getrusage(3), introduced in 70fc1da7, the millisecond-precision 'times' command from the last version of ksh2020 can easily be backported. src/cmd/ksh93/bltins/misc.c: - Incorporate ksh2020 'times' command, with a couple of tweaks: * Use locale's radix point instead of '.'. * Pad seconds with initial zero if < 10. src/cmd/ksh93/data/builtins.c: - Update version date for 'times --man'. src/cmd/ksh93/tests/builtins.sh: - Update 'times' test for 3 digits after radix point.
2020-07-15 03:11:33 +00:00
- Following the 'time' keyword, the 'times' builtin command now also
supports millisecond precision.
Fix a fork bomb when vi is run from a script and sent Ctrl-Z (#73) This bug was reported on the old mailing list: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg00207.html A fork bomb can occur when SIGTSTP is sent to the vi editor. Vi must be launched from a script run with exec (tested with BusyBox vi, nvi and vim): $ cat /tmp/foo vi /tmp/bar echo end $ ksh $ chmod +x /tmp/foo $ exec /tmp/foo While in vi, send SIGTSTP using Ctrl-Z src/cmd/ksh93/sh/fault.c: - Only fork after Ctrl-Z if job control is available. The patch used checks 'job.jobcontrol' instead of 'SH_MONITOR': https://git.centos.org/rpms/ksh/blob/c8/f/SOURCES/ksh-20120801-forkbomb.patch
2020-07-13 18:10:23 +00:00
2020-07-13:
- Fixed a fork bomb that could occur when the vi editor was sent SIGTSTP
while running in a ksh script.
Fix the max precision of the 'time' keyword (#72) This commit backports the required fixes from ksh2020 for using millisecond precision with the 'time' keyword. The bugfix refactors a decent amount of code to rely on the BSD 'timeradd' and 'timersub' macros for calculating the total amount of time elapsed (as these aren't standard, they are selectively implemented in an iffe feature test for platforms without them). getrusage(3) is now preferred since it usually has higher precision than times(3) (the latter is used as a fallback). There are three other fixes as well: src/lib/libast/features/time: - Test for getrusage with an iffe feature test rather than assume _sys_times == _lib_getrusage. src/cmd/ksh93/sh/xec.c: - A single percent at the end of a format specifier is now treated as a literal '%' (like in Bash). - Zero-pad seconds if seconds < 10. This was already done for the times builtin in commit 5c677a4c, although it wasn't applied to the time keyword. - Backport the ksh2020 bugfix for the time keyword by using timeradd and timersub with gettimeofday (which is used with a timeofday macro). Prefer getrusage when it is available. - Allow compiling without the 'timeofday' ifdef for better portability. This is the order of priority for getting the elapsed time: 1) getrusage (most precise) 2) times + gettimeofday (best fallback) 3) only times (doesn't support millisecond precision) This was tested by using debug '#undef' statements in xec.c. src/cmd/ksh93/features/time: - Implement feature tests for the 'timeradd' and 'timersub' macros. - Do a feature test for getrusage like in the libast time test. src/cmd/ksh93/tests/basic.sh: - Add test for millisecond precision. - Add test for handling of '%' at the end of a format specifier. - Add test for locale-specific radix point.
2020-07-14 21:48:04 +00:00
- Appending a lone percent to the end of a format specifier no longer
causes a syntax error. The extra percent will be treated as a literal
'%', like in Bash and zsh.
- The 'time' keyword now has proper support for millisecond precision.
Although this feature was previously documented, the 'time' keyword
only supported up to centisecond precision, which caused a command
like the one below to return '0.000' on certain operating systems:
$ TIMEFORMAT='%3R'; time sleep .003
- The 'time' keyword now zero-pads seconds less than ten (like mksh).
Fix type names starting with lowercase 'a' (#69) Type names that start with a lowercase 'a' cause an error when used: $ typeset -T al=(typeset bar) $ al foo=(bar=testset) /usr/bin/ksh: al: : invalid variable name The error occurs because when the parser checks for the alias builtin (to set 'assignment' to two instead of one), only the first letter of 'argp->argval' is checked (rather than the entire string). This was fixed in ksh93v- by comparing argp->argval against "alias", but in ksh93u+m the check can simply be removed because it is only run when a builtin has the BLT_DCL flag. As of 04b9171, the alias builtin does not have that flag. src/cmd/ksh93/sh/parse.c: - Remove the bugged check for the alias builtin. src/cmd/ksh93/tests/types.sh: - Add a regression test for type names starting with a lowercase 'a'.
2020-07-10 16:52:47 +00:00
2020-07-10:
- Fixed a bug that caused types created with 'typeset -T' to throw an error
when used if the type name started with a lowercase 'a'.
Fix possible crash due to failure to update shell FD state This applies ksh-20100621-fdstatus.patch from Red Hat. Not very much information is available, so this one is more or less taken on faith. But it seems to make sense on the face of it: calling sh_fcntl() instead of fcntl(2) directly makes the shell update its internal file descriptor state more frequently. It claims to fix Red Hat bug 924440. The report is currently closed to the public: https://bugzilla.redhat.com/show_bug.cgi?id=924440 However, Kamil Dudka at Red Hat writes: https://github.com/ksh93/ksh/issues/67#issuecomment-656379993 | Yes, the summary of RHBZ#924440 is "crash in bestreclaim() after | traversing a memory block with a very large size". We did not have | any in house reproducer for the bug. The mentioned patch was | provided and verified by a customer. ...and Marc Wilson dug up a Red Hat erratum containing this info: https://download.rhn.redhat.com/errata/RHBA-2013-1599.html | Previously, the ksh shell did not resize the file descriptor list | every time it was necessary. This could lead to memory corruption | when several file descriptors were used. As a consequence, ksh | terminated unexpectedly. This updated version resizes the file | descriptor list every time it is needed, and ksh no longer | crashes in the described scenario. (BZ#924440) No reproducer means no regression test can be added now. src/cmd/ksh93/sh/io.c, src/cmd/ksh93/sh/subshell.c, src/cmd/ksh93/sh/xec.c: - Change several fcntl(2) calls to sh_fcntl(). This function calls fcntl(2) and then updates the shell's file descriptor state.
2020-07-10 17:26:28 +00:00
- A potential crash due to memory corruption when using many file
descriptors has been fixed.
Fix crash due to double free() when sourcing multiple files There is a bug in sh_eval() that may cause ksh to crash due to a double free() after sourcing multiple files with '.' or 'source' if a longjmp is triggered, e.g. by a syntax error. This applies a fix from Siteshwar Vashist: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01943.html src/cmd/ksh93/sh/xec.c: sh_eval(): - Zero file descriptor io_save after closing it. This prevents a double free() after returning from a longjmp. src/cmd/ksh93/tests/basic.sh: - Add reproducer as regression test.
2020-07-09 14:09:52 +00:00
2020-07-09:
- Fixed a crash on syntax error when sourcing/dotting multiple files.
Fix crash when listing indexed arrays with 'typeset -a' There is a bug in print_scan() function that may cause ksh to crash while listing indexed arrays. The crash happens in nv_search() when called from print_scan(). This applies a fix from Siteshwar Vashist: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01944.html src/cmd/ksh93/bltins/typeset.c: - Call nv_scan() without the NV_IARRAY flag, even for a null scan. src/cmd/ksh93/tests/arrays.sh: - Add regression test for 'typeset -a' crash and check output.
2020-07-09 15:02:15 +00:00
- Fixed a crash when listing indexed arrays.
Fix hash table memory leak when restoring PATH There is a bug in path_alias() that may cause a memory leak when clearing the hash table while setting/restoring PATH. This applies a fix from Siteshwar Vashist: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01945.html Note that, contrary to Siteshwar's analysis linked above, this bug has nothing directly to do with subshells, forked or otherwise; it can also be reproduced by temporarily setting PATH for a command, for example, 'PATH=/dev/null true', and then doing a PATH search. Modified analysis: ksh maintains the value of PATH as a linked list. When a local scope for PATH is created (e.g. in a virtual subshell or when doing something like PATH=/foo/bar command ...), ksh duplicates PATH by increasing the refcount for every element in the linked list by calling the path_dup() and path_alias() functions. However, when the state of PATH is restored, this refcount is not decreased. Next time when PATH is reset to a new value, ksh calls the path_delete() function to delete the linked list that stored the older path. But the path_delete() function does not free elements whose refcount is greater than 1, causing a memory leak. src/cmd/ksh93/sh/path.c: path_alias(): - Decrease refcount and free old item if needed. (The 'old' variable was already introduced in 99065353, but its value was never used there; this fixes that as well.) src/cmd/ksh93/tests/leaks.sh: - Add regression test. With the bug, setting/restoring PATH (which clears the hash table) and doing a PATH search 16 times causes about 1.5 KiB of memory to be leaked.
2020-07-09 17:34:15 +00:00
- Fixed a memory leak when restoring PATH when temporarily setting PATH
for a command (e.g. PATH=/foo/bar command ...) or in a virtual subshell.
Fix a syntax error when ((...)) is combined with redirections (#68) This bugfix was backported from ksh93v- 2013-10-10-alpha. src/cmd/ksh93/sh/parse: item(): - The done label is placed after the 'inout' call for handling I/O redirections. This causes the command below to produce a syntax error because the '>' is not handled as a redirection operator after 'goto done': $ ((1+2)) > /dev/null /usr/bin/ksh: syntax error: `>' unexpected Moving the done label fixes the syntax error as 'inout' is now called to handle the redirection operator. src/cmd/ksh93/tests/arith.sh: - Add a simple regression test.
2020-07-09 21:12:04 +00:00
- Combining ((...)) with redirections no longer causes a syntax error
due to the parser handling '>' incorrectly.
Fix spurious warning output in KIA (-R) database file The ksh -R option creates a cross-reference database that can be parsed with a "C Query Language" (CQL) tool. See cql-1994.pdf at: http://gsf.cococlyde.org/files The -R option puts ksh in noexec mode as it parses the script, and this can produce warnings as the syntax is parsed. The bug is that these warnings can end up in the database file, corrupting it. This applies a fix from Paulo Andrade, via Siteshwar Vashisht: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01952.html src/cmd/ksh93/sh/parse.c: - Terminate names with a zero character when writing database output. A regression test is not very feasible because the majority of the database output consists of cryptic IDs/hashes that vary depending on the session and/or system and possibly other things.
2020-07-09 21:59:45 +00:00
- Fixed a bug that corrupted KIA/CQL cross-reference databases created using
ksh's -R option; shell warnings were wrongly included in the database file.
Fix UTF-8 shellquoting for xtrace, printf %q, etc. This fixes an annoying issue in the shell's quoting algorithm (used for xtrace (set -x), printf %q, and other things) for UTF-8 locales, that caused it to encode perfectly printable UTF-8 characters unnecessarily and inconsistently. For example: $ (set -x; : 'aeu aéu') + : $'aeu a\u[e9]u' $ (set -x; : 'aéu aeu') + : 'aéu aeu' $ (set -x; : '正常終了 aeu') + : '正常終了 aeu' $ (set -x; : 'aeu 正常終了') + : $'aeu \u[6b63]\u[5e38]\u[7d42]\u[4e86]' This issue was originally reported by lijo george in May 2017: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01958.html src/cmd/ksh93/sh/string.c: - Add is_invisible() function that returns true if a character is a Unicode invisible (non-graph) character, excluding ASCII space. Ref.: https://unicode.org/charts/PDF/U2000.pdf - Use a fallback in is_invisible() if we cannot use the system's iswprint(3); this is the case for the ksh C.UTF-8 locale if the OS doesn't support that. Fall back to a hardcoded blacklist of invisible and control characters and put up with not encoding nonexistent characters into \u[xxxx] escapes. Ref.: https://unicode.org/charts/PDF/U2000.pdf - When deciding whether to switch to $'...' quoting mode (state=2), use is_invisible() instead of testing for ASCII 0-127 range. - In $'...' quoting mode, use is_invisible() to decide whether to encode wide characters into \u[xxxx] escapes. src/cmd/ksh93/tests/builtins.sh: - Add regression tests for shellquoting Arabic, Japanese and Latin UTF-8 characters, to be run only in a UTF-8 locale. The Arabic sample text[*] contains a couple of direction markers that are expected to be encoded into \u[xxxx] escapes. [*] source: https://r12a.github.io/scripts/tutorial/summaries/arabic
2020-07-10 00:38:13 +00:00
- The shell's quoting algorithm (used in xtrace, printf %q, and more) has been
fixed for UTF-8 (Unicode) locales; it no longer needlessly and inconsistently
encodes normal printable UTF-8 characters into hexadecimal \u[xxxx] codes.
Fix unexpected output from 'printf %T' with certain formats (#65) This commit changes the behavior of four date formats accepted by 'printf %()T' because the old behavior is not compatible with modern implementations of date(1): - %k and %l now return a blank-padded hour, the former based on a 24-hour clock and the latter a 12-hour clock (these are common extensions present on Linux and *BSD). - %f now returns a date with the format '%Y.%m.%d-%H:%M:%S' (BusyBox extension). - %q now returns the quarter of the current year (GNU extension). src/cmd/ksh93/data/builtins.c: - Copy the date format documentation from date in libcmd to the printf man page (for documenting 'printf %T'). src/cmd/ksh93/tests/builtins.sh: - Add four regression tests for the changed date formats. src/cmd/ksh93/sh.1: - Remove inaccurate information about the date formats accepted by printf %T'. The KornShell uses a custom version of strftime(3) that isn't guaranteed to accepts the same formats as the native strftime function. src/lib/libast/tm/tmxfmt.c: - Change the behavior of %f, %k, %l and %q to the common behavior. %k and %l are implemented as aliases to %_H and %_I to avoid duplicating code. src/lib/libcmd/date.c: - Update the documentation for the AST date command since it is also affected by the changes to 'printf %T'. Fixes #62
2020-07-09 04:08:28 +00:00
2020-07-07:
- Four of the date formats accepted by 'printf %()T' have had their
functionality altered to the common behavior of date(1):
- '%k' and '%l' print the current hour with blank padding, the former
based on a 24-hour clock and the latter a twelve hour clock. These
are common extensions present on Linux and *BSD.
- '%f' prints a date with the format string '%Y.%m.%d-%H:%M:%S' (BusyBox).
- '%q' prints the quarter of the year (GNU).
Fix use of strdup on a NULL pointer (#63) The following set of commands can rarely cause a memory fault when auditing[*] is enabled, although most of the time it will simply cause ksh to write '(null)' to the auditing file in place of a tty name: $ [ -e /etc/ksh_audit ] || echo "/tmp/ksh_auditfile;$(id -u)" | sudo tee /etc/ksh_audit; $ v=$(ksh 2> /dev/null +o rc -ic $'getopts a:bc: opt --man\nprint $?') $ cat /tmp/ksh_auditfile 1000;1593599493;(null); getopts a:bc: opt --man This happens because strdup is used unconditionally on the pointer returned by 'ttyname', which can be NULL if stderr is closed. This then causes 'hp->tty' to be set to null, as strdup returns NULL. See https://github.com/att/ast/issues/1028 src/cmd/ksh93/edit/history.c: - Make strdup duplicate 'notty' instead of NULL to prevent crashes. [*] https://blog.fpmurphy.com/2008/12/ksh93-auditing-and-accounting.html
2020-07-06 20:51:44 +00:00
2020-07-06:
- 'notty' is now written to the ksh auditing file instead of '(null)' if
the user's tty could not be determined.
Fix memory leak on unset of associative array (#64) Associative arrays weren't being properly freed from memory, which was causing a memory leak. This commit incorporates a patch and reproducer/regress test from: https://www.mail-archive.com/ast-users@lists.research.att.com/msg01016.html src/cmd/ksh93/sh/name.c: - Properly free associative arrays from memory in nv_delete(). src/cmd/ksh93/tests/leaks.sh: - Add regression test.
2020-07-09 00:09:40 +00:00
- Unsetting an associative array no longer causes a memory leak to occur.
Fix corrupt UTF-8 char processing & shellquoting after aborted read If the processing of a multibyte character was interrupted in UTF-8 locales, e.g. by reading just one byte of a two-byte character 'ü' (\303\274) with a command like: print -nr $'\303\274' | read -n1 g then the shellquoting algorithm was corrupted in such a way that the final quote in simple single-quoted string was missing. This bug may have had other, as yet undiscovered, effects as well. The problem was with corrupted multibyte character processing and not with the shell-quoting routine sh_fmtq() itself. Full trace and discussion at: https://github.com/ksh93/ksh/issues/5 (which is also an attempt to begin to understand the esoteric workings of the libast mb* macros that process UTF-8 characters). src/lib/libast/comp/setlocale.c: utf8_mbtowc(): - If called from the mbinit() macro (i.e. if both pointer parameters are null), reset the global multibyte character synchronisation state variable. This fixes the problem with interrupted processing leaving an inconsistent state, provided that mbinit() is called before processing multibyte characters (which it is, in most (?) places that do this). Before this fix, calling mbinit() in UTF-8 locales was a no-op. src/cmd/ksh93/sh/string.c: sh_fmtq(): - Call mbinit() before potentially processing multibyte characters. Testing suggests that this could be superfluous, but at worst, it's harmless; better be sure. src/cmd/ksh93/tests/builtins.sh: - Add regression test for shellquoting with 'printf %q' after interrupting the processing of a multibyte characeter with 'read -n1'. This test only fails in a UTF-8 locale, e.g. when running: bin/shtests -u builtins SHELL=/buggy/ksh-2012-08-01 Fixes #5.
2020-07-05 17:24:41 +00:00
2020-07-05:
- In UTF-8 locales, fix corruption of the shell's internal string quoting
algorithm (as used by xtrace, 'printf %q', and more) that occurred when
the processing of a multibyte character was interrupted.
Fix bugs with backslash escaping in interactive vi mode (#57) This commit fixes the following bugs in the 'vi' editing mode backslash escape feature. Ref.: Bolsky & Korn (1995), p. 113, which states for \: "Similar to Control+V [...] except that it escapes only the next Erase or Kill charactrer". 1. The vi mode now only escapes the next character if the last character input was a backslash, fixing the bug demonstrated at: https://asciinema.org/a/E3Rq3et07MMQG5BaF7vkXQTg0 2. Escaping backslashes are now disabled in vi.c if the vi mode is disabled (note that vi.c handles raw editing mode in UTF-8 locales). This makes the behavior of the raw editing mode consistent in C/POSIX and UTF-8 locales. 3. An odd interaction with Backspace when the character prior to a separate buffer entered with Shift-C was a backslash has been fixed. Demonstration at: https://asciinema.org/a/314833 ^? will no longer be output repeatedly when attempting to erase a separate buffer with a Backspace, although, to be consistent with vi(1), you still cannot backspace past it before escaping out of it. Ref.: https://github.com/ksh93/ksh/issues/56#issuecomment-653586994 src/cmd/ksh93/edit/vi.c: - Prevent a backslash from escaping the next input if the previous input wasn't a backslash. This is done by unsetting a variable named backslash if a backslash escaped a character. backslash is set to the result of c == '\\' when the user enters a new character. - Disable escaping backslashes in the raw editing mode because it should not be enabled there. src/cmd/ksh93/tests/pty.sh: - Add some tests for how ksh handles backslashes in each editing mode to test for the bugs fixed by this commit. Fixes #56.
2020-07-03 19:08:00 +00:00
2020-07-03:
- Backslashes are no longer escaped in the raw Bourne Shell-like editing
mode in multibyte locales, i.e. backslashes are no longer treated like
Control-V if the emacs and vi modes are disabled.
- Deleting a backslash in vi mode with Control-H or Backspace now only
escapes a backslash if it was the previous input. This means erasing a
string such as 'ab\\\' will only cause the first backslash to escape a
Backspace as '^?', like in emacs mode.
- An odd interaction with Backspace when the last character of a separate
buffer created with Shift-C was '\' has been fixed. '^?' will no longer
be output repeatedly when attempting to erase a separate buffer with
a Backspace. Note that buffers created with Shift-C are not meant to be
erasable:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/vi.html#tag_20_152_13_49
Fix 'kill -INFO' on systems that support SIGINFO (#59) src/cmd/ksh93/data/signals.c: - SIGINFO was absent from the table of signals, which caused commands like 'kill -INFO $$' to fail even on platforms with SIGINFO (such as macOS and FreeBSD). Fix that by adding it to the signal table. src/cmd/ksh93/tests/signal.sh: - Add a regression tests for using SIGINFO with the kill builtin. The test will only be run if the external kill command supports SIGINFO.
2020-07-04 14:57:47 +00:00
- The 'kill' builtin now supports the SIGINFO signal (on operating systems
with support for SIGINFO).
Fix segfault if $PATH contains a .paths directory (#55) ksh crashed if it encountered a .paths directory in any of the directories in $PATH. Ref: https://bugs.launchpad.net/ubuntu/+source/ksh/+bug/1534855 src/cmd/ksh93/sh/path.c: path_chkpaths(): - Refuse to read .paths if it's not a regular file or a symlink to a regular file.
2020-07-02 22:29:07 +00:00
2020-07-02:
- Fixed a crash that occurred if a directory named '.paths' existed in any
directory listed in $PATH. The fix was to only read '.paths' if it is a
regular file or a symlink to a regular file.
Fix a crash when 'read -u' is given an invalid fd (#53) File descriptors that are too far out of range will cause the read builtin to crash. The following example will generate two crashes: $ ksh -c 'read -u 2000000' || ksh -c 'read -u-2000000' The fix is to error out when the given file descriptor is out of range. This bugfix is from Tomas Klacko, although it was modified to use 'sh_iovalidfd' and reject numbers greater than 'INT_MAX': https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01912.html The question about 'shp->fdstatus[-1]' only applies to ksh93v- (ksh93u+ doesn't have any references to 'shp->fdstatus[-1]'). src/cmd/ksh93/bltins/read.c: - File descriptors that are out of range should be rejected with an error message (like invalid file descriptors that are in range). The seemingly redundant check for negative numbers is there because out of range negative numbers also cause memory faults despite the later 'fd<0' check. src/cmd/ksh93/tests/io.sh: - Add three tests for attempting 'read -u' on various invalid file descriptor numbers.
2020-07-01 17:14:10 +00:00
2020-06-30:
- 'read -u' will no longer crash with a memory fault when given an out of
range or negative file descriptor.
Fix ERE repetition expressions in [[ ... =~ ERE{x,y} ]] (#54) Regular expressions that combine a repetition expression with a parenthesized sub-expression throw a garbled syntax error: $ [[ AATAAT =~ (AAT){2} ]] ksh: syntax error: `~(E)(AAT){2} ]] :'%Cred%h%Creseksh: syntax error: `~(E)(AAT){2} ]] :'%Cred%h%Creseksh: syntax' unexpected The syntax error occurs because ksh is not fully accounting for '=~' when it runs into a curly bracket. This fix disables the syntax error when the operator is '=~' and adds handling for '(str){x}' (to allow for more than one sub-expression). This bugfix and the regression tests for it were backported from ksh93v- 2014-12-24-beta. src/cmd/ksh93/sh/lex.c: - Do not trigger a syntax error for '{x}' when the operator is '=~' and add handling for multiple parentheses when combined with '{x}'. src/cmd/ksh93/tests/bracket.sh: - Add two tests from ksh93v- to test sub-expressions combined with the '{x}' quantifier.
2020-07-02 17:40:15 +00:00
- The '=~' operator no longer raises an error if a regular expression
combines the '{x}' quantifier with a sub-expression.
Fix BUG_CASELIT: pattern matching as literal string in 'case' This fixes an undocumented 'case' pattern matching misbehaviour (labelled BUG_CASELIT in modernish) that goes back to the original Bourne shell, but wasn't discovered until 2018. If a pattern doesn't match as a pattern, it's tried again as a literal string. This breaks common validation use cases, such as: n='[0-9]' case $n in ( [0-9] ) echo "$n is a number" ;; esac would output "[0-9] is a number" as the literal string fallback matches the pattern. As this misbehaviour was never documented anywhere (not for Bourne, ksh88, or ksh93), and it was never replicated in other shells (not even in ksh88 clones pdksh and mksh), it is unlikely any scripts rely on it. Of course, a literal string fallback, should it be needed, is trivial to implement correctly without this breakage: case $n in ( [0-9] | "[0-9]") echo "$n is a number or the number pattern" ;; esac src/cmd/ksh93/sh/xec.c: - Remove trim_eq() function responsible for implementing the misbehaviour described above. NEWS: - Added. Document this bugfix. Ref.: - The problem: thread starting at https://www.mail-archive.com/austin-group-l@opengroup.org/msg02127.html - The solution, thanks to George Koehler: comments/commits in https://github.com/att/ast/issues/476 - Modernish BUG_CASELIT bug test & documentation: https://github.com/modernish/modernish/commit/b2024ae3 (cherry picked from commit 8d6c8ce69884767a160c1e20049e77bdd849c248 with some extra edits to NEWS to upate the info for this reboot)
2020-06-11 15:14:31 +00:00
Fix crashes caused by 'typeset -RF' (#47) Variables created with 'typeset -RF' were being treated as short integers, even though they are actually floating point values. As a result the following example will cause a crash: $ typeset -RF foo=1 $ test "$foo" This is fixed by checking for 'NV_DOUBLE' with 'nv_isattr', which prevents ksh from treating floating point values as short integers due to '== NV_INT16P' excluding 'NV_DOUBLE'. This bugfix was backported from ksh93v- 2013-10-10-alpha. src/cmd/ksh93/sh/array.c, src/cmd/ksh93/sh/name.c, src/cmd/ksh93/sh/nvdisc: - Avoid treating floating point values as short integers by checking for 'NV_DOUBLE' with 'nv_isattr'. src/cmd/ksh93/tests/types.sh: - Add a regression test for the 'typeset -RF' crash. The crash cannot be replicated if 'typeset -RF' sets 'foo' to zero.
2020-06-28 22:30:27 +00:00
2020-06-28:
- Variables created with 'typeset -RF' no longer cause a memory fault
when accessed.
Fix memory corruption when a compound variable is unset (#49) The following set of commands ends with a memory fault under certain circumstances because ksh attempts to free memory twice, causing memory corruption: $ testarray=(1 2) $ compound testarray $ unset testarray $ eval testarray= The fix is to make sure 'np->nvfun' is a valid pointer before attempting to free memory in 'put_tree'. This patch is from OpenSUSE: https://build.opensuse.org/package/view_file/shells/ksh/ksh93-nvtree-free.dif?expand=1 src/cmd/ksh93/sh/nvtree.c: - Do not try to free memory when 'np->nvfun' and 'val' are false. src/cmd/ksh93/tests/comvar.sh: - Add a regression test for the double free problem. The reproducer must be run from an executable script with 'ksh -c'.
2020-06-29 17:08:28 +00:00
- Unsetting an array that was turned into a compound variable will no
longer cause silent memory corruption.
Fix the readonly builtin's scope in functions (#51) * Fix the readonly builtin's scope in functions This bug was first reported at https://github.com/att/ast/issues/881 'tdata.sh->prefix' is only set to the correct value when 'b_readonly' is called as 'export', which breaks 'readonly' in functions because the correct scope isn't set. As a result, the following example will only print a newline: $ function show_bar { readonly foo=bar; echo $foo; }; show_bar The fix is to move the required code out of the if statement for 'export', as it needs to be run for 'readonly' as well. This bugfix is from https://github.com/att/ast/pull/906 src/cmd/ksh93/bltins/typeset.c: - Set 'tdata.sh->prefix' to the correct value, otherwise 'readonly' uses the wrong scope. src/cmd/ksh93/tests/builtins.sh: - Add the regression test from ksh2020, modified to run in a subshell. src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/sh.1: - Add documentation of 'readonly' vs. 'typeset -r' difference: 'readonly' does not create a function-local scope. Co-authored-by: Martijn Dekker <martijn@inlv.org>
2020-06-29 18:09:20 +00:00
- Variables created with 'readonly' in functions are now set to the
specified value instead of nothing. Note that 'readonly' does not
create a function-local scope, unlike 'typeset -r' which does.
Fix incorrect behavior of 'cd ../.foo' (#46) The cd builtin was removing '.' from directory names when combined with a preceding '../', which caused commands like 'cd ../.local' to become 'cd ../local'. This patch fixes the problem by limiting the extra handling to leading '..'. The bugfix comes from ksh93v- 2013-10-10-alpha, although this version is a shortened patch from Solaris (as ksh93v- refactored a decent amount of the code for the cd builtin). src/cmd/ksh93/bltins/cd_pwd.c: - cd should only check for leading '..', as trying to handle a lone '.' only causes problems. src/cmd/ksh93/tests/builtins.sh: - Add a regression test for this problem based on the test present in ksh93v- 2013-10-10-alpha. Patch from Solaris: https://github.com/oracle/solaris-userland/blob/860d27f/components/ksh93/patches/270-23319761.patch
2020-06-26 22:36:29 +00:00
2020-06-26:
- Changing to a directory that has a name starting with a '.' will no
longer fail if preceded by '../' (i.e. 'cd ../.local' will now work).
Fix buggy completion of ~/some in vi mode (#41) This commit fixes the bug reported in: https://github.com/att/ast/issues/682 The following sequence fails in vi mode because ksh looks in the wrong part of the 'virtual' buffer: $ touch ~/testfile $ ls ~/test<tab> The fix is to change 'virtual[i]' to 'virtual[last_virt]' in the bugged section of code. The other changes are to make sure listing files in a directory with something like 'ls /etc/<tab>' calls the code for Ctrl+L to preserve 'ls /etc/' rather than try (and fail) to complete the directory name, producing 'ls /etc\n/'. This bugfix was backported from ksh93v- 2013-10-10-alpha. src/cmd/ksh93/edit/vi.c - Backport the bugfix from ksh93v- 2013-10-10-alpha for this problem. src/cmd/ksh93/tests/pty.sh - Add a regression test for this issue using pty, adjusted slightly for a fake home directory in /tmp.
2020-06-25 21:08:43 +00:00
2020-06-24:
- Fixed buggy tab completion of tilde-expanded paths such as
~/some in 'vi' mode.
Fix behaviour of tabs in raw Bourne Shell-like editing mode When neither '-o emacs' nor '-o vi' is active, there were a couple of bugs with entering tab characters: 1. Tab completion was erroneously left active. The cause of this was that raw Bourne edit mode is handled by ed_viread() in vi.c on shells with wide character support, instead of the default ed_read() in edit.c, and the former failed to check if vi mode is active when processing tab characters. 2. When entering literal tab characters, the cursor was moved to the right only one character, instead of the amount of characters corresponding to the tab. src/cmd/ksh93/edit/vi.c: getline(): - Before processing '\t' (tab) for command completion, check that the 'vi' shell option (SH_VI) is active. src/cmd/ksh93/edit/edit.c: ed_virt_to_phys(): - When translating literal tabs to on-terminal spaces and when recalculating the cursor position, remove erroneous checks for SH_VI; this is also needed in raw Bourne mode. According to my own testing, this has no effect on emacs mode (knock on wood). src/cmd/ksh93/tests/pty.sh: - Add two regression tests. An odd race condition reveals itself in either pty or in ksh's raw/Bourne edit mode; see comment in test. Effect is we have to expect either literal tabs or tabs expanded to spaces, until that is tracked down and fixed. Fixes #43.
2020-06-26 05:19:58 +00:00
- In the raw/default Bourne Shell-like editing mode that occurs when neither
the 'emacs' nor the 'vi' shell option is active:
* tab completion is now correctly disabled, instead of enabled and broken;
* entering tab characters now moves the cursor the correct amount.
Fix process substitution combined with redirection (#40) The code for handling process substitution with redirection was never being run because IORAW is usually set when IOPROCSUB is set. This commit fixes the problem by moving the required code out of the !IORAW if statement. The following command now prints 'good' instead of writing 'ok' to a bizzare file: $ ksh -c 'echo ok > >(sed s/ok/good/); wait' good This commit also fixes a bug that caused the process ID of the asynchronous process to print when the shell was in interactive mode. The following command no longer prints a process ID, behaving like in Bash and zsh: $ echo >(true) /dev/fd/5 src/cmd/ksh93/sh/args.c: - Temporarily turn off the interactive state while in a process substitution to prevent the shell from printing the PID of the asynchronous process. src/cmd/ksh93/sh/io.c: - Move the code for process substitution with redirection into a separate if statement. src/cmd/ksh93/tests/io.sh: - Add two tests for both process substitution bugs fixed by this commit. src/cmd/ksh93/tests/shtests: - Update shtests with a patch from Martijn Dekker to use pretty-printing for the output from the times builtin (if it is available). Fixes #2
2020-06-23 22:02:16 +00:00
2020-06-23:
- Fixed a bug that caused combining process substitution with redirection
to create a bizarre file in the user's current working directory.
- Using process substitution while the shell is interactive no longer
causes the process ID of the asynchronous process to be printed.
Make 'stop' and 'suspend' regular built-ins The 'stop' and 'suspend' default aliases are now converted into regular built-in commands so that 'unalias -a' does not remove them, 'suspend' can do some sanity checks, and something like cmd=stop; $cmd $! will now work. src/cmd/ksh93/bltins/trap.c: - b_kill(): Incorporate 'stop' functionality, which is simply setting the same flag and variable as '-s STOP' would have done. - b_suspend(): Add simple builtin function that sends SIGSTOP to the main shell. Check for no operands, and refuse to suspend a login shell (which would leave the user stuck with no way out). Also check that 'kill' succeeds; if we're in an asynchronous subshell, it is possible the main shell no longer exists. src/cmd/ksh93/data/aliases.c: - Remove "stop" and "suspend" default aliases. (Why were these conditional upon SIGTSTP when they actually issued SIGSTOP?) src/cmd/ksh93/include/builtins.h, src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/data/msg.c: - Add declarations of "stop" and "suspend" regular built-ins. - Add option strings (AST manual/--man pages) for them. - Add e_toomanyops ("too many operands") reusable error message for b_suspend(). Other new commands may want this at some point. src/cmd/ksh93/sh.1: - Remove "stop" and "suspend" default aliases. - Document "stop" and "suspend" regular built-in commands.
2020-06-22 12:59:24 +00:00
2020-06-22:
- The 'stop' and 'suspend' default aliases have been converted into regular
built-in commands, so that 'unalias -a' does not remove them, 'suspend'
can do a couple of sanity checks, and something like
cmd=stop; $cmd $!
will now work. See 'stop --man' and 'suspend --man' for more information.
Fix a crash when 'kill %%' and 'kill %+' are run (#35) Ksh was trying to use the 'pw' variable as a valid pointer even when it was NULL. This is fixed by doing the error check for 'pw' before doing anything else in 'job_kill'. This bugfix is from Red Hat: https://git.centos.org/rpms/ksh/blob/44e0a643a93492b1f6beebbf6ffcfd453d9ab8f2/f/SOURCES/ksh-20130214-fixkill.patch Fixes #34
2020-06-22 17:11:19 +00:00
- Fixed a bug that caused the kill and stop commands to segfault when given
a non-existent job.
Fix nested functions ignoring prefixed variable assignments (#37) This commit fixes the bug described in att/ast#32. The fix and following explanation is from att/ast#467: While copying variables from function's local scope to a new scope, variable attributes were not copied. Such variables were not marked to be exported in the new function. For e.g. function f2 { env | grep -i "^foo"; } function f1 { env | grep -i "^foo"; f2; } foo=bar f1 prints 'foo=bar' only once, but it should print be twice. src/cmd/ksh93/sh/xec.c: - When variables from the local scope of a function are copied into the scope of a nested function, the attributes of the variables need to be copied as well. src/cmd/ksh93/tests/functions.sh: - Add regression tests from ksh2020 to check environment variables passed to functions.
2020-06-22 23:27:05 +00:00
- Nested functions no longer ignore variable assignments that were prefixed
to their parent function, i.e. 'VAR=foo func' will now set $VAR to 'foo'
in the scope of any nested function 'func' runs.
Fix unreliable behavior when special vars are readonly or unset (#27) src/cmd/ksh93/data/variables.c: - Running 'unset .sh.lineno' creates a memory fault, so fix that by giving it the NV_NOFREE attribute. This crash was happening because ${.sh.lineno} is an integer that cannot be freed from memory with free(3). src/cmd/ksh93/sh/init.c: - Tell _nv_unset to ignore NV_RDONLY when $RANDOM and $LINENO are restored from the subshell scope. This is required to fully restore the original state of these variables after a virtual subshell finishes. src/cmd/ksh93/bltins/typeset.c, src/cmd/ksh93/sh/subshell.c: - Disabled some optimizations for two instances of 'sh_assignok' to fix 'readonly' in virtual subshells and '(unset .sh.level)' in nested functions. This fixes the following variables when '(readonly $varname); enum varname=' is run: $_ ${.sh.name} ${.sh.subscript} ${.sh.level} The optimization in question prevents sh_assignok from saving the original state of these variables by making the sh_assignok call a no-op. Ksh needs the original state of a variable for it to be properly restored after a virtual subshell has run, otherwise ksh will simply carry over any new flags (being NV_RDONLY in this case) from the subshell into the main shell. src/cmd/ksh93/tests/variables.sh: - Add regression tests from Martijn Dekker for setting special variables as readonly in virtual subshells and for unsetting special variables in general. Fixes #4
2020-06-20 17:08:41 +00:00
2020-06-20:
- Fixed a bug that caused setting the following variables as readonly in
a virtual subshell to affect the environment outside of the subshell:
$_
${.sh.name}
${.sh.subscript}
${.sh.level}
$RANDOM
$LINENO
Fix various minor problems and update the documentation (#237) These are minor fixes I've accumulated over time. The following changes are somewhat notable: - Added a missing entry for 'typeset -s' to the man page. - Add strftime(3) to the 'see also' section. This and the date(1) addition are meant to add onto the documentation for 'printf %T'. - Removed the man page the entry for ksh reading $PWD/.profile on login. That feature was removed in commit aa7713c2. - Added date(1) to the 'see also' section of the man page. - Note that the 'hash' command can be used instead of 'alias -t' to workaround one of the caveats listed in the man page. - Use an 'out of memory' error message rather than 'out of space' when memory allocation fails. - Replaced backticks with quotes in some places for consistency. - Added missing documentation for the %P date format. - Added missing documentation for the printf %Q and %p formats (backported from ksh2020: https://github.com/att/ast/pull/1032). - The comments that show each builtin's options have been updated.
2021-03-21 14:39:03 +00:00
- Fixed two bugs that caused 'unset .sh.lineno' to always produce a memory
fault and '(unset .sh.level)' to memory fault when run in nested
Fix unreliable behavior when special vars are readonly or unset (#27) src/cmd/ksh93/data/variables.c: - Running 'unset .sh.lineno' creates a memory fault, so fix that by giving it the NV_NOFREE attribute. This crash was happening because ${.sh.lineno} is an integer that cannot be freed from memory with free(3). src/cmd/ksh93/sh/init.c: - Tell _nv_unset to ignore NV_RDONLY when $RANDOM and $LINENO are restored from the subshell scope. This is required to fully restore the original state of these variables after a virtual subshell finishes. src/cmd/ksh93/bltins/typeset.c, src/cmd/ksh93/sh/subshell.c: - Disabled some optimizations for two instances of 'sh_assignok' to fix 'readonly' in virtual subshells and '(unset .sh.level)' in nested functions. This fixes the following variables when '(readonly $varname); enum varname=' is run: $_ ${.sh.name} ${.sh.subscript} ${.sh.level} The optimization in question prevents sh_assignok from saving the original state of these variables by making the sh_assignok call a no-op. Ksh needs the original state of a variable for it to be properly restored after a virtual subshell has run, otherwise ksh will simply carry over any new flags (being NV_RDONLY in this case) from the subshell into the main shell. src/cmd/ksh93/tests/variables.sh: - Add regression tests from Martijn Dekker for setting special variables as readonly in virtual subshells and for unsetting special variables in general. Fixes #4
2020-06-20 17:08:41 +00:00
functions.
Fix 'whence -a' to print correct path for tracked alias (#25) 'whence -a' bases the path for tracked aliases on the user's current working directory if an enabled ksh builtin of the same name is also available. The following example will claim 'cat' is in the user's current working directory: $ whence -a cat cat is a tracked alias for /usr/bin/cat $ builtin cat $ whence -a cat cat is a shell builtin cat is /usr/bin/cat cat is a tracked alias for /current/working/directory/cat This patch from ksh2020 fixes this problem by properly saving the path of the tracked alias for use with 'whence -a', since 'path_pwd' (as implied by the function's name) only gets the users current working directory, not the location of tracked aliases. Ref.: https://github.com/att/ast/issues/1049 This bug was originally reported by David Morano about two decades ago to the AST team: https://github.com/att/ast/issues/954 src/cmd/ksh93/bltins/whence.c: - Print the actual path of a tracked alias, path_pwd doesn't have this functionality. src/cmd/ksh93/include/name.h: - Add 'pathcomp' for saving the value of tracked aliases. src/cmd/ksh93/sh/path.c: - Save the value of tracked aliases for use by whence. src/cmd/ksh93/tests/builtins.sh: - Add a regression test for using 'whence -a' on tracked aliases with a builtin equivalent.
2020-06-19 12:01:55 +00:00
2020-06-18:
- A two decade old bug that caused 'whence -a' to base the path of
tracked aliases on the user's current working directory has been
Fix some errors in the documentation and other minor issues (#42) Somewhat notable changes in this commit: - The 'set +r' bugfix (re: 74b41621) is now documented in the changelog. - Missing options have been added to the synopsis section of the ksh man page. - The minor formatting fix from https://github.com/ksh-community/ksh/pull/5 has been applied to the ksh man page. - A few fixes from https://github.com/att/ast/commit/5e747cfb have been applied to the ksh man page. - The man page fixes from https://github.com/att/ast/pull/353 have been applied, being: - An addition to document the behavior of 'set -H'. - A fix for the cd section appending rksh93. - A fix for some options being indented too far. - Removal of a duplicate section documenting '-D'. - Reordering the options for 'set' in alphabetical order. - A minor fix for the documentation of 'ksh -i'.
2020-06-25 15:21:40 +00:00
fixed. Now the real path to tracked aliases is shown when '-a' is
passed to the whence command.
Fix 'whence -a' to print correct path for tracked alias (#25) 'whence -a' bases the path for tracked aliases on the user's current working directory if an enabled ksh builtin of the same name is also available. The following example will claim 'cat' is in the user's current working directory: $ whence -a cat cat is a tracked alias for /usr/bin/cat $ builtin cat $ whence -a cat cat is a shell builtin cat is /usr/bin/cat cat is a tracked alias for /current/working/directory/cat This patch from ksh2020 fixes this problem by properly saving the path of the tracked alias for use with 'whence -a', since 'path_pwd' (as implied by the function's name) only gets the users current working directory, not the location of tracked aliases. Ref.: https://github.com/att/ast/issues/1049 This bug was originally reported by David Morano about two decades ago to the AST team: https://github.com/att/ast/issues/954 src/cmd/ksh93/bltins/whence.c: - Print the actual path of a tracked alias, path_pwd doesn't have this functionality. src/cmd/ksh93/include/name.h: - Add 'pathcomp' for saving the value of tracked aliases. src/cmd/ksh93/sh/path.c: - Save the value of tracked aliases for use by whence. src/cmd/ksh93/tests/builtins.sh: - Add a regression test for using 'whence -a' on tracked aliases with a builtin equivalent.
2020-06-19 12:01:55 +00:00
Restore #22 'unset -f' fix minus segfault (re: b7932e87, 97511748) Applying the fix for 'unset -f' exposed a crashing bug in lookup() in sh/nvdisc.c, which is the function for looking up discipline functions. This is what caused tests/variables.sh to crash. Ref.: https://github.com/ksh93/ksh/issues/23#issuecomment-645699614 src/cmd/ksh93/sh/nvdisc.c: lookup(): - To avoid segfault, check that the function pointer nq->nvalue.rp is actually set before checking if nq->nvalue.rp->running==1. src/cmd/ksh93/sh/xec.c, src/cmd/ksh93/tests/functions.sh: - Uncomment the 'unset -f' fix from b7932e87. Resolves #21 (again).
2020-06-18 00:48:51 +00:00
2020-06-17:
- A bug in 'unset -f' was fixed that prevented shell functions from
unsetting themselves while they were running. A POSIX function no longer
crashes when doing so, and a KornShell-style function no longer silently
ignores an 'unset -f' on itself. A function of either form now continues
running after unsetting itself, and is removed at the end of the run.
`read -r -d` should not ignore `-r` This bug was previously reported in att/ast#37. Ksh ignores `-r` when `read -r -d` is run because when the bit for `D_FLAG` is set, the bit for `R_FLAG` is unset as a side effect of setting `D_FLAG`. The following set of commands fails to print a backslash: $ printf '\\\000' | read -r -d '' $ echo $REPLY The fix for this bug is to set `D_FLAG` with `D_FLAG + 1`, which prevents `R_FLAG` from being unset. This bugfix has been backported from ksh93v- 2013-10-10-alpha. src/cmd/ksh93/bltins/read.c: - Set `D_FLAG` with `D_FLAG + 1` to prevent the bit for `R_FLAG` from being unset. src/cmd/ksh93/tests/builtins.sh: - Add the regression test for `read -r -d` from ksh93v-.
2020-06-16 11:21:42 +00:00
2020-06-16:
- Passing the '-d' flag to the read builtin will no longer cause the '-r'
flag to be discarded when 'read -r -d' is run.
Fix assignments preceding 'command <special builtin>' (#19) Ksh was not checking for `command` when running a special builtin, which caused preceding invocation-local variable assignments to become global. This is the reproducer from the att/ast#72: $ foo=BUG command eval ':' $ echo "$foo" This no longer prints 'BUG', as ksh now makes sure the command builtin is not running a special builtin before making invocation-local variable assignments global. src/cmd/ksh93/sh/xec.c: - Backport the bugfix for BUG_CMDSPASGN from ksh93v- 2013-10-10-alpha. src/cmd/ksh93/tests/builtins.sh: - Add a regression test based on the reproducer in att/ast#72.
2020-06-16 21:58:05 +00:00
- Fix BUG_CMDSPASGN: preceding a "special builtin"[*] with 'command' now
prevents preceding invocation-local variable assignments from becoming global.
[*] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_14
Make 'source' a regular built-in The 'source' alias is now converted into a regular built-in command so that 'unalias -a' does not remove it, and something like cmd=source; $cmd name args will now work. This is part of the project to replace default aliases that define essential commands by proper builtins that act identically (except you now get the actual command's name in any error/usage messages). src/cmd/ksh93/data/aliases.c: - Remove 'source' default alias. src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/include/builtins.h: - Define 'source' regular builtin with extra parser ID "SYSSOURCE". Same definition as '.', minus the BLT_SPC flag indicating a special builtin. This preserves the behaviour of 'command .'. - Update sh_optdot[] to include info for 'source --man'. (Note that \f?\f expands to the current command name. This allows several commands to share a single --man page.) src/cmd/ksh93/sh/parse.c: - In the two places that SYSDOT is checked for, also check for SYSSOURCE, making sure the two commands are parsed identically. src/cmd/ksh93/sh.1: - Remove 'source' default alias. - Document 'source' regular builtin.
2020-06-15 07:03:44 +00:00
2020-06-15:
- The 'source' alias has been converted into a regular built-in command.
Remove a buggy optimization for variables in subshells This bug was originally reported by @lijog in att/ast#7 and has been reported again in #15. KSH does not save the state of a variable if it is in a newer scope. This is because of an optimization in sh_assignok first introduced in ksh93t+ 2010-05-24. Here is the code change in that version: return(np); /* don't bother to save if in newer scope */ - if(!(rp=shp->st.real_fun) || !(dp=rp->sdict)) - dp = sp->var; - if(np->nvenv && !nv_isattr(np,NV_MINIMAL|NV_EXPORT) && shp->last_root) - dp = shp->last_root; - if((mp=nv_search((char*)np,dp,HASH_BUCKET))!=np) - { - if(mp || !np->nvfun || np->nvfun->subshell>=sh.subshell) - return(np); - } + if(sp->var!=shp->var_tree && shp->last_root==shp->var_tree) + return(np); if((ap=nv_arrayptr(np)) && (mp=nv_opensub(np))) { This change was originally made to replace a buggier optimization. However, the current optimization causes variables set in subshells to wrongly affect the environment outside of the subshell, as the variable does not get set back to its original value. This patch simply removes the buggy optimization to fix this problem. src/cmd/ksh93/sh/subshell.c: - Remove a buggy optimization that caused variables set in subshells to affect the environment outside of the subshell. src/cmd/ksh93/tests/subshell.sh: - Add a regression test for setting variables in subshells. This test has to be run from the disk after being created with a here document because it always returns the expected result when run directly in the regression test script.
2020-06-15 13:43:37 +00:00
- Functions that set variables in a virtual subshell will no longer affect
variables of the same name outside of the virtual subshell's environment.
optget: display --help and --man in terse usage messages The fact that every ksh builtin command self-documents with the options --help and --man (and others, see 'getopts --man'; but these are the essential ones) is poorly known; the information is buried somewhere deep in the sh.1 manual page, and is incomplete at that. None of the terse usage messages displayed on error point the user to these options. So let's fix that. src/lib/libast/misc/optget.c: - Change generic 'options' placeholder, used in all terse usage messages, to 'options | --help | --man'. src/cmd/ksh93/sh.1: - Edit documentation of --man/-?, adding documentation on --help which was completely undocumented. Refer to 'getopts --man' for more advanced info. - Separate these from the (important) documentation on special builtins using a paragraph break.
2020-06-15 14:56:11 +00:00
- Terse usage messages written by builtin commands now point the user to
the --help and --man options for more information.
`read -S` now correctly handles nested double quotes Prior to this bugfix, the following set of commands would fail to print two double quotes: IFS=',' read -S a b c <<<'foo,"""title"" data",bar' echo $b This fix is from ksh93v- 2013-10-10-alpha, although it has been revised to use stakputc to put the required double quote into the buffer for consistency with the ksh93u+ codebase. src/cmd/ksh93/bltins/read.c: - When handling nested double quotes, put the required double quote in read's buffer with stakputc. src/cmd/ksh93/tests/builtins.sh: - Add the regression test for `read -S` from ksh93v-. src/cmd/ksh93/sh.1: - Fix a minor formatting error to highlight '-S' in the ksh(1) man page.
2020-06-14 16:28:22 +00:00
2020-06-14:
Make 'source' a regular built-in The 'source' alias is now converted into a regular built-in command so that 'unalias -a' does not remove it, and something like cmd=source; $cmd name args will now work. This is part of the project to replace default aliases that define essential commands by proper builtins that act identically (except you now get the actual command's name in any error/usage messages). src/cmd/ksh93/data/aliases.c: - Remove 'source' default alias. src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/include/builtins.h: - Define 'source' regular builtin with extra parser ID "SYSSOURCE". Same definition as '.', minus the BLT_SPC flag indicating a special builtin. This preserves the behaviour of 'command .'. - Update sh_optdot[] to include info for 'source --man'. (Note that \f?\f expands to the current command name. This allows several commands to share a single --man page.) src/cmd/ksh93/sh/parse.c: - In the two places that SYSDOT is checked for, also check for SYSSOURCE, making sure the two commands are parsed identically. src/cmd/ksh93/sh.1: - Remove 'source' default alias. - Document 'source' regular builtin.
2020-06-15 07:03:44 +00:00
`read -S` now correctly handles nested double quotes Prior to this bugfix, the following set of commands would fail to print two double quotes: IFS=',' read -S a b c <<<'foo,"""title"" data",bar' echo $b This fix is from ksh93v- 2013-10-10-alpha, although it has been revised to use stakputc to put the required double quote into the buffer for consistency with the ksh93u+ codebase. src/cmd/ksh93/bltins/read.c: - When handling nested double quotes, put the required double quote in read's buffer with stakputc. src/cmd/ksh93/tests/builtins.sh: - Add the regression test for `read -S` from ksh93v-. src/cmd/ksh93/sh.1: - Fix a minor formatting error to highlight '-S' in the ksh(1) man page.
2020-06-14 16:28:22 +00:00
- 'read -S' is now able to correctly handle strings with double quotes
nested inside of double quotes.
Backport the ksh2020 fix for timezone name determination This fix for `printf '%T' now` on FreeBSD was written by @krader1961. This is from https://github.com/att/ast/pull/591: On FreeBSD calling tzset() does not guarantee the tzname array will be correctly populated. On most systems that works but on FreeBSD you have to call localtime() or a related function (e.g., ctime()). This change also eliminates a potential, very small, memory leak due to the strdup()'ed tznames not being freed. src/lib/libast/tm/tminit.c: - Fix timezone name determination on FreeBSD and a memory leak.
2020-06-13 21:16:08 +00:00
2020-06-13:
NEWS: update repo URI
2020-06-14 04:28:38 +00:00
Backport the ksh2020 fix for timezone name determination This fix for `printf '%T' now` on FreeBSD was written by @krader1961. This is from https://github.com/att/ast/pull/591: On FreeBSD calling tzset() does not guarantee the tzname array will be correctly populated. On most systems that works but on FreeBSD you have to call localtime() or a related function (e.g., ctime()). This change also eliminates a potential, very small, memory leak due to the strdup()'ed tznames not being freed. src/lib/libast/tm/tminit.c: - Fix timezone name determination on FreeBSD and a memory leak.
2020-06-13 21:16:08 +00:00
- Fixed a timezone name determination bug on FreeBSD that caused the
Fix various minor problems and update the documentation (#237) These are minor fixes I've accumulated over time. The following changes are somewhat notable: - Added a missing entry for 'typeset -s' to the man page. - Add strftime(3) to the 'see also' section. This and the date(1) addition are meant to add onto the documentation for 'printf %T'. - Removed the man page the entry for ksh reading $PWD/.profile on login. That feature was removed in commit aa7713c2. - Added date(1) to the 'see also' section of the man page. - Note that the 'hash' command can be used instead of 'alias -t' to workaround one of the caveats listed in the man page. - Use an 'out of memory' error message rather than 'out of space' when memory allocation fails. - Replaced backticks with quotes in some places for consistency. - Added missing documentation for the %P date format. - Added missing documentation for the printf %Q and %p formats (backported from ksh2020: https://github.com/att/ast/pull/1032). - The comments that show each builtin's options have been updated.
2021-03-21 14:39:03 +00:00
output from "LC_ALL=C printf '%T\n' now" to print the wrong time zone name.
Backport the ksh2020 fix for timezone name determination This fix for `printf '%T' now` on FreeBSD was written by @krader1961. This is from https://github.com/att/ast/pull/591: On FreeBSD calling tzset() does not guarantee the tzname array will be correctly populated. On most systems that works but on FreeBSD you have to call localtime() or a related function (e.g., ctime()). This change also eliminates a potential, very small, memory leak due to the strdup()'ed tznames not being freed. src/lib/libast/tm/tminit.c: - Fix timezone name determination on FreeBSD and a memory leak.
2020-06-13 21:16:08 +00:00
POSIX compliance fix: make 'unalias' a regular builtin Both 'alias' and 'unalias' are specified as regular builtins. Among a few other things, that means it ought to be portable to use these names for shell functions. But ksh93 disallowed that until now. src/cmd/ksh93/data/builtins.c: - Make 'unalias' a regular builtin by removing the BLT_SPC flag. - (same fix for 'alias' was already done in afa68dca) - Add the BLT_ENV flag to the 'alias' and 'hash' commands. In include/name.h, this flag is commented: "non-stoppable, can modify environment". The "non-stoppable" bit seems like a good idea: these operations should not be interruptable as that would cause an inconsistent state. src/cmd/ksh93/sh.1: - Remove the '-', indicating special builtin, from 'alias' entry. - Minor cosmetic fix: space after the '-' for 'unset'. (cherry picked from commit a4315d7672204acb543010b4d4916b22dcb9cb08)
2020-06-11 10:43:17 +00:00
2020-06-11:
`builtin -d` should not delete special builtins The man page for the builtin command says special builtins cannot be deleted. This wasn't the case though, running `builtin -d` on a special builtin was deleting it. As an example, the following set of commands was ending with 'export: not found': $ builtin -d export $ export foo=bar This commit backports the bugfix from ksh93v- (2014-12-24-beta), which added an error check to prevent special builtins from being deleted. src/cmd/ksh93/sh/nvdisc.c: - Add an error check to prevent special builtins from being deleted. src/cmd/ksh93/tests/builtins.sh - Add a regression test for using `builtin -d` on special builtins.
2020-06-12 03:18:01 +00:00
- Fixed a bug that caused running 'builtin -d' on a special builtin to
delete it. The man page for the 'builtin' command documents that special
builtins cannot be deleted.
POSIX compliance fix: make 'unalias' a regular builtin Both 'alias' and 'unalias' are specified as regular builtins. Among a few other things, that means it ought to be portable to use these names for shell functions. But ksh93 disallowed that until now. src/cmd/ksh93/data/builtins.c: - Make 'unalias' a regular builtin by removing the BLT_SPC flag. - (same fix for 'alias' was already done in afa68dca) - Add the BLT_ENV flag to the 'alias' and 'hash' commands. In include/name.h, this flag is commented: "non-stoppable, can modify environment". The "non-stoppable" bit seems like a good idea: these operations should not be interruptable as that would cause an inconsistent state. src/cmd/ksh93/sh.1: - Remove the '-', indicating special builtin, from 'alias' entry. - Minor cosmetic fix: space after the '-' for 'unset'. (cherry picked from commit a4315d7672204acb543010b4d4916b22dcb9cb08)
2020-06-11 10:43:17 +00:00
- POSIX compliance fix: It is now possible to set shell functions named
'alias' or 'unalias', overriding the commands by the same names. In
technical terms, they are now regular builtins, not special builtins.
Make 'redirect' a regular builtin instead of an alias of 'exec' This commit converts the redirect='command exec' alias to a regular 'redirect' builtin command that only accepts I/O redirections, which persist as in 'exec'. This means that: * 'unlias -a' no longer removes the 'redirect' command; * users no longer accidentally get logged out of their shells if they type something intuitive but wrong, like 'redirect ls >file'. This should not introduce any legitimate change in behaviour. If someone did accidentally pass non-redirection arguments to 'redirect', unexpected behaviour would occur; this now produces an 'incorrect syntax' error. src/cmd/ksh93/bltins/misc.c: b_exec(): - Recognise 'redirect' when parsing options. - If invoked as 'redirect', produce error if there are arguments. src/cmd/ksh93/data/aliases.c: - Remove redirect='command exec' alias. src/cmd/ksh93/data/builtins.c: - Update/improve comments re ordering. - Add 'redirect' builtin entry. - sh_optexec[]: Abbreviate redirection-related documentation; refer to redirect(1) instead. - sh_optredirect[]: Add documentation. src/cmd/ksh93/include/builtins.h: - Add SYSREDIR parser ID, renumbering those following it. - Improve comments. - Add extern sh_optredirect[]. src/cmd/ksh93/sh.1: - exec: Abbreviate redirection-related documentation; refer to 'redirect' instead. - redirect: Add documentation. src/cmd/ksh93/sh/xec.c: - Recognise SYSREDIR parser ID in addition to SYSEXEC when determining whether to make redirections persistent. src/cmd/ksh93/tests/io.sh: - To regress-test the new builtin, change most 'command exec' uses to 'redirect'. - Add tests verifying the exit behaviour of 'exec', 'command exec', 'redirect' on redirections.
2020-06-12 02:17:14 +00:00
- The redirect='command exec' alias has been converted to a regular
'redirect' builtin command that only accepts I/O redirections, which
persist as in 'exec'. This means that:
* 'unlias -a' no longer removes the 'redirect' command;
* users no longer accidentally get logged out of their shells if
they type something intuitive but wrong, like 'redirect ls >file'.
Remove 'login' and 'newgrp' builtins: not sane default behaviour This commit removes the undocumented 'login' and 'newgrp' builtin commands. They already stopped blocking shell functions by that name by changing from special to regular builtins in 04b91718 (a change I forgot to mention in that commit message), but there is another obnoxious aspect to these: being glorified hooks into 'exec', they replaced your shell session with the external commands by the same name. This makes argument and error checking impossible, so if you made so much as a typo, you would be immediately logged out. Even if that behaviour is wanted by a few, having it as the default is user-hostile enough to be called a bug. It also violates the POSIX definition of the 'newgrp' utility which explicitly says that it "shall create a new shell execution environment", not replace the existing one. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/newgrp.html Users who do want this behaviour can easily restore it by setting: alias login='exec login' alias newgrp='exec newgrp' src/cmd/ksh93/bltins/misc.c: - As there is no more 'login' builtin, combine b_exec() and B_login() functions, which allows eliminating a few variables. Note that most of 'exec' was actually implemented in B_login()! src/cmd/ksh93/data/builtins.c: - Remove "login" and "newgrp" table entries. src/cmd/ksh93/include/builtins.h: - Remove SYSLOGIN parser ID. As this was the first, all the others needed renumbering. src/cmd/ksh93/sh/xec.c: - Remove SYSLOGIN parser check that made 'login' and 'newgrp' act like 'exec' and replace the shell.
2020-06-12 04:57:57 +00:00
- The undocumented 'login' and 'newgrp' builtin commands have been removed.
These replaced your shell session with the external commands by the same
name, as in 'exec'. If an error occurred (e.g. due to a typo), you would
end up immediately logged out.
If you do want this behaviour, you can restore it by setting:
alias login='exec login'
alias newgrp='exec newgrp'
Replace the hash alias with a proper builtin This commit replaces the old hash alias with a proper builtin. I based this builtin off of the code alias uses for handling `alias -t --`, but with the hack for `--` removed as it has no use in the new builtin. `alias -t --` will no longer work, that hack is now gone. While I was testing this builtin, I found a bug with hash tables in non-forking subshells. If the hash table of a non-forking subshell is changed, the parent shell's hash table is also changed. As an example, running `(hash -r)` was resetting the parent shell's hash table. The workaround is to force the subshell to fork if the hash table will be changed. src/cmd/ksh93/bltins/typeset.c: - Move the code for hash out of the alias builtin into a dedicated hash builtin. `alias -t --` is no longer supported. src/cmd/ksh93/data/aliases.c: - Remove the old alias for hash from the table of predefined aliases. src/cmd/ksh93/data/builtins.c: - Fix the broken entry for the hash builtin and add a man page for the new builtin. src/cmd/ksh93/sh.1: - Replace the entry for the hash alias with a more detailed entry for the hash builtin. src/cmd/ksh93/sh/name.c: - Force non-forking subshells to fork if the PATH is being reset to workaround a bug with the hash tree. src/cmd/ksh93/tests/alias.sh: - Add a regression test for resetting a hash table, then adding a utility to the refreshed hash table. src/cmd/ksh93/tests/subshell.sh: - Add regression tests for changing the hash table in subshells. (cherry picked from commit d8428a833afe9270b61745ba3d6df355fe1d5499)
2020-06-10 11:00:35 +00:00
2020-06-10:
- The 'hash' utility is now a regular builtin instead of an alias to
'alias -t --'. The functionality of the old command has been removed
from the alias builtin.
Fix some errors in the documentation and other minor issues (#42) Somewhat notable changes in this commit: - The 'set +r' bugfix (re: 74b41621) is now documented in the changelog. - Missing options have been added to the synopsis section of the ksh man page. - The minor formatting fix from https://github.com/ksh-community/ksh/pull/5 has been applied to the ksh man page. - A few fixes from https://github.com/att/ast/commit/5e747cfb have been applied to the ksh man page. - The man page fixes from https://github.com/att/ast/pull/353 have been applied, being: - An addition to document the behavior of 'set -H'. - A fix for the cd section appending rksh93. - A fix for some options being indented too far. - Removal of a duplicate section documenting '-D'. - Reordering the options for 'set' in alphabetical order. - A minor fix for the documentation of 'ksh -i'.
2020-06-25 15:21:40 +00:00
- 'set +r' is no longer able to unset the restricted option. This change
makes the behavior of 'set +r' identical to 'set +o restricted'.
The unalias builtin should return an error for non-existent aliases This commit fixes a bug that caused unalias to return a zero status when it tries to remove an alias twice. The following set of commands will no longer end with an error: $ alias foo=bar $ unalias foo $ unalias foo && echo 'Error' This commit is based on the fix present in ksh2020, but it has been extended with another bugfix. The initial fix for this problem tried to remove aliases from the alias tree without accounting for NV_NOFREE. This caused any attempt to remove a predefined aliases (e.g. `unalias float`) to trigger an error with free, as all predefined aliases are in read-only memory. The fix for this problem is to set NV_NOFREE when removing aliases from the alias tree, but only if the alias is in read-only memory. All other aliases must be freed from memory to prevent memory leaks. I'll also note that I am using an `isalias` variable rather than the `type` enum from ksh2020, as the `VARIABLE` value is never used and was replaced with a bool called `aliases` in the ksh2020 release. The `isalias` variable is an int as the ksh93u+ codebase does not use C99 bools. Previous discussion: https://github.com/att/ast/issues/909 - src/cmd/ksh93/bltins/typeset.c: Remove aliases from the alias tree by using nv_delete. NV_NOFREE is only used when it is necessary. - src/cmd/ksh93/tests/alias.sh: Add two regression tests for the bugs fixed by this commit. (cherry picked from commit 16d5ea9b52ba51f9d1bca115ce8f4f18e97abbc4)
2020-06-09 15:31:00 +00:00
2020-06-09:
- The 'unalias' builtin will now return a non-zero status if it tries
to remove a previously set alias that is not currently set.
POSIX compliance fix: apply 'set -u' to $1, $2, ... POSIX requires[*] that expanding any unset parameter other than $@ and $* is an error when 'set -u'/'set -o nounset' is active. However, on ksh93, $1, $2, ... were exempt as well. That is a bug. [*] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_25 src/cmd/ksh93/sh/macro.c: - varsub(): Backport code for handling 'set -u' for positional parameters from the ast 2016-10-01-beta branch. src/cmd/ksh93/tests/options.sh: - Add relevant regression tests. src/cmd/ksh93/sh.1: - Document that $@ and $* are exempt from 'set -u'. (cherry picked from commit f954c6be0748c4c38a680a75f27564965fbd328e)
2020-06-07 22:01:24 +00:00
2020-06-08:
Fix 39 spelling errors and a formatting issue A column of whitespace in the NEWS file was removed for consistent formatting. Most of the spelling errors were found with this codespell dictionary: https://github.com/orbitcowboy/codespell_dictionary (cherry picked from commit 0e36b17abe5609c461a3e4da7041eb0fdf9991b7)
2020-06-10 12:16:53 +00:00
- Fix an issue with the up arrow key in Emacs editing mode.
Emacs editing mode is bugged in ksh93u+ and ksh2020. Let's
say you were to run the following commands after starting
a fresh instance of ksh:
Fix various minor problems and update the documentation (#237) These are minor fixes I've accumulated over time. The following changes are somewhat notable: - Added a missing entry for 'typeset -s' to the man page. - Add strftime(3) to the 'see also' section. This and the date(1) addition are meant to add onto the documentation for 'printf %T'. - Removed the man page the entry for ksh reading $PWD/.profile on login. That feature was removed in commit aa7713c2. - Added date(1) to the 'see also' section of the man page. - Note that the 'hash' command can be used instead of 'alias -t' to workaround one of the caveats listed in the man page. - Use an 'out of memory' error message rather than 'out of space' when memory allocation fails. - Replaced backticks with quotes in some places for consistency. - Added missing documentation for the %P date format. - Added missing documentation for the printf %Q and %p formats (backported from ksh2020: https://github.com/att/ast/pull/1032). - The comments that show each builtin's options have been updated.
2021-03-21 14:39:03 +00:00
$ alias foo=true
Fix 39 spelling errors and a formatting issue A column of whitespace in the NEWS file was removed for consistent formatting. Most of the spelling errors were found with this codespell dictionary: https://github.com/orbitcowboy/codespell_dictionary (cherry picked from commit 0e36b17abe5609c461a3e4da7041eb0fdf9991b7)
2020-06-10 12:16:53 +00:00
$ unalias foo
If you type 'a' and then press the up arrow on your keyboard,
Fix various minor problems and update the documentation (#237) These are minor fixes I've accumulated over time. The following changes are somewhat notable: - Added a missing entry for 'typeset -s' to the man page. - Add strftime(3) to the 'see also' section. This and the date(1) addition are meant to add onto the documentation for 'printf %T'. - Removed the man page the entry for ksh reading $PWD/.profile on login. That feature was removed in commit aa7713c2. - Added date(1) to the 'see also' section of the man page. - Note that the 'hash' command can be used instead of 'alias -t' to workaround one of the caveats listed in the man page. - Use an 'out of memory' error message rather than 'out of space' when memory allocation fails. - Replaced backticks with quotes in some places for consistency. - Added missing documentation for the %P date format. - Added missing documentation for the printf %Q and %p formats (backported from ksh2020: https://github.com/att/ast/pull/1032). - The comments that show each builtin's options have been updated.
2021-03-21 14:39:03 +00:00
ksh will complete 'a' to 'alias foo=true' by doing a reverse
Fix 39 spelling errors and a formatting issue A column of whitespace in the NEWS file was removed for consistent formatting. Most of the spelling errors were found with this codespell dictionary: https://github.com/orbitcowboy/codespell_dictionary (cherry picked from commit 0e36b17abe5609c461a3e4da7041eb0fdf9991b7)
2020-06-10 12:16:53 +00:00
search for the last command that starts with 'a'.
Run the alias command again, then type 'u' and press the up
Fix various minor problems and update the documentation (#237) These are minor fixes I've accumulated over time. The following changes are somewhat notable: - Added a missing entry for 'typeset -s' to the man page. - Add strftime(3) to the 'see also' section. This and the date(1) addition are meant to add onto the documentation for 'printf %T'. - Removed the man page the entry for ksh reading $PWD/.profile on login. That feature was removed in commit aa7713c2. - Added date(1) to the 'see also' section of the man page. - Note that the 'hash' command can be used instead of 'alias -t' to workaround one of the caveats listed in the man page. - Use an 'out of memory' error message rather than 'out of space' when memory allocation fails. - Replaced backticks with quotes in some places for consistency. - Added missing documentation for the %P date format. - Added missing documentation for the printf %Q and %p formats (backported from ksh2020: https://github.com/att/ast/pull/1032). - The comments that show each builtin's options have been updated.
2021-03-21 14:39:03 +00:00
arrow key again. If ksh is in Vi mode, you will get 'unalias foo',
but in Emacs mode you will get 'alias foo=true' again.
Fix 39 spelling errors and a formatting issue A column of whitespace in the NEWS file was removed for consistent formatting. Most of the spelling errors were found with this codespell dictionary: https://github.com/orbitcowboy/codespell_dictionary (cherry picked from commit 0e36b17abe5609c461a3e4da7041eb0fdf9991b7)
2020-06-10 12:16:53 +00:00
All subsequent commands were ignored as ksh was saving the first
command and only based later searches off of it.
Fix an issue with the up arrow key in Emacs editing mode. Emacs editing mode is bugged in ksh93u+ and ksh2020. Let's say you were to run the following commands after starting a fresh instance of ksh: $ alias foo='true' $ unalias foo If you type 'a' and then press the up arrow on your keyboard, ksh will complete 'a' to `alias foo='true'` by doing a reverse search for the last command that starts with 'a'. Run the alias command again, then type 'u' and press the up arrow key again. If ksh is in Vi mode, you will get `unalias foo`, but in Emacs mode you will get `alias foo='true'` again. This bug was occurring because ksh was only doing a reverse search based on the first command that was completed using the up arrow. All subsequent commands were ignored as ksh was saving the first command and only based later searches off of it. NEWS: - Add instructions for reproducing this bug with the up arrow key and information about why this bug was happening in the first place. src/cmd/ksh93/edit/emacs.c: - Remove a bad check that was preventing ksh from using the latest input for reverse search. (cherry picked from commit 745b065487ad6bac693ec6f44752f96e87f9a63b)
2020-06-08 10:22:51 +00:00
POSIX compliance fix: apply 'set -u' to $1, $2, ... POSIX requires[*] that expanding any unset parameter other than $@ and $* is an error when 'set -u'/'set -o nounset' is active. However, on ksh93, $1, $2, ... were exempt as well. That is a bug. [*] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_25 src/cmd/ksh93/sh/macro.c: - varsub(): Backport code for handling 'set -u' for positional parameters from the ast 2016-10-01-beta branch. src/cmd/ksh93/tests/options.sh: - Add relevant regression tests. src/cmd/ksh93/sh.1: - Document that $@ and $* are exempt from 'set -u'. (cherry picked from commit f954c6be0748c4c38a680a75f27564965fbd328e)
2020-06-07 22:01:24 +00:00
- If 'set -u'/'set -o nounset' is active, then the shell now errors out if a
nonexistent positional parameter such as $1, $2, ... is accessed, as other
shells do and POSIX requires. (This does *not* apply to "$@" and "$*".)
POSIX compliance fix: apply 'set -u' to $! POSIX requires[*] that expanding any unset parameter other than $@ and $* is an error when 'set -u'/'set -o nounset' is active. However, on ksh93, $! was exempt as well. That is a bug. [*] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_25 src/cmd/ksh93/sh/macro.c: - special(): Handle 'set -u' for special parameters if/when it is about to return NIL. That code path is currently only possible to reach for "$!", but this is future-proof and will do the right thing if any other special parameter can ever have no value. src/cmd/ksh93/tests/options.sh: - Add and tweak 'set -u' regression tests. (cherry picked from commit 75cc7a38cafe3a9929e1ed17d8b952babda22a09)
2020-06-07 23:05:27 +00:00
- If 'set -u'/'set -o nounset' is active, then the shell now errors out if $!
is accessed before the shell has launched any background process.
Remove code related to long-dead 3DFS research project This commit gets rid of dead weight related to an obscure early 1990s Bell Labs versioning file system research project called 3DFS, which has not existed for decades and for which I have not managed to find any evidence that it was ever used outside the lab. This removes: - the SHOPT_FS_3D compile option (which was forced on even when 0) - the obnoxious default alias 2d='set -f;_2d' that turned off your globbing and then tried to run a nonexistent _2d command - undocumented builtins 'vmap' and 'vpath' that only errored out - a non-functional -V unary operator for the test and [[ commands - some specific code for Apollo workstations (last made in 1997), which was inseparably intertwined with the 3DFS code (cherry picked from commit 20cdf3709f4fb4e468057b534dcee819b1961fb6)
2020-06-08 01:28:36 +00:00
- Removed support for an obscure early 1990s Bell Labs file system research
project called 3DFS, which has not existed for decades. This removes:
- an obnoxious default alias 2d='set -f;_2d' that turned off your file name
wildcard expansion and then tried to run a nonexistent '_2d' command
- undocumented builtins 'vmap' and 'vpath' that only printed error messages
- a non-functional -V unary operator for the test and [[ commands
Fix signal handling due to exit status > 256 This fixes two bugs: issuing the 'exit' command with a value > 256 would cause ksh 93u+ to kill itself with the corresponding signal (try 'exit 265' to SIGKILL your interactive shell), and, if the last command of a script exits due to a signal, the shell would repeat that signal to itself, causing any parent ksh to also be killed. Discussion: https://bugzilla.redhat.com/show_bug.cgi?id=1469624 https://rainbow.chard.org/2017/03/21/ksh-deliberately-segfaults-if-the-last-command-in-a-script-crashes/ This commit is loosely based on a patch applied to the 93v- beta and the abandoned ksh2020, but that patch was incomplete & broken: $ ksh-2020.0.0 -c 'exit 265'; echo $? 137 Expected: 9. Since the exit was *not* due to a signal, the value should simply be cropped to the 8 bits supported by the OS. src/cmd/ksh93/bltins/cflow.c: b_exit(): - For the 'exit' builtin command, bitwise-AND the argument to 'exit' with SH_EXITMASK (8 bits, crop to 0-255) before passing it on to sh_exit(). This restores the behaviour of <=2011 ksh93 versions and is in line with all other POSIX shells. It also fixes this bogosity: $ (exit 265); echo $? # non-forked subshell 265 $ (ulimit -t unlimited; exit 265); echo $? # forked subshell 9 Forked or non-forked should make no difference at all (see commit message a0e0e29e for why). src/cmd/ksh93/sh/fault.c: sh_done(): - If the current exit status is equal to the status for the last signal that was received from a child process, remove the SH_EXITSIG (9th) bit, so that the shell doesn't kill itself. - If the shell's last child process exits due to a signal, exit with a portable 8-bit exit status (128 + signal number). This avoids the exit status being < 128 by being cropped to 8 bits. src/cmd/ksh93/tests/signal.sh: - Add regression test for exit with status > 256. - Add regression test verifying the shell no longer kills itself. (cherry picked from commit 98e0fc94393e175ce6adfee390327c320795bf12)
2020-06-08 10:23:37 +00:00
- If the last program run by a ksh script exits with a signal (e.g. crashed),
ksh itself now exits normally instead of repeating that same signal.
In addition, using 'exit x' for x > 256 no longer makes ksh issue a signal.
POSIX compliance fix: make 'times' a proper builtin As of this commit, the 'times' command is a POSIX-compliant special builtin command instead of an alias that doesn't produce the required output. It displays the accumulated user and system CPU times, one line with the times used by the shell and another with those used by all of the shell's child processes. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_27 This was originally written by Kurtis Rader and is now backported and tweaked from the abandoned ksh2020 branch. I chose an earlier and simpler version[*1] that uses times(3), with a precision of hundredths of seconds, so it outputs the same precision as mksh and zsh. Rader later wrote another version[*2] that uses getrusage(2), giving it the same millisecond precision as bash. But that required adding a feature test and a fallback to the old version, which is non-trivial in the old INIT/iffe system. This simpler version is enough to gain POSIX compliance and I think it will do very nicely in this stable bugfix branch. [*1] https://github.com/att/ast/pull/1332 [*2] https://github.com/att/ast/commit/038045f6 src/cmd/ksh93/bltins/misc.c - Add b_times() function for 'times' builtin. - Note we include <times.h>, not <sys/times.h>, so that we use the AST feature-tested version with fallback on systems that need it. src/cmd/ksh93/data/aliases.c: - Remove times='{ { time;} 2>&1;}' builtin alias. src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/include/builtins.h: - Add entry for 'times' special builtin. - Add --help/--man info for same. src/cmd/ksh93/sh.1: - Update manual page. src/cmd/ksh93/tests/builtins.sh: - Add a couple of simple regression tests. (cherry picked from commit ebf71e619eb298ec1cf6b81d1828fa7cdf6e9203)
2020-06-06 19:25:59 +00:00
2020-06-06:
- The 'times' command is now a builtin command that conforms to POSIX
instead of an alias for the 'time' command. It displays the accumulated
user and system CPU times, one line with the times used by the shell and
another with those used by all of the shell's child processes.
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_27
POSIX compliance: rm harmful default aliases 'command '/'nohup ' Continuing alias substitution after 'command' (due to the final space in the alias) is inherently broken and doing so by default is incompatible with the POSIX standard, as aliases may contain arbitrary shell grammar. For instance, until the previous commit, the POSIX standard 'times' command was an alias: times='{ { time;} 2>&1;}' -- and so, of course, 'command times' gave a syntax error, although this is a perfectly valid POSIX idiom that must be supported. 'command' is specified by POSIX as a regular builtin, not an alias. Therefore it should always bypass aliases just as it bypasses functions to expose standard builtin and external commands. I can only imagine that the reason for this command='command ' alias was that some standard commands themselves were implemented as aliases, and POSIX requires that standard commands are accessible with the 'command' prefix. But implementing standard commands as aliases is itself inherently broken. It never worked for 'command times', as shown; and in any case, removing all aliases with 'unalias -a' should not get rid of standard commands. Similarly, the default alias nohup='nohup ' is also harmful. Anyone who really wants to keep this behaviour can just define these aliases themselves in their script or ~/.kshrc file. src/cmd/ksh93/data/aliases.c: - Remove default alias command='command '. - Remove default alias nohup='nohup '. src/cmd/ksh93/sh.1 - Remove the above default aliases from the list. - Mention that the 'command' builtin does not search for aliases. (cherry picked from commit 5cfe7c4e2015b7445da24983af5008035c4b6e1e)
2020-06-07 00:36:54 +00:00
- The default aliases command='command ' and nohup='nohup ' have been
removed because they caused breakage in an attempt to circumvent other
breakage which is being fixed. In the unlikely even that anyone still
needs alias substitution to continue on the command argument following
'command' or 'nohup', it's easy to set these aliases yourself.
Fix unsetting special vars in subshells (re: efa31503, 8b7f8f9b) This fixes (or at least works around) a bug that caused special variables such as PATH, LANG, LC_ALL, LINENO, etc. to lose their effect after being unset in a subshell. For example: (unset PATH; PATH=/dev/null; ls); : wrongly ran 'ls' (unset LC_ALL; LC_ALL=badlocale); : failed to print a diagnostic This is yet another problem with non-forking/virtual subshells. If you forced the subshell to fork (one way of doing this is using the 'ulimit' builtin, e.g. ulimit -t unlimited) before unsetting the special variable, the problem vanished. I've tried to localise the problem. I suspect the sh_assignok() function, which is called from unall(), is to blame. This function is supposed to make a copy of a variable node in the virtual subshell's variable tree. Apparently, it fails to copy the associated permanent discipline function settings (stored in the np->nvfun->disc pointer) that gave these variables their special effect, and which survive unset. However, my attempts to fix that have been unsuccessful. If anyone can figure out a fix, please send a patch/pull request! Data point: This bug existed in 93u 2011-02-08, but did not yet exist in M-1993-12-28-s+. So it is a regression. Meanwhile, pending a proper fix, this commit adds a safe workaround: it forces a non-forked subshell to fork before unsetting such a special variable. src/cmd/ksh93/bltins/typeset.c: unall(): - If we're in a non-forked, non-${ ...; } subshell, then before unsetting any variable, check for variables with internal trap/discipline functions, and call sh_subfork() if any are found. To avoid crashing, this must be done before calling sh_pushcontext(), so we need to loop through the args separately. src/cmd/ksh93/tests/variables.sh: - Remove the 'ulimit' that forced the fork; we do this in C now. (cherry picked from commit 21b1a67156582e3cbd36936f4af908bb45211a4b)
2020-06-05 13:31:26 +00:00
2020-06-05:
- Fix a bug that caused special variables such as PATH, LANG, LC_ALL,
etc. to lose their effect after being unset in a subshell. For example:
(unset PATH; PATH=/dev/null; ls); : wrongly ran 'ls'
(unset LC_ALL; LC_ALL=badlocale); : failed to print a diagnostic
Add regress test for fixed BUG_KUNSETIFS (re: 6f0e008c, 7b994b6a) Modernish is no longer detecting BUG_KUNSETIFS, as I've just discovered. Always nice when bugs spontaneously vanish... A 'git reset HEAD~1'/recompile/retest loop reveals this bug was fixed by 6f0e008c, as later modified by 7b994b6a. So, let's make sure it stays fixed. src/cmd/ksh93/tests/variables.sh: - Add a couple of regression tests for BUG_KUNSETIFS presence, detection and known workaround, based on the same in modernish. Ref.: https://github.com/modernish/modernish/blob/3ddcbd13/lib/modernish/cap/BUG_KUNSETIFS.t https://github.com/modernish/modernish/blob/3ddcbd13/lib/modernish/tst/isset.t#L204-L222
2020-06-24 17:38:09 +00:00
This also fixes BUG_KUNSETIFS: unsetting IFS in a subshell failed if IFS
was set to the empty value in the parent shell.
Fix unsetting special vars in subshells (re: efa31503, 8b7f8f9b) This fixes (or at least works around) a bug that caused special variables such as PATH, LANG, LC_ALL, LINENO, etc. to lose their effect after being unset in a subshell. For example: (unset PATH; PATH=/dev/null; ls); : wrongly ran 'ls' (unset LC_ALL; LC_ALL=badlocale); : failed to print a diagnostic This is yet another problem with non-forking/virtual subshells. If you forced the subshell to fork (one way of doing this is using the 'ulimit' builtin, e.g. ulimit -t unlimited) before unsetting the special variable, the problem vanished. I've tried to localise the problem. I suspect the sh_assignok() function, which is called from unall(), is to blame. This function is supposed to make a copy of a variable node in the virtual subshell's variable tree. Apparently, it fails to copy the associated permanent discipline function settings (stored in the np->nvfun->disc pointer) that gave these variables their special effect, and which survive unset. However, my attempts to fix that have been unsuccessful. If anyone can figure out a fix, please send a patch/pull request! Data point: This bug existed in 93u 2011-02-08, but did not yet exist in M-1993-12-28-s+. So it is a regression. Meanwhile, pending a proper fix, this commit adds a safe workaround: it forces a non-forked subshell to fork before unsetting such a special variable. src/cmd/ksh93/bltins/typeset.c: unall(): - If we're in a non-forked, non-${ ...; } subshell, then before unsetting any variable, check for variables with internal trap/discipline functions, and call sh_subfork() if any are found. To avoid crashing, this must be done before calling sh_pushcontext(), so we need to loop through the args separately. src/cmd/ksh93/tests/variables.sh: - Remove the 'ulimit' that forced the fork; we do this in C now. (cherry picked from commit 21b1a67156582e3cbd36936f4af908bb45211a4b)
2020-06-05 13:31:26 +00:00
Fix various crashes by removing invalid memccpy() use The sfputr() function (put out a null-terminated string) contained a use of memccpy() that was invalid and could cause crashes, because the buffer it was copying into could overlap or even be identical with the buffer being copied from. Among (probably) other things, this commit fixes a crash in 'print -v' (print a compound variable structure) on macOS, that caused the comvar.sh and comvario.sh regression tests to fail spectacularly. Now they pass. Issue discovered and fixed by Kurtis Rader in the abandoned ksh2020 branch; this commit backports the fix. He wrote: | #if _lib_memccpy && !__ia64 /* these guys may never get it right */ | | The problem is that assertion is wrong. It implies that the libc | implementation of memccpy() on IA64 is broken. Which is | incorrect. The problem is the AST sfputr() function is depending | on what is explicitly undefined behavior in the face of | overlapping source and destination buffers. | [...] Using memccpy() simply complicates the code and is unlikely | to be measurably, let alone noticeably, faster. Further discussion/analysis: https://github.com/att/ast/issues/78 src/lib/libast/sfio/sfputr.c: - Remove memccpy use. Always use the manual copying loop. (cherry picked from commit fbe3c83335256cb714a4aa21f555083c9f1d71d8)
2020-06-05 14:03:07 +00:00
- Fix crashes on some systems, including at least a crash in 'print -v' on
macOS, by eliminating an invalid/undefined use of memccpy() on overlapping
buffers in the commonly used sfputr() function.
Fix ${.sh.subshell} counter to actually count level of subshells This counter is documented as follows: "The current depth for subshells and command substitution." But before this commit, the actual behaviour was that the counter was reset to zero whenever a subshell forked for any reason: a pipe, background job, running 'ulimit', redirecting stdout in a command substitution, and more. This behaviour was: 1. Not consistent with the documentation. Non-forked (a.k.a. virtual) subshells are an internal implementation detail which scripts should not have to be concerned with. The manual page doesn't mention them at all. 2. Inherently broken. Since a subshell may fork for any number of reasons, even mid-run, and those reasons may change with bugfixes and further development, scripts have never actually been able to rely on the value of ${.sh.subshell}. So, this commit fixes the counter to count the levels of all subshells, both virtual and forked. src/cmd/ksh93/sh/xec.c: _sh_fork(): - Increase ${.sh.subshell} whenever we fork. src/cmd/ksh93/sh/subshell.c: - sh_subfork(): * Fix comment to properly explain what it does. It doesn't "create" a subshell, it forks off an existing virtual subshell. * Don't zero ${.sh.subshell}. Instead, since sh_fork() increases it but we're forking an existing subshell, undo the increase. - sh_subshell(): * Remove 'int16_t subshell' variable. It was unnecessary and mostly unused. It was also the wrong type: it was assigned the value from shp->subshell which is of type short. * Increase and decrease the level of virtual subshells and ${.sh.subshell} independently. src/cmd/ksh93/tests/variables.sh: - Add regression tests for ${.sh.subshell} in virtual and forked subshells of several kinds: comsub, parentheses, pipe, bg job. - Undo wrong error test count fix from 04b4aef0. (cherry picked from commit a0e0e29e7e0dbf21e4b3958ae02bde6665fb2696)
2020-06-05 19:39:47 +00:00
- Fix the ${.sh.subshell} level counter; it is no longer reset to zero when a
non-forked subshell happens to fork into a separate process for some reason
(an internal implementation detail that should be unnoticeable to scripts).
Fix BUG_KBGPID: $! was not updated under certain conditions The $! special parameter was not set if a background job (somecommand &) or co-process (somecommand |&) was launched as the only command within a braces block with an attached redirection, for example: { somecommand & } >&2 With the bug, $! was unchanged; now it contains the PID of somecommand. Ref.: https://github.com/att/ast/issues/1357 src/cmd/ksh93/sh/parse.c: item(): - When processing redirections following a compound command, always create a parent node with the TSETIO (I/O redirection) token. Before this commit, if the last command was of type TFORK (and the last command only tested as TFORK if the bg job or coprocess was the only command in a braces block, because the ksh parser optimises away the braces in that case), then the parent node was created with the TFORK token instead. I have no idea what David Korn's intention was with that, but this is clearly very wrong. Creating another TFORK node when parsing the redirection caused sh_exec() in sh/xec.c to execute the redirection in an extra forked, non-background subshell. Since redirections are executed before anything else, this subshell is what then launched the background job between the braces, so $! (a.k.a. shp->bckpid) was updated in that subshell only, and never in the main shell. The extra subshell also prevented the background job from being noticed by job control on interactive shells. So, the fix is simply to remove the broken test for TFORK. src/cmd/ksh93/tests/variables.sh: - Add regression tests for a bg job and a co-process as the only command within a braces block with attached redirection. (cherry picked from commit ffe5df30e69f7b596941a98498014d8e838861f2)
2020-06-04 03:19:59 +00:00
2020-06-04:
- Fix BUG_KBGPID: the $! special parameter was not set if a background job
(somecommand &) or co-process (somecommand |&) was launched as the only
command within a braces block with an attached redirection, for example:
{
somecommand &
} >&2
With the bug, $! was unchanged; now it contains the PID of somecommand.
Fix handling of skipped directories when autoloading functions Fix a bug in autoloading functions. Directories in the path search list which should be skipped (e.g. because they don't exist) did not interact correctly with autoloaded functions, so that a function to autoload was not always found. Details: https://github.com/att/ast/issues/1454 Fix backported (and cleaned up) from: https://github.com/att/ast/commit/3bc58164 src/cmd/ksh93/sh/path.c: - path_opentype(): Fix the path search loop so that entries marked with PATH_SKIP are handled correctly. src/cmd/ksh93/tests/functions.sh: - Add regression test verifying an autoloaded function with a PATH that triggered the bug. The bug in path_opentype() fixed by this commit may affect other scenarios but we know it affects autoloaded functions. Hence the test for that scenario. (cherry picked from commit a27903165775309f4f032de5d42ec1785f14cfbc)
2020-05-31 13:29:15 +00:00
2020-05-31:
- Fix a bug in autoloading functions. Directories in the path search list
which should be skipped (e.g. because they don't exist) did not interact
correctly with autoloaded functions, so that a function to autoload was
not always found correctly.
Details: https://github.com/att/ast/issues/1454
Fix 'test'/'[' exit status >1 on error in arithmetic expression Fix BUG_TESTERR1A: POSIX non-compliance of 'test'/'[' exit status on error. The command now returns status 2 instead of 1 when given an invalid number or arithmetic expression, e.g.: [ 123 -eq 123x ] The problem was that the test builtin (b_test()) calls the generic arithmetic evaluation subsystem (sh/arith.c, sh/streval.c) which has no awareness of the test builtin. A simple solution would be to always make the arithmetic subsystem use an exit status > 1 for arithmetic errors, but globally changing this may cause backwards compatibility issues. So it's best to change the behaviour of the 'test' builtin only. This requires the arithmetic subsystem to be aware of whether it was called from the 'test' builtin or not. To that end, this commit adds a global flag and overrides the ERROR_exit macro where needed. src/cmd/ksh93/include/defs.h, src/cmd/ksh93/sh/defs.c: - Declare and initialise a global sh_in_test_builtin flag. - Declare internal function for ERROR_exit override in test.c. src/cmd/ksh93/bltins/test.c: - Add override for ERROR_exit macro using a function that checks if the exit status is at least 2 if the error occurred while running the test builtin. - b_test(): Set sh_in_test_builtin flag while running test builtin. src/cmd/ksh93/sh/arith.c, src/cmd/ksh93/sh/streval.c: - Override ERROR_exit macro using function from test.c. src/cmd/ksh93/tests/bracket.sh: - Add regression test verifying status > 1 on arith error in test. (cherry picked from commit 5eeae5eb9fd5ed961a5096764ad11ab870a223a9)
2020-05-30 14:21:30 +00:00
2020-05-30:
- Fix POSIX compliance of 'test'/'[' exit status on error. The command now
returns status 2 instead of 1 when given an invalid number or arithmetic
expression, e.g.:
[ 123 -eq 123x ]; echo $?
now outputs 2 instead of 1.
Fix redefining & unsetting functions in subshells (BUG_FNSUBSH) Functions can now be correctly redefined and unset in subshell environments (such as ( ... ), $(command substitutions), etc). Before this fix, attempts to do this were silently ignored (!!!), causing the wrong code (i.e.: the function by the same name from the parent shell environment) to be executed. Redefining and unsetting functions within "shared" command substitutions of the form '${ ...; }' is also fixed. Prior discussion: https://github.com/att/ast/issues/73 src/cmd/ksh93/sh/parse.c: - A fix from George Koelher (URL above). He writes: | The parser can set t->comnamp to the wrong function. | Suppose that the shell has executed | foo() { echo WRONG; } | and is now parsing | (foo() { echo ok; } && foo) | The parser was setting t->comnamp to the wrong foo. [This | fix] doesn't set t->comnamp unless it was a builtin. Now the | subshell can't call t->comnamp, so it looks for foo and finds | the ok foo in the subshell's function tree. src/cmd/ksh93/bltins/typeset.c: - Unsetting functions in a virtual/non-forked subshell still doesn't work: nv_open() fails to find the function. To work around this problem, make 'unset -f' fork the subshell into its own process with sh_subfork(). - The workaround exposed another bug: if we unset a function in a subshell tree that overrode a function by the same name in the main shell, then nv_delete() exposes the function from the main shell scope. Since 'unset -f' now always forks a subshell, the fix is to simply walk though troot's parent views and delete any such zombie functions as well. (Without this, the 4 'more fun' tests in tests/subshell.sh fail.) src/cmd/ksh93/sh/subshell.c: sh_subfuntree(): - Fix function (re)definitions and unsetting in "shared" command substitutions of the form '${ commandlist; }' (i.e.: if sp->shp->subshare is true). Though internally this is a weird form of virtual subshell, the manual page says it does not execute in a subshell (meaning, all changes must survive it), so a subshell function tree must not be created for these. src/cmd/ksh93/tests/subshell.sh: - Add regression tests related to these bugfixes. Test unsetting and redefining a function in all three forms of virtual subshell. (cherry picked from commit dde387825ab1bbd9f2eafc5dc38d5fd0bf9c3652)
2020-05-29 07:27:20 +00:00
2020-05-29:
- Fix BUG_FNSUBSH: functions can now be correctly redefined and unset in
subshell environments (such as ( ... ), $(command substitutions), etc).
Before this fix, this was silently ignored, causing the function by the
same name from the parent shell environment to be executed instead.
fn() { echo mainsh; }
(fn() { echo subsh; }; fn); fn
This now correctly outputs "subsh mainsh" instead of "mainsh mainsh".
ls() { echo "ls executed"; }
(unset -f ls; ls); ls
This now correctly lists your directory and then prints "ls executed",
instead of printing "ls executed" twice.
Fix unsetting aliases in subshells Aliases can now be correctly unset within subshell environments (such as ( ... ), $(command substitutions), etc), as well as non-subshell "shared" command substitutions (${ ...; }). Before, attempts to unset aliases within these were silently ignored. Prior discussion: https://github.com/att/ast/issues/108 Subshell alias trees are only referenced in a few places in the code, *and* have always been broken, so this commit gets rid of the whole notion of a subshell alias tree. Instead, there is now just one flat alias tree, and subshells fork into a separate process when aliases are set or unset within them. It is not really conceivable that this could be a performance-sensitive operation, or even a common one, so this is a clean fix with no downside. src/cmd/ksh93/include/defs.h: - Remove sh_subaliastree() definition. src/cmd/ksh93/sh/subshell.c: - Remove salias element (pointer to subshell alias tree) from subshell struct. - Remove sh_subaliastree() function. - sh_subshell(): Remove alias subshell tree cleanup. src/cmd/ksh93/bltins/typeset.c: - b_alias(): If in subshell, fork before setting alias. - b_unalias(): If in subshell, fork before unsetting alias. - unall(): Remove sh_subaliastree() call. src/cmd/ksh93/sh/name.c: - nv_open(): Remove sh_subaliastree() call. src/cmd/ksh93/tests/subshell.sh: - Add regression tests for unsetting or redefining aliases within subshells. (cherry picked from commit 12a15605b9521a2564a6e657905705a060e89095)
2020-05-29 07:27:53 +00:00
- Fix a similar bug with aliases. These can now be correctly unset
in subshell environments.
Fix truncating of files with <>;file combined with <#pattern The issue with truncating files was caused by out-of-sync streams. Details and discussion: https://github.com/att/ast/issues/61 src/cmd/ksh93/sh/io.c: sh_iorestore(): - To be safe, sync all streams before restoring file descriptors. src/cmd/ksh93/tests/io.sh: - Add two regression tests for truncating files with this combination of redirections. - The second test, which invokes a -c script, is disabled for now as this triggers another corner case bug involving the SH_NOFORK optimisaton for -c scripts. That fix is for another commit. (cherry picked from commit 18fb64840365c2ff4608188e5487bd79d08f67d1)
2020-05-20 21:42:00 +00:00
2020-05-21:
- Fix truncating of files with the combined redirections '<>;file' and
'<#pattern'. The bug was caused by out-of-sync streams.
Details and discussion: https://github.com/att/ast/issues/61
fix 24 more typos found with the help of codespell (cherry picked from commit a92198bc5f196ec1b4a34dc042ff3a594e316ad7)
2020-05-30 14:55:30 +00:00
- Patched code injection vulnerability CVE-2019-14868. As a result, you can
Patch vulnerability CVE-2019-14868 Certain environment variables were interpreted as arithmetic expressions on startup, leading to code injection. Ref.: https://bugzilla.redhat.com/show_bug.cgi?id=1757324 https://github.com/att/ast/commit/c7de8b641266bac7c77942239ac659edfee9ecd2 (cherry picked from commit ee6b001d0611ad2e00b6da2c2b42051995c0a678)
2020-05-21 12:27:51 +00:00
no longer use expressions in imported numeric environment variables; only
integer literals are allowed.
Fix bugs in testing if a parameter is set This fixes three related bugs: 1. Expansions like ${var+set} remained static when used within a 'for', 'while' or 'until' loop; the expansions din't change along with the state of the variable, so they could not be used to check whether a variable is set within a loop if the state of that variable changed in the course of the loop. (BUG_ISSETLOOP) 2. ${IFS+s} always yielded 's', and [[ -v IFS ]] always yielded true, even if IFS is unset. (BUG_IFSISSET) 3. IFS was incorrectly exempt from '-u' ('-o nounset') checks. src/cmd/ksh93/sh/macro.c: varsub(): - When getting a node pointer (np) to the parameter to test, special-case IFS by checking if it has a value and not setting the pointer if not. The node to IFS always exists, even after 'unset -v IFS', so before this fix it always followed the code path for a parameter that is set. This fixes BUG_IFSISSET for ${IFS+s} and also fixes set -u (-o nounset) with IFS. - Before using the 'nv_isnull' macro to check if a regular variable is set, call nv_optimize() if needed. This fixes BUG_ISSETLOOP. Idea from Kurtis Rader: https://github.com/att/ast/issues/1090 Of course this only works if SHOPT_OPTIMIZE==1 (the default), but if not, then this bug is not triggered in the first place. - Add some comments for future reference. src/cmd/ksh93/bltins/test.c: test_unop(): - Fix BUG_IFSISSET for [[ -v IFS ]]. The nv_optimize() method doesn't seem to have any effect here, so the only way that I can figure out is to special-case IFS, nv_getval()'ing it to check if IFS has a value in the current scope. src/cmd/ksh93/tests/variables.sh: - Add regression tests for checking if a varariable is set within a loop, within and outside a function with that variable made local (to check if the scope is honoured). Repeat these tests for a regular variable and for IFS, for ${foo+set} and [[ -v foo ]]. (cherry picked from commit a2cf79cb98fa3e47eca85d9049d1d831636c9b16)
2020-05-20 14:50:43 +00:00
2020-05-20:
- Fix BUG_ISSETLOOP. Expansions like ${var+set} remained static when used
within a 'for', 'while' or 'until' loop; the expansions din't change along
with the state of the variable, so they could not be used to check whether a
variable is set within a loop if the state of that variable changed in the
course of the loop.
- Fix BUG_IFSISSET. ${IFS+s} always yielded 's', and [[ -v IFS ]] always
yielded true, even if IFS is unset. This applied to IFS only.
Fix 'command -p' by fixing initialisation of default PATH variable 'command -p' was broken for non-interactive shells as the variable used to store the default system PATH, std_path, was not initialised correctly. For instance: $ ksh -c 'command -p ls' ksh: ls: not found This fix by Siteshwar Vashisht is backported from ksh2020. Ref.: https://github.com/att/ast/issues/426 https://github.com/att/ast/pull/448 src/cmd/ksh93/sh/path.c: - Correctly initialise std_path (the default PATH) when ksh is started as a non-interactive shell. src/cmd/ksh93/sh.1: - Fix vague explanation of 'command -p'. src/cmd/ksh93/tests/path.sh: - Add regression test. (cherry picked from commit a76439d60b70c18cf44d84c1962fcd8df84c947c)
2020-05-19 14:49:56 +00:00
2020-05-19:
- Fix 'command -p'. The -p option causes the operating system's standard
utilities path (as output by 'getconf PATH') to be searched instead of $PATH.
Before this fix, this was broken on non-interactive shells as the internal
variable holding the default PATH value was not correctly initialised.
Fix 'test -t 1' in $(command substitutions) Standard output (FD 1) tested as being on a terminal within a command substitution, which makes no sense as the command substitution is supposed to be catching standard output. ksh -c 'v=$(echo begincomsub [ -t 1 ] && echo oops echo endcomsub) echo "$v"' This should not output "oops". This is one of the many bugs with ksh93 virtual (non-forked) subshells. On the abandoned Vashist/Rader ksh2020 branch, this bug was fixed by changing quite a lot of code, which introduced and/or exposed another bug: https://github.com/att/ast/issues/1079 https://github.com/att/ast/commit/8e1e405e https://github.com/att/ast/issues/1088 That issue was unresolved when the ksh2020 branch was abandoned. The safer and more conservative fix is simply forcing the subshell to fork if we're in a non-forked command substitution and testing '-t 1'. It is hard to imagine a situation where this would cause a noticable performance hit. Note that this fix does not affect ksh93-specific "shared" non-subshell ${ command substitutions; } which are executed in the main shell environment, so that variables survive, etcetera. 'test -t 1' continues to wrongly return true there, but command substitutions of that form cannot be forked because that would defeat their purpose. src/cmd/ksh93/bltins/test.c: - Fix 'test -t 1', '[ -t 1 ]' and '[[ -t 1 ]]' by forking the current subshell if it is a virtual/non-forked subshell (shp->subshell), and a command substitution (shp->comsub), but NOT a "shared" ${ command substitution; } (!shp->subshare). src/cmd/ksh93/tests/bracket.sh: - Add two regression tests for this issue, which were adapted from the Vashist/Rader ksh2020 branch. NEWS, src/cmd/ksh93/include/version.h: - Update. (cherry picked from commit b8ef05e457ead65b83417699b8dd8632f855e2fa)
2020-05-16 19:06:49 +00:00
2020-05-16:
- Fix 'test -t 1', '[ -t 1 ]', '[[ -t 1 ]]' in command substitutions.
Standard output (file descriptor 1) tested as being on a terminal within a
command substitution, which makes no sense as the command substitution is
supposed to be catching standard output.
v=$(echo begincomsub
[ -t 1 ] && echo oops
echo endcomsub)
echo "$v"
This now does not output "oops".
Fix 'print -s -f' This fix is backported from the Vashisht/Rader ksh2020 branch. src/cmd/ksh93/bltins/print.c: - Fix syncing history when print -s -f is used. For example, the following now correctly adds a 'cd' command to the history: print -s -f 'cd -- %q\n' "$PWD" Ref.: https://github.com/att/ast/issues/425 https://github.com/att/ast/pull/442 src/cmd/ksh93/include/version.h: - Version date bump. (cherry picked from commit 46ba7ecdc5c14cd73f6cb24b16c50bdc331a000e)
2020-05-14 10:36:16 +00:00
2020-05-14:
- Fix syncing history when print -s -f is used. For example, the
following now correctly adds a 'cd' command to the history:
print -s -f 'cd -- %q\n' "$PWD"
Ref.: https://github.com/att/ast/issues/425
https://github.com/att/ast/pull/442
rm "I/O error" error msg; just keep >0 exit status (re: 9011fa93) The bug was really that I/O errors in output builtins were undetectable by any means. Having a >0 exit status is sufficient. Adding an error message risks making existing ksh scripts noisier, or even breaking them if they redirect stderr to stdout. Note to self: in future, implement the minimum change necessary to fix bugs, nothing more. The fact that I needed to add four extra 2>/dev/null to the regression tests should have been a hint. src/cmd/ksh93/bltins/print.c, src/cmd/ksh93/data/msg.c, src/cmd/ksh93/include/io.h: - Remove "I/O error" message. src/cmd/ksh93/tests/builtins.sh: - Update to check for exit status only. src/cmd/ksh93/tests/basic.sh, src/cmd/ksh93/tests/coprocess.sh: - Revert four new '2>/dev/null' to suppress the error message. (cherry picked from commit 5e17be24d18455b575b6e98bc631c6935ffc795a)
2020-06-09 19:57:05 +00:00
- Fix BUG_PUTIOERR: Output builtins now correctly detect
Fix BUG_PUTIOERR: Check for and report I/O error in output builtins This allows scripts to check for a nonzero exit status on the 'print', 'printf' and 'echo' builtins and prevent possible infinite loops if SIGPIPE is ignored. sfsync() was already returning a negative value on I/O error, so all we need to do is add a check. The stream buffer will need to be filled before an I/O error can be detected, but this is the same on other shells. See manual page: src/lib/libast/man/sfio.3 Ref.: https://github.com/att/ast/issues/1093 https://github.com/att/ast/pull/1363 src/cmd/ksh93/bltins/print.c: b_print(): - Make sure an error result from sfsync() is reflected in the output builtin's exit status (exitval). - Write an I/O error message (e_io) if the exit status is nonzero. src/cmd/ksh93/data/msg.c, src/cmd/ksh93/include/io.h: - Add the e_io[] error message. src/cmd/ksh93/tests/builtins.sh: - Add I/O error regression test, checking for the correct error message and exit status. All three output builtins use the same b_print() function so we only need to test one. src/cmd/ksh93/tests/basic.sh, src/cmd/ksh93/tests/coprocess.sh: - Redirect stderr on a few 'print' commands to /dev/null; these now issue an expected I/O error. This does not cause failures. NEWS, TODO: - Update. (cherry picked from commit 9011fa933552e483dab460f7dd1593d64e059d94)
2020-05-16 14:04:35 +00:00
input/output errors. This allows scripts to check for a nonzero exit
status on the 'print', 'printf' and 'echo' builtins and prevent possible
infinite loops if SIGPIPE is ignored.
Add bin/shtests, convenient wrapper for regression tests This new wrapper script sets up the correct environment for running the ksh93 regression test suite. It allows running the tests without AST nmake, which is not maintained in this repository. An alternative ksh to test may be passed in the $KSH env var. bin/shtests: - Added. Sets up environment before passing control to 'src/cmd/ksh93/tests/shtest'. Passes on any options given. NEWS: - Updated. (cherry picked from commit 14ced94ed83991687c645a09bd2e45a5c2ffe8dc)
2020-05-14 16:41:26 +00:00
- Add a convenient bin/run_ksh_tests script to the source tree that
sets up the necessary environment and runs the ksh regression tests.
Fix BUG_CASELIT: pattern matching as literal string in 'case' This fixes an undocumented 'case' pattern matching misbehaviour (labelled BUG_CASELIT in modernish) that goes back to the original Bourne shell, but wasn't discovered until 2018. If a pattern doesn't match as a pattern, it's tried again as a literal string. This breaks common validation use cases, such as: n='[0-9]' case $n in ( [0-9] ) echo "$n is a number" ;; esac would output "[0-9] is a number" as the literal string fallback matches the pattern. As this misbehaviour was never documented anywhere (not for Bourne, ksh88, or ksh93), and it was never replicated in other shells (not even in ksh88 clones pdksh and mksh), it is unlikely any scripts rely on it. Of course, a literal string fallback, should it be needed, is trivial to implement correctly without this breakage: case $n in ( [0-9] | "[0-9]") echo "$n is a number or the number pattern" ;; esac src/cmd/ksh93/sh/xec.c: - Remove trim_eq() function responsible for implementing the misbehaviour described above. NEWS: - Added. Document this bugfix. Ref.: - The problem: thread starting at https://www.mail-archive.com/austin-group-l@opengroup.org/msg02127.html - The solution, thanks to George Koehler: comments/commits in https://github.com/att/ast/issues/476 - Modernish BUG_CASELIT bug test & documentation: https://github.com/modernish/modernish/commit/b2024ae3 (cherry picked from commit 8d6c8ce69884767a160c1e20049e77bdd849c248 with some extra edits to NEWS to upate the info for this reboot)
2020-06-11 15:14:31 +00:00
2020-05-13:
- Fix BUG_CASELIT: an undocumented 'case' pattern matching misbehaviour that
goes back to the original Bourne shell, but wasn't discovered until 2018.
If a pattern doesn't match as a pattern, it was tried again as a literal
string. This broke common validation use cases, e.g.:
n='[0-9]'
case $n in
( [0-9] ) echo "$n is a number" ;;
esac
would output "[0-9] is a number" as the literal string fallback matches the
pattern. As this misbehaviour was never documented anywhere (not for Bourne,
ksh88, or ksh93), and it was never replicated in other shells (not even in
ksh88 clones pdksh and mksh), it is unlikely any scripts rely on it.
Of course, a literal string fallback, should it be needed, is trivial to
implement correctly without this breakage:
case $n in
( [0-9] | "[0-9]") echo "$n is a number or the number pattern" ;;
esac
Fix BUG_REDIRIO ksh used to redirect standard output by default when no file descriptor was specified with the rarely used '<>' reading/writing redirection operator. It now redirects standard input by default, as POSIX specifies and as all other POSIX shells do. To redirect standard output for reading and writing, you now need '1<>'. Ref.: https://github.com/att/ast/issues/75 http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_07 (cherry picked from commit 29afc16c47824fc79ed092ae7704c525b1db6a0a)
2020-05-13 14:00:33 +00:00
Ref.: https://github.com/att/ast/issues/476
- Fix BUG_REDIRIO: ksh used to redirect standard output by default when no
file descriptor was specified with the rarely used '<>' reading/writing
redirection operator. It now redirects standard input by default, as POSIX
specifies and as all other POSIX shells do. To redirect standard output
for reading and writing, you now need '1<>'.
Ref.: https://github.com/att/ast/issues/75
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_07
Reference in a new issue Copy permalink