src/cmd/ksh93/tests/pty.sh:
- init: Remove superfluous lineno=$LINENO assignments. They aren't
needed if we avoid alias expansion on the err_exit function call.
- In the test "vi mode file name completion", append the main
shell's PID to /tmp/fakehome to make a slightly less insecure
temporary directory name. Unfortunately we cannot use $tmp as
that uses $TMPDIR which may cause a false pass. (re: 4cecde1d)
Apparently, on FreeBSD, the stty command does not work correctly
for setting 'erase' or 'kill' on a pty pseudoterminal. I've no
idea whether the bug is in FreeBSD stty or in AST pty, but in any
case, a workaround is needed for the time being.
src/cmd/ksh93/tests/pty.sh:
- Save terminal state on init; set a trap to restore it on exit.
- Issue 'stty erase ^H kill ^X' on the real terminal before
entering pty pseudoterminals.
Resolves#44.
For unknown reasons, the test for a memory leak in 'read -C stat
<<< "$data"' can show an intermittent minor variation in memory
usage when run with shcomp on certain versions of macOS.
The reported variations are 48 bytes or 80 bytes. This is too small
to be the result of an actual memory leak in the tested command;
it is repeated 500 times so that any real leak should show a
difference of at least 500 bytes.
src/cmd/ksh93/tests/leaks.sh:
- Add a tolerance of 128 bytes to get rid of the false failure.
Fixes#70 (hopefully).
Type names that start with a lowercase 'a' cause an error when used:
$ typeset -T al=(typeset bar)
$ al foo=(bar=testset)
/usr/bin/ksh: al: : invalid variable name
The error occurs because when the parser checks for the alias
builtin (to set 'assignment' to two instead of one), only the first
letter of 'argp->argval' is checked (rather than the entire
string). This was fixed in ksh93v- by comparing argp->argval
against "alias", but in ksh93u+m the check can simply be removed
because it is only run when a builtin has the BLT_DCL flag. As of
04b9171, the alias builtin does not have that flag.
src/cmd/ksh93/sh/parse.c:
- Remove the bugged check for the alias builtin.
src/cmd/ksh93/tests/types.sh:
- Add a regression test for type names starting with a lowercase 'a'.
This fixes an annoying issue in the shell's quoting algorithm
(used for xtrace (set -x), printf %q, and other things) for UTF-8
locales, that caused it to encode perfectly printable UTF-8
characters unnecessarily and inconsistently. For example:
$ (set -x; : 'aeu aéu')
+ : $'aeu a\u[e9]u'
$ (set -x; : 'aéu aeu')
+ : 'aéu aeu'
$ (set -x; : '正常終了 aeu')
+ : '正常終了 aeu'
$ (set -x; : 'aeu 正常終了')
+ : $'aeu \u[6b63]\u[5e38]\u[7d42]\u[4e86]'
This issue was originally reported by lijo george in May 2017:
https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01958.html
src/cmd/ksh93/sh/string.c:
- Add is_invisible() function that returns true if a character is a
Unicode invisible (non-graph) character, excluding ASCII space.
Ref.: https://unicode.org/charts/PDF/U2000.pdf
- Use a fallback in is_invisible() if we cannot use the system's
iswprint(3); this is the case for the ksh C.UTF-8 locale if the
OS doesn't support that. Fall back to a hardcoded blacklist of
invisible and control characters and put up with not encoding
nonexistent characters into \u[xxxx] escapes.
Ref.: https://unicode.org/charts/PDF/U2000.pdf
- When deciding whether to switch to $'...' quoting mode (state=2),
use is_invisible() instead of testing for ASCII 0-127 range.
- In $'...' quoting mode, use is_invisible() to decide whether to
encode wide characters into \u[xxxx] escapes.
src/cmd/ksh93/tests/builtins.sh:
- Add regression tests for shellquoting Arabic, Japanese and Latin
UTF-8 characters, to be run only in a UTF-8 locale. The Arabic
sample text[*] contains a couple of direction markers that are
expected to be encoded into \u[xxxx] escapes.
[*] source: https://r12a.github.io/scripts/tutorial/summaries/arabic
This bugfix was backported from ksh93v- 2013-10-10-alpha.
src/cmd/ksh93/sh/parse: item():
- The done label is placed after the 'inout' call for handling I/O
redirections. This causes the command below to produce a syntax
error because the '>' is not handled as a redirection operator
after 'goto done':
$ ((1+2)) > /dev/null
/usr/bin/ksh: syntax error: `>' unexpected
Moving the done label fixes the syntax error as 'inout' is now
called to handle the redirection operator.
src/cmd/ksh93/tests/arith.sh:
- Add a simple regression test.
There is a bug in path_alias() that may cause a memory leak when
clearing the hash table while setting/restoring PATH.
This applies a fix from Siteshwar Vashist:
https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01945.html
Note that, contrary to Siteshwar's analysis linked above, this bug
has nothing directly to do with subshells, forked or otherwise; it
can also be reproduced by temporarily setting PATH for a command,
for example, 'PATH=/dev/null true', and then doing a PATH search.
Modified analysis:
ksh maintains the value of PATH as a linked list. When a local
scope for PATH is created (e.g. in a virtual subshell or when doing
something like PATH=/foo/bar command ...), ksh duplicates PATH by
increasing the refcount for every element in the linked list by
calling the path_dup() and path_alias() functions. However, when
the state of PATH is restored, this refcount is not decreased. Next
time when PATH is reset to a new value, ksh calls the path_delete()
function to delete the linked list that stored the older path. But
the path_delete() function does not free elements whose refcount is
greater than 1, causing a memory leak.
src/cmd/ksh93/sh/path.c: path_alias():
- Decrease refcount and free old item if needed.
(The 'old' variable was already introduced in 99065353, but
its value was never used there; this fixes that as well.)
src/cmd/ksh93/tests/leaks.sh:
- Add regression test. With the bug, setting/restoring PATH
(which clears the hash table) and doing a PATH search 16 times
causes about 1.5 KiB of memory to be leaked.
There is a bug in print_scan() function that may cause ksh to crash
while listing indexed arrays. The crash happens in nv_search() when
called from print_scan().
This applies a fix from Siteshwar Vashist:
https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01944.html
src/cmd/ksh93/bltins/typeset.c:
- Call nv_scan() without the NV_IARRAY flag, even for a null scan.
src/cmd/ksh93/tests/arrays.sh:
- Add regression test for 'typeset -a' crash and check output.
There is a bug in sh_eval() that may cause ksh to crash due to a
double free() after sourcing multiple files with '.' or 'source'
if a longjmp is triggered, e.g. by a syntax error.
This applies a fix from Siteshwar Vashist:
https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01943.html
src/cmd/ksh93/sh/xec.c: sh_eval():
- Zero file descriptor io_save after closing it. This prevents a
double free() after returning from a longjmp.
src/cmd/ksh93/tests/basic.sh:
- Add reproducer as regression test.
This commit changes the behavior of four date formats accepted
by 'printf %()T' because the old behavior is not compatible with
modern implementations of date(1):
- %k and %l now return a blank-padded hour, the former based on a
24-hour clock and the latter a 12-hour clock (these are common
extensions present on Linux and *BSD).
- %f now returns a date with the format '%Y.%m.%d-%H:%M:%S'
(BusyBox extension).
- %q now returns the quarter of the current year (GNU extension).
src/cmd/ksh93/data/builtins.c:
- Copy the date format documentation from date in libcmd to
the printf man page (for documenting 'printf %T').
src/cmd/ksh93/tests/builtins.sh:
- Add four regression tests for the changed date formats.
src/cmd/ksh93/sh.1:
- Remove inaccurate information about the date formats accepted by
printf %T'. The KornShell uses a custom version of strftime(3)
that isn't guaranteed to accepts the same formats as the native
strftime function.
src/lib/libast/tm/tmxfmt.c:
- Change the behavior of %f, %k, %l and %q to the common behavior.
%k and %l are implemented as aliases to %_H and %_I to avoid
duplicating code.
src/lib/libcmd/date.c:
- Update the documentation for the AST date command since it is
also affected by the changes to 'printf %T'.
Fixes#62
Associative arrays weren't being properly freed from memory, which
was causing a memory leak.
This commit incorporates a patch and reproducer/regress test from:
https://www.mail-archive.com/ast-users@lists.research.att.com/msg01016.html
src/cmd/ksh93/sh/name.c:
- Properly free associative arrays from memory in nv_delete().
src/cmd/ksh93/tests/leaks.sh:
- Add regression test.
'ps' does not always give reliable results; on macOS, 'ps' appears
to produce nondeterministic (i.e. randomly varying) results for
'vsz' and 'rss', making it unusable for memory leak tests. See:
https://github.com/ksh93/ksh/pull/64#issuecomment-655094931
and further comments.
So let's compile in the vmstate builtin so that we can make sure to
measure things properly. It also reports bytes instead of 1024-byte
blocks, so smaller leaks can be detected.
To be decided: whether or not to disable the vmstate builtin for
release builds in order to save about 12K in the ksh binary.
src/cmd/ksh93/data/builtins.c:
- Add vmstate to the list of builtins that are compiled in.
src/cmd/ksh93/tests/leaks.sh:
- getmem(): get size using: vmstate --format='%(busy_size)u'
(Using busy_size instead of size seems to make more sense as it
excludes freed blocks. See vmstate --man)
- Introduce a $unit variable for reporting leaks and set it to
'bytes'; this makes it easier to change the unit in future.
- Since the tests are now more sensitive, initialise all variables
before use to avoid false leak detections.
- The last test seemed to need a few more 'read' invocations in
order to get memory usage to a steady state before the test.
If the processing of a multibyte character was interrupted in UTF-8
locales, e.g. by reading just one byte of a two-byte character 'ü'
(\303\274) with a command like:
print -nr $'\303\274' | read -n1 g
then the shellquoting algorithm was corrupted in such a way that
the final quote in simple single-quoted string was missing. This
bug may have had other, as yet undiscovered, effects as well. The
problem was with corrupted multibyte character processing and not
with the shell-quoting routine sh_fmtq() itself.
Full trace and discussion at: https://github.com/ksh93/ksh/issues/5
(which is also an attempt to begin to understand the esoteric
workings of the libast mb* macros that process UTF-8 characters).
src/lib/libast/comp/setlocale.c: utf8_mbtowc():
- If called from the mbinit() macro (i.e. if both pointer
parameters are null), reset the global multibyte character
synchronisation state variable. This fixes the problem with
interrupted processing leaving an inconsistent state, provided
that mbinit() is called before processing multibyte characters
(which it is, in most (?) places that do this). Before this fix,
calling mbinit() in UTF-8 locales was a no-op.
src/cmd/ksh93/sh/string.c: sh_fmtq():
- Call mbinit() before potentially processing multibyte characters.
Testing suggests that this could be superfluous, but at worst,
it's harmless; better be sure.
src/cmd/ksh93/tests/builtins.sh:
- Add regression test for shellquoting with 'printf %q' after
interrupting the processing of a multibyte characeter with
'read -n1'. This test only fails in a UTF-8 locale, e.g. when
running: bin/shtests -u builtins SHELL=/buggy/ksh-2012-08-01
Fixes#5.
src/cmd/ksh93/data/signals.c:
- SIGINFO was absent from the table of signals, which caused
commands like 'kill -INFO $$' to fail even on platforms with
SIGINFO (such as macOS and FreeBSD). Fix that by adding
it to the signal table.
src/cmd/ksh93/tests/signal.sh:
- Add a regression tests for using SIGINFO with the kill builtin.
The test will only be run if the external kill command supports
SIGINFO.
This gets rid of repetitive code in test scripts to create their
own temporary directories. Instead, shtests exports a $tmp to each
test script that is a subdirectory of its own temporary directory.
This has the advantage of having all test script temporary
directories in one hierarchy. Along with a new option to keep
temporary files, this makes it easy to inspect them if wanted.
This does make the test scripts less self-contained as they now
depend on a temporary directory being exported as $tmp. But they
already depended on $SHELL being the shell to test, so they already
were not quite self-contained.
src/cmd/ksh93/tests/shtests:
- Add -k/--keep option to keep temporary directory. Make the EXIT
trap report its location instead of deleting it.
- For each test, create a subdirectory of $tmp (named after the
test script plus the tested locale or 'shcomp') and export that
subdirectory to the test script as its own $tmp.
- If -k is not given, delete each script's temporary files
immediately after running it to minimise disk usage.
src/cmd/ksh93/tests/*.sh:
- Don't make own temp directory.
- Refuse to run if $tmp is not set.
- Miscellaneous tweaks.
src/cmd/ksh93/tests/pty.sh:
- Fix race condition in the test "raw Bourne mode literal tab
characters with wide characters enabled" by adding 'd 10' to add
a 10-millisecond delay before every write. Thanks to @JohnoKing:
https://github.com/ksh93/ksh/pull/57#issuecomment-653617531
- Fix locale for test "raw Bourne mode backslash handling" (should
be UTF-8, not UTF8) (re: a0dcdeea).
- Add a few more dummy # err_exit # comments to allow shtests to
count the number of tests.
This commit fixes the following bugs in the 'vi' editing mode
backslash escape feature. Ref.: Bolsky & Korn (1995), p. 113, which
states for \: "Similar to Control+V [...] except that it escapes
only the next Erase or Kill charactrer".
1. The vi mode now only escapes the next character if the last
character input was a backslash, fixing the bug demonstrated at:
https://asciinema.org/a/E3Rq3et07MMQG5BaF7vkXQTg0
2. Escaping backslashes are now disabled in vi.c if the vi mode is
disabled (note that vi.c handles raw editing mode in UTF-8
locales). This makes the behavior of the raw editing mode
consistent in C/POSIX and UTF-8 locales.
3. An odd interaction with Backspace when the character prior to a
separate buffer entered with Shift-C was a backslash has been
fixed. Demonstration at: https://asciinema.org/a/314833
^? will no longer be output repeatedly when attempting to erase
a separate buffer with a Backspace, although, to be consistent
with vi(1), you still cannot backspace past it before escaping
out of it. Ref.:
https://github.com/ksh93/ksh/issues/56#issuecomment-653586994
src/cmd/ksh93/edit/vi.c:
- Prevent a backslash from escaping the next input if the previous
input wasn't a backslash. This is done by unsetting a variable
named backslash if a backslash escaped a character. backslash is
set to the result of c == '\\' when the user enters a new
character.
- Disable escaping backslashes in the raw editing mode because
it should not be enabled there.
src/cmd/ksh93/tests/pty.sh:
- Add some tests for how ksh handles backslashes in each
editing mode to test for the bugs fixed by this commit.
Fixes#56.
ksh crashed if it encountered a .paths directory in any of the
directories in $PATH.
Ref: https://bugs.launchpad.net/ubuntu/+source/ksh/+bug/1534855
src/cmd/ksh93/sh/path.c: path_chkpaths():
- Refuse to read .paths if it's not a regular file
or a symlink to a regular file.
Regular expressions that combine a repetition expression with
a parenthesized sub-expression throw a garbled syntax error:
$ [[ AATAAT =~ (AAT){2} ]]
ksh: syntax error: `~(E)(AAT){2} ]]
:'%Cred%h%Creseksh: syntax error: `~(E)(AAT){2} ]]
:'%Cred%h%Creseksh: syntax' unexpected
The syntax error occurs because ksh is not fully
accounting for '=~' when it runs into a curly bracket.
This fix disables the syntax error when the operator
is '=~' and adds handling for '(str){x}' (to allow for
more than one sub-expression). This bugfix and the
regression tests for it were backported from ksh93v-
2014-12-24-beta.
src/cmd/ksh93/sh/lex.c:
- Do not trigger a syntax error for '{x}' when the operator
is '=~' and add handling for multiple parentheses when
combined with '{x}'.
src/cmd/ksh93/tests/bracket.sh:
- Add two tests from ksh93v- to test sub-expressions
combined with the '{x}' quantifier.
src/cmd/ksh93/tests/leaks.sh:
- This script was never actually running the regression
tests because 'vmstate' isn't available as a builtin.
While this can be fixed by adding vmstate to the builtin
table, that has the downside of increasing the binary
size of ksh. This commit replaces all usage of 'vmstate'
with 'ps' and 'awk' as a different way to measure
memory usage. The memory leaks regression tests are now
always run.
- Rename old $n to $N due to new $n interfering with the old
regression test.
- Add before and after results for the number of 1024-byte
blocks leaked in each test.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
File descriptors that are too far out of range will cause the
read builtin to crash. The following example will generate
two crashes:
$ ksh -c 'read -u 2000000' || ksh -c 'read -u-2000000'
The fix is to error out when the given file descriptor is out
of range. This bugfix is from Tomas Klacko, although it was
modified to use 'sh_iovalidfd' and reject numbers greater
than 'INT_MAX':
https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01912.html
The question about 'shp->fdstatus[-1]' only applies to ksh93v-
(ksh93u+ doesn't have any references to 'shp->fdstatus[-1]').
src/cmd/ksh93/bltins/read.c:
- File descriptors that are out of range should be rejected
with an error message (like invalid file descriptors that
are in range). The seemingly redundant check for negative
numbers is there because out of range negative numbers also
cause memory faults despite the later 'fd<0' check.
src/cmd/ksh93/tests/io.sh:
- Add three tests for attempting 'read -u' on various invalid
file descriptor numbers.
The regression test failed on systems where 'chmod' exists at more
than one location, e.g. Slackware where it's at both /bin/chmod and
/usr/bin/chmod.
src/cmd/ksh93/tests/builtins.sh: 'whence -a'/tracked aliases test:
- In the expected value, use modified 'whence -a -p chmod' output
to get all of the paths to chmod.
- On failure, report both expected and actual values.
* Fix the readonly builtin's scope in functions
This bug was first reported at https://github.com/att/ast/issues/881
'tdata.sh->prefix' is only set to the correct value when
'b_readonly' is called as 'export', which breaks 'readonly' in
functions because the correct scope isn't set. As a result, the
following example will only print a newline:
$ function show_bar { readonly foo=bar; echo $foo; }; show_bar
The fix is to move the required code out of the if statement for
'export', as it needs to be run for 'readonly' as well. This bugfix
is from https://github.com/att/ast/pull/906
src/cmd/ksh93/bltins/typeset.c:
- Set 'tdata.sh->prefix' to the correct value, otherwise 'readonly'
uses the wrong scope.
src/cmd/ksh93/tests/builtins.sh:
- Add the regression test from ksh2020, modified to run in a
subshell.
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/sh.1:
- Add documentation of 'readonly' vs. 'typeset -r' difference:
'readonly' does not create a function-local scope.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
The following set of commands ends with a memory fault under
certain circumstances because ksh attempts to free memory
twice, causing memory corruption:
$ testarray=(1 2)
$ compound testarray
$ unset testarray
$ eval testarray=
The fix is to make sure 'np->nvfun' is a valid pointer before
attempting to free memory in 'put_tree'. This patch is from
OpenSUSE: https://build.opensuse.org/package/view_file/shells/ksh/ksh93-nvtree-free.dif?expand=1
src/cmd/ksh93/sh/nvtree.c:
- Do not try to free memory when 'np->nvfun' and 'val'
are false.
src/cmd/ksh93/tests/comvar.sh:
- Add a regression test for the double free problem. The
reproducer must be run from an executable script
with 'ksh -c'.
Variables created with 'typeset -RF' were being treated as
short integers, even though they are actually floating point
values. As a result the following example will cause a crash:
$ typeset -RF foo=1
$ test "$foo"
This is fixed by checking for 'NV_DOUBLE' with 'nv_isattr',
which prevents ksh from treating floating point values as
short integers due to '== NV_INT16P' excluding 'NV_DOUBLE'.
This bugfix was backported from ksh93v- 2013-10-10-alpha.
src/cmd/ksh93/sh/array.c,
src/cmd/ksh93/sh/name.c,
src/cmd/ksh93/sh/nvdisc:
- Avoid treating floating point values as short integers by
checking for 'NV_DOUBLE' with 'nv_isattr'.
src/cmd/ksh93/tests/types.sh:
- Add a regression test for the 'typeset -RF' crash. The
crash cannot be replicated if 'typeset -RF' sets 'foo'
to zero.
ksh, even non-interactive, loads /etc/ksh.kshrc by default. On
some systems this can be a problem, e.g. OpenBSD, which installs a
default /etc/ksh.kshrc which is designed for its version of pdksh.
Quoth sh.1:
On systems that support a system wide /etc/ksh.kshrc
initialization file, if the filename generated by the expansion
of ENV begins with /./ or ././ the system wide initialization
file will not be executed.
src/cmd/ksh93/tests/shtests,
src/cmd/ksh93/tests/builtins.sh,
src/cmd/ksh93/tests/options.sh,
src/cmd/ksh93/tests/pty.sh:
- Instead of emptying or unsetting ENV, ensure it is exported with
a default value of /./dev/null so we skip loading the system-wide
profile and load an empty user profile.
- Where a specific ENV path was required for the tests, prefix it
with '/.' so it starts with '/./'.
The cd builtin was removing '.' from directory names when combined
with a preceding '../', which caused commands like 'cd ../.local'
to become 'cd ../local'. This patch fixes the problem by limiting
the extra handling to leading '..'. The bugfix comes from ksh93v-
2013-10-10-alpha, although this version is a shortened patch from
Solaris (as ksh93v- refactored a decent amount of the code for the
cd builtin).
src/cmd/ksh93/bltins/cd_pwd.c:
- cd should only check for leading '..', as trying to handle a lone
'.' only causes problems.
src/cmd/ksh93/tests/builtins.sh:
- Add a regression test for this problem based on the test present in
ksh93v- 2013-10-10-alpha.
Patch from Solaris:
https://github.com/oracle/solaris-userland/blob/860d27f/components/ksh93/patches/270-23319761.patch
Apparently some systems are still configured to use GMT instead of
UTC after all. This included our own GitHub CI runner config.
Oops. This made the previous commit fail to pass the CI test run.
We can't win this one, it's got to be either one or the other.
UTC is the international standard on which civil time is based.
GMT is often taken as synonymous for UTC, but in navigation,
it can differ from UTC by up to 0.9 seconds. Ref.:
https://en.wikipedia.org/w/index.php?title=Greenwich_Mean_Time&oldid=963422787
The more ambiguous term should not be the first preference.
src/cmd/ksh93/tests/builtins.sh:
- Before checking 'printf %T now' output against 'date' output,
change any ' GMT ' in the latter to ' UTC '.
.github/workflows/ci.yml:
- Set time zone to UTC, not GMT.
"UTC" is the modern name for what used to be "GMT", but ksh still
preferred GMT. On systems configured to use the UTC time zone, this
caused a 'printf %T' regression test failure in tests/builtins.sh
as the external 'data' utility will prefer UTC these days.
src/lib/libast/tm/tmdata.c:
- Reorder the name alternatives for UTC/GMT so that UTC is
the first preference.
src/cmd/ksh93/tests/builtins.sh:
- Report expected and actual values on 'printf %T' failure.
Related: #6
When neither '-o emacs' nor '-o vi' is active, there were a couple
of bugs with entering tab characters:
1. Tab completion was erroneously left active. The cause of this
was that raw Bourne edit mode is handled by ed_viread() in vi.c
on shells with wide character support, instead of the default
ed_read() in edit.c, and the former failed to check if vi mode
is active when processing tab characters.
2. When entering literal tab characters, the cursor was moved to
the right only one character, instead of the amount of
characters corresponding to the tab.
src/cmd/ksh93/edit/vi.c: getline():
- Before processing '\t' (tab) for command completion, check that
the 'vi' shell option (SH_VI) is active.
src/cmd/ksh93/edit/edit.c: ed_virt_to_phys():
- When translating literal tabs to on-terminal spaces and when
recalculating the cursor position, remove erroneous checks for
SH_VI; this is also needed in raw Bourne mode. According to my
own testing, this has no effect on emacs mode (knock on wood).
src/cmd/ksh93/tests/pty.sh:
- Add two regression tests. An odd race condition reveals itself in
either pty or in ksh's raw/Bourne edit mode; see comment in test.
Effect is we have to expect either literal tabs or tabs expanded
to spaces, until that is tracked down and fixed.
Fixes#43.
This commit fixes the bug reported in:
https://github.com/att/ast/issues/682
The following sequence fails in vi mode because ksh looks in the
wrong part of the 'virtual' buffer:
$ touch ~/testfile
$ ls ~/test<tab>
The fix is to change 'virtual[i]' to 'virtual[last_virt]' in the
bugged section of code. The other changes are to make sure listing
files in a directory with something like 'ls /etc/<tab>' calls the
code for Ctrl+L to preserve 'ls /etc/' rather than try (and fail)
to complete the directory name, producing 'ls /etc\n/'. This bugfix
was backported from ksh93v- 2013-10-10-alpha.
src/cmd/ksh93/edit/vi.c
- Backport the bugfix from ksh93v- 2013-10-10-alpha for this
problem.
src/cmd/ksh93/tests/pty.sh
- Add a regression test for this issue using pty, adjusted slightly
for a fake home directory in /tmp.
Somewhat notable changes in this commit:
- The 'set +r' bugfix (re: 74b41621) is now documented in the
changelog.
- Missing options have been added to the synopsis section of the
ksh man page.
- The minor formatting fix from https://github.com/ksh-community/ksh/pull/5
has been applied to the ksh man page.
- A few fixes from https://github.com/att/ast/commit/5e747cfb
have been applied to the ksh man page.
- The man page fixes from https://github.com/att/ast/pull/353
have been applied, being:
- An addition to document the behavior of 'set -H'.
- A fix for the cd section appending rksh93.
- A fix for some options being indented too far.
- Removal of a duplicate section documenting '-D'.
- Reordering the options for 'set' in alphabetical order.
- A minor fix for the documentation of 'ksh -i'.
src/cmd/ksh93/tests/bracket.sh:
- Disable tests for [[ -N ... ]] (test -N ...), because it is
expected to break on systems where $TMPDIR (or even the entire
root file system) is mounted with noatime for better performance.
Ref.: https://opensource.com/article/20/6/linux-noatime
(It also needs annoyingly long sleep times on older systems with
a 1-second timestamp granularity.)
Testing the behaviour of an external editor, even the standard one,
is outside the scope of the ksh regression tests.
src/cmd/ksh93/tests/pty.sh:
- Disable a test that invoked vi(1) and that failed, either
intermittently or consistently, on too many systems because
whatever vi(1) is installed locally doesn't write the string
"/tmp/" exactly as and/or when expected.
The output format is now identical to mksh's except for
the locale-dependent radix point ('.' or ',').
src/cmd/ksh93/bltins/misc.c:
- Output format tweak: pad seconds with initial zero if < 10.
- Use "too many operands" (e_toomanyops) error msg from 3ba4900e
if there are operands, instead of "bad syntax" (e_badsyntax).
- Consolidate repetitive calculating and printing code
into print_times().
- Get rid of some excessive variables.
src/cmd/ksh93/tests/builtins.sh:
- Update regression tests to match the above.
src/cmd/ksh93/data/builtins.c:
- Update sh_opttimes[] version string.
When a nonexistent test script was given as an argument to
shtests, this was not counted as an error and shtests exited
successfully (with status 0).
src/cmd/ksh93/tests/shtests:
- Increase total error count if a test script is not found.
The code for handling process substitution with redirection was
never being run because IORAW is usually set when IOPROCSUB is
set. This commit fixes the problem by moving the required code
out of the !IORAW if statement. The following command now prints
'good' instead of writing 'ok' to a bizzare file:
$ ksh -c 'echo ok > >(sed s/ok/good/); wait'
good
This commit also fixes a bug that caused the process ID of the
asynchronous process to print when the shell was in interactive
mode. The following command no longer prints a process ID,
behaving like in Bash and zsh:
$ echo >(true)
/dev/fd/5
src/cmd/ksh93/sh/args.c:
- Temporarily turn off the interactive state while in a process
substitution to prevent the shell from printing the PID of
the asynchronous process.
src/cmd/ksh93/sh/io.c:
- Move the code for process substitution with redirection into
a separate if statement.
src/cmd/ksh93/tests/io.sh:
- Add two tests for both process substitution bugs fixed by this
commit.
src/cmd/ksh93/tests/shtests:
- Update shtests with a patch from Martijn Dekker to use
pretty-printing for the output from the times builtin (if it
is available).
Fixes#2
This commit fixes the bug described in att/ast#32. The fix and
following explanation is from att/ast#467:
While copying variables from function's local scope to a new scope,
variable attributes were not copied. Such variables were not marked
to be exported in the new function. For e.g.
function f2 { env | grep -i "^foo"; }
function f1 { env | grep -i "^foo"; f2; }
foo=bar f1
prints 'foo=bar' only once, but it should print be twice.
src/cmd/ksh93/sh/xec.c:
- When variables from the local scope of a function are copied into
the scope of a nested function, the attributes of the variables
need to be copied as well.
src/cmd/ksh93/tests/functions.sh:
- Add regression tests from ksh2020 to check environment variables
passed to functions.
src/cmd/ksh93/tests/builtins.sh:
- The output of 'printf %T now' and the external 'date'
command aren't guaranteed to be the same unless $LC_ALL
is set to 'C'. Set LC_ALL in these command substitutions
to fix a spurious test failure on Linux.
Ksh was trying to use the 'pw' variable as a valid pointer even
when it was NULL. This is fixed by doing the error check for
'pw' before doing anything else in 'job_kill'.
This bugfix is from Red Hat:
44e0a643a9/f/SOURCES/ksh-20130214-fixkill.patchFixes#34
After making PATH readonly in a virtual subshell (without otherwise
changing it, so the subshell is never forked), then the main shell
would erroneously fork into a background process immediately after
leaving the virtual subshell. This was caused by a bug in the
forking workaround that prevents changes in PATH in a virtual
subshell from clearing the parent shell's hash table.
src/cmd/ksh93/sh/name.c: nv_putval():
- If we're either setting or restoring PATH, do an additional check
for the NV_RDONLY flag, which means the function was told to
ignore the variable's readonly state. It is told to ignore that
when restoring the parent shell state after exiting a virtual
subshell. If we don't fork then, we don't fork the parent shell.
src/cmd/ksh93/tests/subshell.sh:
- Add regression test verifying that no forking happens when making
PATH readonly in a subshell.
Fixes#30.
src/cmd/ksh93/data/variables.c:
- Running 'unset .sh.lineno' creates a memory fault, so fix that
by giving it the NV_NOFREE attribute. This crash was happening
because ${.sh.lineno} is an integer that cannot be freed from
memory with free(3).
src/cmd/ksh93/sh/init.c:
- Tell _nv_unset to ignore NV_RDONLY when $RANDOM and $LINENO are
restored from the subshell scope. This is required to fully
restore the original state of these variables after a virtual
subshell finishes.
src/cmd/ksh93/bltins/typeset.c,
src/cmd/ksh93/sh/subshell.c:
- Disabled some optimizations for two instances of 'sh_assignok' to
fix 'readonly' in virtual subshells and '(unset .sh.level)' in
nested functions. This fixes the following variables when
'(readonly $varname); enum varname=' is run:
$_
${.sh.name}
${.sh.subscript}
${.sh.level}
The optimization in question prevents sh_assignok from saving the
original state of these variables by making the sh_assignok call
a no-op. Ksh needs the original state of a variable for it to be
properly restored after a virtual subshell has run, otherwise ksh
will simply carry over any new flags (being NV_RDONLY in this case)
from the subshell into the main shell.
src/cmd/ksh93/tests/variables.sh:
- Add regression tests from Martijn Dekker for setting special
variables as readonly in virtual subshells and for unsetting
special variables in general.
Fixes#4
'whence -a' bases the path for tracked aliases on the user's
current working directory if an enabled ksh builtin of the same
name is also available. The following example will claim 'cat'
is in the user's current working directory:
$ whence -a cat
cat is a tracked alias for /usr/bin/cat
$ builtin cat
$ whence -a cat
cat is a shell builtin
cat is /usr/bin/cat
cat is a tracked alias for /current/working/directory/cat
This patch from ksh2020 fixes this problem by properly saving the
path of the tracked alias for use with 'whence -a', since
'path_pwd' (as implied by the function's name) only gets the users
current working directory, not the location of tracked aliases.
Ref.: https://github.com/att/ast/issues/1049
This bug was originally reported by David Morano about two decades
ago to the AST team: https://github.com/att/ast/issues/954
src/cmd/ksh93/bltins/whence.c:
- Print the actual path of a tracked alias, path_pwd doesn't
have this functionality.
src/cmd/ksh93/include/name.h:
- Add 'pathcomp' for saving the value of tracked aliases.
src/cmd/ksh93/sh/path.c:
- Save the value of tracked aliases for use by whence.
src/cmd/ksh93/tests/builtins.sh:
- Add a regression test for using 'whence -a' on tracked
aliases with a builtin equivalent.
Applying the fix for 'unset -f' exposed a crashing bug in lookup()
in sh/nvdisc.c, which is the function for looking up discipline
functions. This is what caused tests/variables.sh to crash.
Ref.: https://github.com/ksh93/ksh/issues/23#issuecomment-645699614
src/cmd/ksh93/sh/nvdisc.c: lookup():
- To avoid segfault, check that the function pointer nq->nvalue.rp
is actually set before checking if nq->nvalue.rp->running==1.
src/cmd/ksh93/sh/xec.c,
src/cmd/ksh93/tests/functions.sh:
- Uncomment the 'unset -f' fix from b7932e87.
Resolves#21 (again).
The fix in sh/xec.c, which was backported from the ksh 93v- beta to
delay the actual removal of a running function that unsets itself,
caused a segfault in the variables.sh regression tests (see #23).
src/cmd/ksh93/sh/xec.c:
- Comment out the backported code pending a correct fix for #21.
Now both types of functions silently fail to unset themselves
(unless they're discipline functions).
src/cmd/ksh93/tests/functions.sh:
- Disable regression tests checking that the function was actually
unset, pending a correct fix for #21.
Resolves: #23Reopens: #21
src/cmd/ksh93/sh/name.c:
- Correct the check for when a function is currently running
to fix a segmentation fault that occurred when a POSIX
function tries to unset itself while it is running.
This bug fix was backported from ksh93v-.
src/cmd/ksh93/sh/xec.c:
- If a function tries to unset itself, unset the function
with '_nv_unset(np, NV_RDONLY)' to fix a silent failure.
This fix was also backported from ksh93v-.
src/cmd/ksh93/tests/functions.sh:
- Add four regression tests for when a function unsets itself.
Resolves#21
Note that shtests simply does a 'grep -c err_exit' and substracts 1
to count the number of regression tests in a test script. Not all
test scripts make temp dirs, so subtracting 2 instead won't do.
src/cmd/ksh93/tests/*.sh:
- Escape the err_exit call in the routine to create a temporary
directory so that it is not counted as a regression test.
That bypasses the alias, so we have to pass $LINENO manually.
Four added tests did not correctly report their line numbers
upon failure and were counted as one, because the err_exit
alias/function pair was called from a shell function.
Note that shtests simply does a 'grep -c err_exit' to count the
number of regression tests in a test script.
src/cmd/ksh93/tests/subshell.sh:
- check_hash_table():
- Take line number as 1st argument.
- Quote a character in err_exit to bypass the alias when calling
it, so we can pass on the argument for the line number. This
also stops this helper function from being counted as a test.
- When calling check_hash_table(), pass $LINENO.
- Add dummy err_exit comments to have the tests counted.
Ksh was not checking for `command` when running a special builtin,
which caused preceding invocation-local variable assignments to
become global. This is the reproducer from the att/ast#72:
$ foo=BUG command eval ':'
$ echo "$foo"
This no longer prints 'BUG', as ksh now makes sure the command builtin
is not running a special builtin before making invocation-local
variable assignments global.
src/cmd/ksh93/sh/xec.c:
- Backport the bugfix for BUG_CMDSPASGN from ksh93v- 2013-10-10-alpha.
src/cmd/ksh93/tests/builtins.sh:
- Add a regression test based on the reproducer in att/ast#72.
This bug was previously reported in att/ast#37.
Ksh ignores `-r` when `read -r -d` is run because when
the bit for `D_FLAG` is set, the bit for `R_FLAG` is unset
as a side effect of setting `D_FLAG`. The following set
of commands fails to print a backslash:
$ printf '\\\000' | read -r -d ''
$ echo $REPLY
The fix for this bug is to set `D_FLAG` with `D_FLAG + 1`,
which prevents `R_FLAG` from being unset. This bugfix
has been backported from ksh93v- 2013-10-10-alpha.
src/cmd/ksh93/bltins/read.c:
- Set `D_FLAG` with `D_FLAG + 1` to prevent the bit for
`R_FLAG` from being unset.
src/cmd/ksh93/tests/builtins.sh:
- Add the regression test for `read -r -d` from ksh93v-.
ksh 93v- beta introduced a regression with nested command
substitutions: backticks nested in $( ) result in misdirected
output. This has never been in 93u+, but since we're often
backporting things, let's avoid backporting this bug. It is also
useful if this shows up when running our bin/shtests against the
actual beta by adding a SHELL=... argument.
Ref.: https://github.com/att/ast/issues/478
src/cmd/ksh93/tests/subshell.sh:
- Add reproducer submitted by the reporter as a regression test.
This bug was originally reported by @lijog in att/ast#7 and has been
reported again in #15. KSH does not save the state of a variable if it
is in a newer scope. This is because of an optimization in sh_assignok
first introduced in ksh93t+ 2010-05-24. Here is the code change in that
version:
return(np);
/* don't bother to save if in newer scope */
- if(!(rp=shp->st.real_fun) || !(dp=rp->sdict))
- dp = sp->var;
- if(np->nvenv && !nv_isattr(np,NV_MINIMAL|NV_EXPORT) && shp->last_root)
- dp = shp->last_root;
- if((mp=nv_search((char*)np,dp,HASH_BUCKET))!=np)
- {
- if(mp || !np->nvfun || np->nvfun->subshell>=sh.subshell)
- return(np);
- }
+ if(sp->var!=shp->var_tree && shp->last_root==shp->var_tree)
+ return(np);
if((ap=nv_arrayptr(np)) && (mp=nv_opensub(np)))
{
This change was originally made to replace a buggier optimization.
However, the current optimization causes variables set in subshells
to wrongly affect the environment outside of the subshell, as the
variable does not get set back to its original value. This patch
simply removes the buggy optimization to fix this problem.
src/cmd/ksh93/sh/subshell.c:
- Remove a buggy optimization that caused variables set in subshells
to affect the environment outside of the subshell.
src/cmd/ksh93/tests/subshell.sh:
- Add a regression test for setting variables in subshells. This
test has to be run from the disk after being created with a here
document because it always returns the expected result when run
directly in the regression test script.
Prior to this bugfix, the following set of commands would
fail to print two double quotes:
IFS=',' read -S a b c <<<'foo,"""title"" data",bar'
echo $b
This fix is from ksh93v- 2013-10-10-alpha, although it has
been revised to use stakputc to put the required double quote
into the buffer for consistency with the ksh93u+ codebase.
src/cmd/ksh93/bltins/read.c:
- When handling nested double quotes, put the required double
quote in read's buffer with stakputc.
src/cmd/ksh93/tests/builtins.sh:
- Add the regression test for `read -S` from ksh93v-.
src/cmd/ksh93/sh.1:
- Fix a minor formatting error to highlight '-S' in the ksh(1)
man page.
The regression this commit fixes was first introduced in ksh93t
2008-07-25. It was previously worked around in 6f0e008c by forking
subshells if any special environment variable is unset.
The reason why this problem doesn't occur in ksh93s+ is because in
that version of ksh sh_assignok never moves nodes, it only clones
them. The second argument doesn't set NV_MOVE, which makes
`sh_assignok(np,0)` is similar to `sh_assignok(np,1)`. In ksh93t and
higher, setting the second argument to zero causes the node to be moved
with NV_MOVE, which causes the discipline function associated with
the variable node to be removed when `np->nvfun` is set to zero (i.e.
NULL). This is why a command like `(unset LC_NUMERIC; LC_NUMERIC=invalid)`
doesn't print a diagnostic, as it looses its discipline function.
This patch fixes the problem by cloning the node with sh_assignok
if it is a special variable with a discipline function. This allows
special variables to work as expected in virtual subshells. The
original workaround has been kept for the $PATH variable only, as
hash tables are still broken in virtual subshells. It has been updated
accordingly to only fork subshells if it detects the variable node
for PATH. I have added two more regression tests for changing the
PATH in subshells to make sure hash tables continue working as
expected with this fix.
src/cmd/ksh93/bltins/typeset.c:
- Only fork virtual subshells if the PATH will be changed. If a
variable is a special variable with a discipline function, it
should be just be cloned, not moved.
src/cmd/ksh93/sh/nvdisc.c:
- Add a comment to clarify that NV_MOVE will delete the discipline
function associated with the node.
src/cmd/ksh93/tests/subshells.sh:
- Add two more regression tests for unsetting the PATH in subshells,
one for if PATH is being pointed to by a nameref. Condense the
hash table tests by moving the main test into a single function.
Test 137(C) was failing on some systems because $TMPDIR was set and
the local vi(1) honours it, so that the expected '/tmp/' string was
never output by vi. For compatibility with vi programs that honour
$TMPDIR and those that always use /tmp, we must export TMPDIR=/tmp.
src/cmd/ksh93/tests/pty.sh:
- Export TMPDIR=/tmp for test 137(C).
Note that this exports TMPDIR to the environment for the
duration of the 'tst' function run because the function was
defined using the ksh 'function tst { ...; }' syntax.
The man page for the builtin command says special builtins cannot
be deleted. This wasn't the case though, running `builtin -d` on
a special builtin was deleting it. As an example, the following
set of commands was ending with 'export: not found':
$ builtin -d export
$ export foo=bar
This commit backports the bugfix from ksh93v- (2014-12-24-beta),
which added an error check to prevent special builtins from being
deleted.
src/cmd/ksh93/sh/nvdisc.c:
- Add an error check to prevent special builtins from being deleted.
src/cmd/ksh93/tests/builtins.sh
- Add a regression test for using `builtin -d` on special builtins.
src/cmd/ksh93/tests/io.sh:
- 16 tests for the 'redirect' builtin used the ((arithmetic
command)) to check the result. This does not tolerate the result
being a non-number, such as the empty string, which may occur on
test failure. So use [[ ... -eq ArithExpression ]] to check these
results instead; it simply returns false, failing gracefully.
This commit converts the redirect='command exec' alias to a regular
'redirect' builtin command that only accepts I/O redirections, which
persist as in 'exec'. This means that:
* 'unlias -a' no longer removes the 'redirect' command;
* users no longer accidentally get logged out of their shells if
they type something intuitive but wrong, like 'redirect ls >file'.
This should not introduce any legitimate change in behaviour. If
someone did accidentally pass non-redirection arguments to
'redirect', unexpected behaviour would occur; this now produces
an 'incorrect syntax' error.
src/cmd/ksh93/bltins/misc.c: b_exec():
- Recognise 'redirect' when parsing options.
- If invoked as 'redirect', produce error if there are arguments.
src/cmd/ksh93/data/aliases.c:
- Remove redirect='command exec' alias.
src/cmd/ksh93/data/builtins.c:
- Update/improve comments re ordering.
- Add 'redirect' builtin entry.
- sh_optexec[]: Abbreviate redirection-related documentation;
refer to redirect(1) instead.
- sh_optredirect[]: Add documentation.
src/cmd/ksh93/include/builtins.h:
- Add SYSREDIR parser ID, renumbering those following it.
- Improve comments.
- Add extern sh_optredirect[].
src/cmd/ksh93/sh.1:
- exec: Abbreviate redirection-related documentation; refer to
'redirect' instead.
- redirect: Add documentation.
src/cmd/ksh93/sh/xec.c:
- Recognise SYSREDIR parser ID in addition to SYSEXEC when
determining whether to make redirections persistent.
src/cmd/ksh93/tests/io.sh:
- To regress-test the new builtin, change most 'command exec' uses
to 'redirect'.
- Add tests verifying the exit behaviour of 'exec', 'command exec',
'redirect' on redirections.
The ksh man page documents that the restricted option cannot be
unset once it is set, which means `set +r` should be invalid.
While this was true for `set +o restricted`, `set +r` was causing
the restricted option to be unset. The fix for this problem comes
from one of Solaris' patches, which adds an error check to prevent
this behavior.
Solaris' patch:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/020-CR6919590.patch
src/cmd/ksh93/sh/args.c:
- Add an error check to stop `set +r` from unsetting the
restricted option.
src/cmd/ksh93/tests/restricted.sh:
- Add two regression tests to make sure the restricted option
cannot be unset.
(cherry picked from commit bef4fee404d8e24b38fce66420c14a39ac4a123e)
Sometimes you have to put ^C on rapid repeat to get out of running
the tests. This is because each test scripts is launch as a child
shell (not a subshell) in the foreground, which thus handle the ^C.
It then exits with a status indicating SIGINT, but the shtest script
wasn't handling this and just kept going.
src/cmd/ksh93/tests/shtests:
- For reasons I don't understand yet, interrupting tests with ^C
tends to make them exit with status 130 (128+2) instead of what
I would expect for ksh93, 258 (256+2). Not a big deal: POSIX
specifies that any exit > 128 signifies a signal in any case,
and 'kill -l' works even on those values. But the checks need
changing to '> 128'.
- Check each result for SIGINT, and issue SIGINT to self if found.
(cherry picked from commit d0dfb37c6c71ac7b157060249125e0959130927d)
This commit replaces the old hash alias with a proper builtin.
I based this builtin off of the code alias uses for handling
`alias -t --`, but with the hack for `--` removed as it has
no use in the new builtin. `alias -t --` will no longer work,
that hack is now gone.
While I was testing this builtin, I found a bug with hash tables
in non-forking subshells. If the hash table of a non-forking
subshell is changed, the parent shell's hash table is also changed.
As an example, running `(hash -r)` was resetting the parent shell's
hash table. The workaround is to force the subshell to fork if the
hash table will be changed.
src/cmd/ksh93/bltins/typeset.c:
- Move the code for hash out of the alias builtin into a dedicated
hash builtin. `alias -t --` is no longer supported.
src/cmd/ksh93/data/aliases.c:
- Remove the old alias for hash from the table of predefined aliases.
src/cmd/ksh93/data/builtins.c:
- Fix the broken entry for the hash builtin and add a man page for
the new builtin.
src/cmd/ksh93/sh.1:
- Replace the entry for the hash alias with a more detailed entry
for the hash builtin.
src/cmd/ksh93/sh/name.c:
- Force non-forking subshells to fork if the PATH is being reset
to workaround a bug with the hash tree.
src/cmd/ksh93/tests/alias.sh:
- Add a regression test for resetting a hash table, then adding
a utility to the refreshed hash table.
src/cmd/ksh93/tests/subshell.sh:
- Add regression tests for changing the hash table in subshells.
(cherry picked from commit d8428a833afe9270b61745ba3d6df355fe1d5499)
Since we now have a shiny new POSIX compliant 'times' builtin,
we might as well use it.
src/cmd/ksh93/tests/shtests:
- Run 'times' at end of test run.
- Skip the pretty-printing until #7 is fixed.
(cherry picked from commit 2c27d9fbc239583004ec70377db98627eea5e294)
A column of whitespace in the NEWS file was removed for consistent
formatting. Most of the spelling errors were found with this
codespell dictionary:
https://github.com/orbitcowboy/codespell_dictionary
(cherry picked from commit 0e36b17abe5609c461a3e4da7041eb0fdf9991b7)
The bug was really that I/O errors in output builtins were
undetectable by any means. Having a >0 exit status is sufficient.
Adding an error message risks making existing ksh scripts noisier,
or even breaking them if they redirect stderr to stdout.
Note to self: in future, implement the minimum change necessary to
fix bugs, nothing more. The fact that I needed to add four extra
2>/dev/null to the regression tests should have been a hint.
src/cmd/ksh93/bltins/print.c,
src/cmd/ksh93/data/msg.c,
src/cmd/ksh93/include/io.h:
- Remove "I/O error" message.
src/cmd/ksh93/tests/builtins.sh:
- Update to check for exit status only.
src/cmd/ksh93/tests/basic.sh,
src/cmd/ksh93/tests/coprocess.sh:
- Revert four new '2>/dev/null' to suppress the error message.
(cherry picked from commit 5e17be24d18455b575b6e98bc631c6935ffc795a)
This commit fixes a bug that caused unalias to return a zero status
when it tries to remove an alias twice. The following set of commands
will no longer end with an error:
$ alias foo=bar
$ unalias foo
$ unalias foo && echo 'Error'
This commit is based on the fix present in ksh2020, but it has been
extended with another bugfix. The initial fix for this problem tried to
remove aliases from the alias tree without accounting for NV_NOFREE. This
caused any attempt to remove a predefined aliases (e.g. `unalias float`)
to trigger an error with free, as all predefined aliases are in read-only
memory. The fix for this problem is to set NV_NOFREE when removing aliases
from the alias tree, but only if the alias is in read-only memory. All
other aliases must be freed from memory to prevent memory leaks.
I'll also note that I am using an `isalias` variable rather than the `type`
enum from ksh2020, as the `VARIABLE` value is never used and was replaced
with a bool called `aliases` in the ksh2020 release. The `isalias` variable
is an int as the ksh93u+ codebase does not use C99 bools.
Previous discussion: https://github.com/att/ast/issues/909
- src/cmd/ksh93/bltins/typeset.c:
Remove aliases from the alias tree by using nv_delete. NV_NOFREE
is only used when it is necessary.
- src/cmd/ksh93/tests/alias.sh:
Add two regression tests for the bugs fixed by this commit.
(cherry picked from commit 16d5ea9b52ba51f9d1bca115ce8f4f18e97abbc4)
Setting the exit status allows build scripts and the like
to actually detect regression test failures.
src/cmd/ksh93/tests/shtests:
- Add counter for total number of errors.
- Report total to stdout at the end.
- Set the exit status to the total number of errors, up to 125
(higher statuses have special meanings).
(cherry picked from commit c0dd80b14d4d316b4e9eff4b1b67d4e47f23a6ba)
src/cmd/ksh93/tests/variables.sh:
- Tolerate a bit more time for the SECONDS verification test.
src/cmd/ksh93/tests/subshell.sh:
- Replace unportable 'head -c 1' by 'dd bs=1 count=1'
- Remove unnecessary uses of 'whence'.
src/cmd/ksh93/tests/builtins.sh:
- Add a regression test for a weirdly specific 'whence' bug exposed
by the aforementioned unneccessary uses of 'whence', which only
shows up on my old Power Mac G5 running Mac OS X 10.3. For all I
know it's a compiler bug, but let's add a more clear failure for
it here, in case that happens anywhere else.
(cherry picked from commit c3898bd1e6e40874845771d33a5b37220ef0b06e)
This reduces a bunch more unnecessarily long sleeps to give
asynchronous processes time to run, etc. (No, we don't need to be
compatible anymore with your cool 1985 Intel 80386DX 16 MHz
battlestation...) Running the test suite is almost tolerable now,
making me much more likely to actually run the regression test
suite and catch my own regressions.
In addition, there are various fixes to make the test suite
compatible with 'set -x' ('set -o xtrace') so that you can now
actually *use* the documented 'bin/shtests -x' option. Recommend
combining with '-p' to avoid tracing everything three times.
I've also added a really useful $PS4 trace prompt to shtests that
traces pretty much everything there is to trace. (It does use
expansions that modify ${.sh.match}, which affected several tests,
particularly in tests/substring.sh; for those we need to set a
temporary simpler $PS4.)
(cherry picked from commit c3a5d47cfe880b526cabb5370ddaced0e8626acd)
The signal.sh regression tests took over a minute and a half to
complete, so with the default three runs (for different locales)
they took about five minutes. Annoying and unnecessary. As ksh
supports floating point shell arithmetic plus a 'sleep' builtin
that supports fractional seconds, this is easy to fix.
src/cmd/ksh93/tests/signal.sh:
- Slash all the 'sleep' arguments and associated factors by a
factor of 10. Since computers are now hundreds or thousands of
times faster than they were in the early 1990s, this should be
plenty safe enough and all the tests should still be valid.
- Change one unportable direct invocation of /bin/sleep to use the
builtin.
(cherry picked from commit 734e5953650e9a62048c45eb9a3b827bb9efcfb5)
This fixes two bugs: issuing the 'exit' command with a value > 256
would cause ksh 93u+ to kill itself with the corresponding signal
(try 'exit 265' to SIGKILL your interactive shell), and, if the
last command of a script exits due to a signal, the shell would
repeat that signal to itself, causing any parent ksh to also be
killed.
Discussion:
https://bugzilla.redhat.com/show_bug.cgi?id=1469624https://rainbow.chard.org/2017/03/21/ksh-deliberately-segfaults-if-the-last-command-in-a-script-crashes/
This commit is loosely based on a patch applied to the 93v- beta
and the abandoned ksh2020, but that patch was incomplete & broken:
$ ksh-2020.0.0 -c 'exit 265'; echo $?
137
Expected: 9. Since the exit was *not* due to a signal, the value
should simply be cropped to the 8 bits supported by the OS.
src/cmd/ksh93/bltins/cflow.c: b_exit():
- For the 'exit' builtin command, bitwise-AND the argument to
'exit' with SH_EXITMASK (8 bits, crop to 0-255) before passing it
on to sh_exit(). This restores the behaviour of <=2011 ksh93
versions and is in line with all other POSIX shells.
It also fixes this bogosity:
$ (exit 265); echo $? # non-forked subshell
265
$ (ulimit -t unlimited; exit 265); echo $? # forked subshell
9
Forked or non-forked should make no difference at all
(see commit message a0e0e29e for why).
src/cmd/ksh93/sh/fault.c: sh_done():
- If the current exit status is equal to the status for the last
signal that was received from a child process, remove the
SH_EXITSIG (9th) bit, so that the shell doesn't kill itself.
- If the shell's last child process exits due to a signal, exit
with a portable 8-bit exit status (128 + signal number). This
avoids the exit status being < 128 by being cropped to 8 bits.
src/cmd/ksh93/tests/signal.sh:
- Add regression test for exit with status > 256.
- Add regression test verifying the shell no longer kills itself.
(cherry picked from commit 98e0fc94393e175ce6adfee390327c320795bf12)
POSIX requires[*] that expanding any unset parameter other than $@
and $* is an error when 'set -u'/'set -o nounset' is active.
However, on ksh93, $! was exempt as well. That is a bug.
[*] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_25
src/cmd/ksh93/sh/macro.c:
- special(): Handle 'set -u' for special parameters if/when it is
about to return NIL. That code path is currently only possible to
reach for "$!", but this is future-proof and will do the right
thing if any other special parameter can ever have no value.
src/cmd/ksh93/tests/options.sh:
- Add and tweak 'set -u' regression tests.
(cherry picked from commit 75cc7a38cafe3a9929e1ed17d8b952babda22a09)
POSIX requires[*] that expanding any unset parameter other than $@
and $* is an error when 'set -u'/'set -o nounset' is active.
However, on ksh93, $1, $2, ... were exempt as well. That is a bug.
[*] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_25
src/cmd/ksh93/sh/macro.c:
- varsub(): Backport code for handling 'set -u' for positional
parameters from the ast 2016-10-01-beta branch.
src/cmd/ksh93/tests/options.sh:
- Add relevant regression tests.
src/cmd/ksh93/sh.1:
- Document that $@ and $* are exempt from 'set -u'.
(cherry picked from commit f954c6be0748c4c38a680a75f27564965fbd328e)
src/cmd/ksh93/tests/options.sh:
- Pipe hang bugfix test: The child shell has no idea how long to
sleep for if we don't export the variable telling it so.
(cherry picked from commit ede0960c4ec84f0f934b17072a892a4e40798e97)
As of this commit, the 'times' command is a POSIX-compliant special
builtin command instead of an alias that doesn't produce the
required output. It displays the accumulated user and system CPU
times, one line with the times used by the shell and another with
those used by all of the shell's child processes.
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_27
This was originally written by Kurtis Rader and is now backported
and tweaked from the abandoned ksh2020 branch. I chose an earlier
and simpler version[*1] that uses times(3), with a precision of
hundredths of seconds, so it outputs the same precision as mksh and
zsh. Rader later wrote another version[*2] that uses getrusage(2),
giving it the same millisecond precision as bash. But that required
adding a feature test and a fallback to the old version, which is
non-trivial in the old INIT/iffe system. This simpler version is
enough to gain POSIX compliance and I think it will do very nicely
in this stable bugfix branch.
[*1] https://github.com/att/ast/pull/1332
[*2] https://github.com/att/ast/commit/038045f6
src/cmd/ksh93/bltins/misc.c
- Add b_times() function for 'times' builtin.
- Note we include <times.h>, not <sys/times.h>, so that we use the
AST feature-tested version with fallback on systems that need it.
src/cmd/ksh93/data/aliases.c:
- Remove times='{ { time;} 2>&1;}' builtin alias.
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/include/builtins.h:
- Add entry for 'times' special builtin.
- Add --help/--man info for same.
src/cmd/ksh93/sh.1:
- Update manual page.
src/cmd/ksh93/tests/builtins.sh:
- Add a couple of simple regression tests.
(cherry picked from commit ebf71e619eb298ec1cf6b81d1828fa7cdf6e9203)
This counter is documented as follows:
"The current depth for subshells and command substitution."
But before this commit, the actual behaviour was that the counter
was reset to zero whenever a subshell forked for any reason: a
pipe, background job, running 'ulimit', redirecting stdout in a
command substitution, and more. This behaviour was:
1. Not consistent with the documentation. Non-forked (a.k.a.
virtual) subshells are an internal implementation detail which
scripts should not have to be concerned with. The manual page
doesn't mention them at all.
2. Inherently broken. Since a subshell may fork for any number of
reasons, even mid-run, and those reasons may change with
bugfixes and further development, scripts have never actually
been able to rely on the value of ${.sh.subshell}.
So, this commit fixes the counter to count the levels of all
subshells, both virtual and forked.
src/cmd/ksh93/sh/xec.c: _sh_fork():
- Increase ${.sh.subshell} whenever we fork.
src/cmd/ksh93/sh/subshell.c:
- sh_subfork():
* Fix comment to properly explain what it does. It doesn't
"create" a subshell, it forks off an existing virtual subshell.
* Don't zero ${.sh.subshell}. Instead, since sh_fork() increases
it but we're forking an existing subshell, undo the increase.
- sh_subshell():
* Remove 'int16_t subshell' variable. It was unnecessary and
mostly unused. It was also the wrong type: it was assigned the
value from shp->subshell which is of type short.
* Increase and decrease the level of virtual subshells and
${.sh.subshell} independently.
src/cmd/ksh93/tests/variables.sh:
- Add regression tests for ${.sh.subshell} in virtual and forked
subshells of several kinds: comsub, parentheses, pipe, bg job.
- Undo wrong error test count fix from 04b4aef0.
(cherry picked from commit a0e0e29e7e0dbf21e4b3958ae02bde6665fb2696)
This fixes (or at least works around) a bug that caused special
variables such as PATH, LANG, LC_ALL, LINENO, etc. to lose their
effect after being unset in a subshell.
For example:
(unset PATH; PATH=/dev/null; ls); : wrongly ran 'ls'
(unset LC_ALL; LC_ALL=badlocale); : failed to print a diagnostic
This is yet another problem with non-forking/virtual subshells. If
you forced the subshell to fork (one way of doing this is using the
'ulimit' builtin, e.g. ulimit -t unlimited) before unsetting the
special variable, the problem vanished.
I've tried to localise the problem. I suspect the sh_assignok()
function, which is called from unall(), is to blame. This function
is supposed to make a copy of a variable node in the virtual
subshell's variable tree. Apparently, it fails to copy the
associated permanent discipline function settings (stored in the
np->nvfun->disc pointer) that gave these variables their special
effect, and which survive unset. However, my attempts to fix that
have been unsuccessful. If anyone can figure out a fix, please send
a patch/pull request!
Data point: This bug existed in 93u 2011-02-08, but did not yet
exist in M-1993-12-28-s+. So it is a regression.
Meanwhile, pending a proper fix, this commit adds a safe
workaround: it forces a non-forked subshell to fork before
unsetting such a special variable.
src/cmd/ksh93/bltins/typeset.c: unall():
- If we're in a non-forked, non-${ ...; } subshell, then before
unsetting any variable, check for variables with internal
trap/discipline functions, and call sh_subfork() if any are
found. To avoid crashing, this must be done before calling
sh_pushcontext(), so we need to loop through the args separately.
src/cmd/ksh93/tests/variables.sh:
- Remove the 'ulimit' that forced the fork; we do this in C now.
(cherry picked from commit 21b1a67156582e3cbd36936f4af908bb45211a4b)
Many tests used direct paths to some commands, mostly /bin/echo and
/bin/cat. This is unportable (breaks on e.g. NixOS).
The correct way is to obtain the direct path using 'whence -p'.
There was also one use of '/usr/bin/pstack' in tests/comvario.sh
that seemed bogus. Apparently this was supposed to analyse a core
file after a crash. Even on Solaris and Linux, where that command
exists, the argument is documented to be a PID, not a core file. If
this ever worked anywhere, then it was system-specific enough to be
useless here, so I've removed it.
(cherry picked from commit 4563b8bc651cd9cb18dc73f56a041f7ac5534395)
When a test aborted due to a signal, the >256 exit code signifying
the signal was incorrectly reported as the number of errors.
src/cmd/ksh93/tests/shtests:
- For $? > 256, obtain and report the signal name using 'kill -l'.
(cherry picked from commit a7d8ae628e228fc3cadcf977fbffc87b90c7bc53)
Namerefs aren't broken in virtual/non-forked subshells after all
(phew). It is changing the locale that is somehow broken.
src/cmd/ksh93/tests/variables.sh:
- Pending further investigation, update the TODO and don't fork the
subshell until actually needed.
(cherry picked from commit efa3150396b383b6a68b2df45eab9005593b2e42)
This fixes a really stupid bug in my own code for unsetting a
function in a subshell. The algorithm for walking through the
subshell tree was broken, resulting in an infinite loop if there
were multiple levels of subshell.
src/cmd/ksh93/bltins/typeset.c:
- Correct the subshell function tree walk that deletes functions
from zombie parent scopes.
src/cmd/ksh93/tests/subshell.sh:
- Add a regression test for setting and unsetting identically named
functions in multiple levels of subshell.
(cherry picked from commit 972a7999c7f16469138daf3d86dfd6c0db3f4879)
The $! special parameter was not set if a background job
(somecommand &) or co-process (somecommand |&) was launched as the
only command within a braces block with an attached redirection,
for example:
{
somecommand &
} >&2
With the bug, $! was unchanged; now it contains the PID of
somecommand.
Ref.: https://github.com/att/ast/issues/1357
src/cmd/ksh93/sh/parse.c: item():
- When processing redirections following a compound command, always
create a parent node with the TSETIO (I/O redirection) token.
Before this commit, if the last command was of type TFORK (and
the last command only tested as TFORK if the bg job or coprocess
was the only command in a braces block, because the ksh parser
optimises away the braces in that case), then the parent node was
created with the TFORK token instead.
I have no idea what David Korn's intention was with that, but
this is clearly very wrong. Creating another TFORK node when
parsing the redirection caused sh_exec() in sh/xec.c to execute
the redirection in an extra forked, non-background subshell.
Since redirections are executed before anything else, this
subshell is what then launched the background job between the
braces, so $! (a.k.a. shp->bckpid) was updated in that subshell
only, and never in the main shell. The extra subshell also
prevented the background job from being noticed by job control
on interactive shells.
So, the fix is simply to remove the broken test for TFORK.
src/cmd/ksh93/tests/variables.sh:
- Add regression tests for a bg job and a co-process as the only
command within a braces block with attached redirection.
(cherry picked from commit ffe5df30e69f7b596941a98498014d8e838861f2)
src/cmd/ksh93/tests/variables.sh:
- Unset -x in a command substitution that redirects stderr to
stdout; this caused a spurious failure with tracing active.
- Execute the nameref tests in a subshell. They modified LINENO, so
that all the line numbers after this test were traced and/or
reported as 'foo'.
. This exposed a bug in namerefs: this is yet another thing that
is broken in non-forked/virtual subshells! That is for another
commit. For now, fork the subshell and leave a TODO.
(cherry picked from commit 8b7f8f9b14523a66bdf337612daef2501c2bb5ba)
src/cmd/ksh93/tests/bracket.sh:
- -r and -w are always true when running as root, so skip these.
- Fix several typos in the [[ tests (apparent copy/paste errors).
(cherry picked from commit b9325e9753b38f087d0c5501d474f7d1f15c4d36)
Fix a bug in autoloading functions. Directories in the path search
list which should be skipped (e.g. because they don't exist) did
not interact correctly with autoloaded functions, so that a
function to autoload was not always found.
Details:
https://github.com/att/ast/issues/1454
Fix backported (and cleaned up) from:
https://github.com/att/ast/commit/3bc58164
src/cmd/ksh93/sh/path.c:
- path_opentype(): Fix the path search loop so that entries marked
with PATH_SKIP are handled correctly.
src/cmd/ksh93/tests/functions.sh:
- Add regression test verifying an autoloaded function with a PATH
that triggered the bug.
The bug in path_opentype() fixed by this commit may affect other
scenarios but we know it affects autoloaded functions. Hence the
test for that scenario.
(cherry picked from commit a27903165775309f4f032de5d42ec1785f14cfbc)
Fix BUG_TESTERR1A: POSIX non-compliance of 'test'/'[' exit status
on error. The command now returns status 2 instead of 1 when given
an invalid number or arithmetic expression, e.g.: [ 123 -eq 123x ]
The problem was that the test builtin (b_test()) calls the generic
arithmetic evaluation subsystem (sh/arith.c, sh/streval.c) which
has no awareness of the test builtin. A simple solution would be to
always make the arithmetic subsystem use an exit status > 1 for
arithmetic errors, but globally changing this may cause backwards
compatibility issues. So it's best to change the behaviour of the
'test' builtin only. This requires the arithmetic subsystem to be
aware of whether it was called from the 'test' builtin or not. To
that end, this commit adds a global flag and overrides the
ERROR_exit macro where needed.
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/sh/defs.c:
- Declare and initialise a global sh_in_test_builtin flag.
- Declare internal function for ERROR_exit override in test.c.
src/cmd/ksh93/bltins/test.c:
- Add override for ERROR_exit macro using a function that checks if
the exit status is at least 2 if the error occurred while running
the test builtin.
- b_test(): Set sh_in_test_builtin flag while running test builtin.
src/cmd/ksh93/sh/arith.c,
src/cmd/ksh93/sh/streval.c:
- Override ERROR_exit macro using function from test.c.
src/cmd/ksh93/tests/bracket.sh:
- Add regression test verifying status > 1 on arith error in test.
(cherry picked from commit 5eeae5eb9fd5ed961a5096764ad11ab870a223a9)
Aliases can now be correctly unset within subshell environments
(such as ( ... ), $(command substitutions), etc), as well as
non-subshell "shared" command substitutions (${ ...; }). Before,
attempts to unset aliases within these were silently ignored.
Prior discussion: https://github.com/att/ast/issues/108
Subshell alias trees are only referenced in a few places in the
code, *and* have always been broken, so this commit gets rid of the
whole notion of a subshell alias tree. Instead, there is now just
one flat alias tree, and subshells fork into a separate process
when aliases are set or unset within them. It is not really
conceivable that this could be a performance-sensitive operation,
or even a common one, so this is a clean fix with no downside.
src/cmd/ksh93/include/defs.h:
- Remove sh_subaliastree() definition.
src/cmd/ksh93/sh/subshell.c:
- Remove salias element (pointer to subshell alias tree) from
subshell struct.
- Remove sh_subaliastree() function.
- sh_subshell(): Remove alias subshell tree cleanup.
src/cmd/ksh93/bltins/typeset.c:
- b_alias(): If in subshell, fork before setting alias.
- b_unalias(): If in subshell, fork before unsetting alias.
- unall(): Remove sh_subaliastree() call.
src/cmd/ksh93/sh/name.c:
- nv_open(): Remove sh_subaliastree() call.
src/cmd/ksh93/tests/subshell.sh:
- Add regression tests for unsetting or redefining aliases within
subshells.
(cherry picked from commit 12a15605b9521a2564a6e657905705a060e89095)
Functions can now be correctly redefined and unset in subshell
environments (such as ( ... ), $(command substitutions), etc).
Before this fix, attempts to do this were silently ignored (!!!),
causing the wrong code (i.e.: the function by the same name from
the parent shell environment) to be executed.
Redefining and unsetting functions within "shared" command
substitutions of the form '${ ...; }' is also fixed.
Prior discussion: https://github.com/att/ast/issues/73
src/cmd/ksh93/sh/parse.c:
- A fix from George Koelher (URL above). He writes:
| The parser can set t->comnamp to the wrong function.
| Suppose that the shell has executed
| foo() { echo WRONG; }
| and is now parsing
| (foo() { echo ok; } && foo)
| The parser was setting t->comnamp to the wrong foo. [This
| fix] doesn't set t->comnamp unless it was a builtin. Now the
| subshell can't call t->comnamp, so it looks for foo and finds
| the ok foo in the subshell's function tree.
src/cmd/ksh93/bltins/typeset.c:
- Unsetting functions in a virtual/non-forked subshell still
doesn't work: nv_open() fails to find the function. To work
around this problem, make 'unset -f' fork the subshell into its
own process with sh_subfork().
- The workaround exposed another bug: if we unset a function in a
subshell tree that overrode a function by the same name in the
main shell, then nv_delete() exposes the function from the main
shell scope. Since 'unset -f' now always forks a subshell, the
fix is to simply walk though troot's parent views and delete any
such zombie functions as well. (Without this, the 4 'more fun'
tests in tests/subshell.sh fail.)
src/cmd/ksh93/sh/subshell.c: sh_subfuntree():
- Fix function (re)definitions and unsetting in "shared" command
substitutions of the form '${ commandlist; }' (i.e.: if
sp->shp->subshare is true). Though internally this is a weird
form of virtual subshell, the manual page says it does not
execute in a subshell (meaning, all changes must survive it), so
a subshell function tree must not be created for these.
src/cmd/ksh93/tests/subshell.sh:
- Add regression tests related to these bugfixes. Test unsetting
and redefining a function in all three forms of virtual subshell.
(cherry picked from commit dde387825ab1bbd9f2eafc5dc38d5fd0bf9c3652)
These always fail when running as root, as root can access files
regardless of permission bits.
(cherry picked from commit a821fe13906ac8ef56162bebd7c3e976b973f91c)
Certain environment variables were interpreted as arithmetic
expressions on startup, leading to code injection.
Ref.:
https://bugzilla.redhat.com/show_bug.cgi?id=1757324c7de8b6412
(cherry picked from commit ee6b001d0611ad2e00b6da2c2b42051995c0a678)
It was annoying me that I always had to type out the .sh in
'shtests builtins.sh', etc. to execute a specific regression test
script. All the test scripts end in .sh, so this is superflous.
src/cmd/ksh93/tests/shtests:
- Add .sh to test script arguments if not already present.
(cherry picked from commit d780e701fbf6f27b6e79f88542a3175bf5dd85e9)
The issue with truncating files was caused by out-of-sync streams.
Details and discussion: https://github.com/att/ast/issues/61
src/cmd/ksh93/sh/io.c: sh_iorestore():
- To be safe, sync all streams before restoring file descriptors.
src/cmd/ksh93/tests/io.sh:
- Add two regression tests for truncating files with this
combination of redirections.
- The second test, which invokes a -c script, is disabled for now
as this triggers another corner case bug involving the SH_NOFORK
optimisaton for -c scripts. That fix is for another commit.
(cherry picked from commit 18fb64840365c2ff4608188e5487bd79d08f67d1)
