src/cmd/ksh93/tests/leaks.sh:
- This script was never actually running the regression
tests because 'vmstate' isn't available as a builtin.
While this can be fixed by adding vmstate to the builtin
table, that has the downside of increasing the binary
size of ksh. This commit replaces all usage of 'vmstate'
with 'ps' and 'awk' as a different way to measure
memory usage. The memory leaks regression tests are now
always run.
- Rename old $n to $N due to new $n interfering with the old
regression test.
- Add before and after results for the number of 1024-byte
blocks leaked in each test.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
File descriptors that are too far out of range will cause the
read builtin to crash. The following example will generate
two crashes:
$ ksh -c 'read -u 2000000' || ksh -c 'read -u-2000000'
The fix is to error out when the given file descriptor is out
of range. This bugfix is from Tomas Klacko, although it was
modified to use 'sh_iovalidfd' and reject numbers greater
than 'INT_MAX':
https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01912.html
The question about 'shp->fdstatus[-1]' only applies to ksh93v-
(ksh93u+ doesn't have any references to 'shp->fdstatus[-1]').
src/cmd/ksh93/bltins/read.c:
- File descriptors that are out of range should be rejected
with an error message (like invalid file descriptors that
are in range). The seemingly redundant check for negative
numbers is there because out of range negative numbers also
cause memory faults despite the later 'fd<0' check.
src/cmd/ksh93/tests/io.sh:
- Add three tests for attempting 'read -u' on various invalid
file descriptor numbers.
Not cleaning up the FIFO broke 'grep -r' in the arch directory,
making it hang forever. So we need a better way of cleaning it up.
bin/package,
src/cmd/INIT/package.sh:
- Unlink the FIFO early after sleeping a second in the background.
This works because the named directory entry is only needed to
establish the pipe, not to keep it going.
To get ksh to prefer UTC over GMT in 'printf %T' output, only the
change in format[] was needed. The corresponding change in zone[]
made it prefer UTC for London time, even in summer time, which is
wrong -- e.g.:
$ LANG=C TZ=Europe/London arch/*/bin/ksh -c 'date; printf %T\\n now'
Tue Jun 30 01:39:09 BST 2020
Tue Jun 30 00:39:09 UTC 2020
src/lib/libast/tm/tmdata.c:
- Revert change in zone[].
The regression test failed on systems where 'chmod' exists at more
than one location, e.g. Slackware where it's at both /bin/chmod and
/usr/bin/chmod.
src/cmd/ksh93/tests/builtins.sh: 'whence -a'/tracked aliases test:
- In the expected value, use modified 'whence -a -p chmod' output
to get all of the paths to chmod.
- On failure, report both expected and actual values.
* Fix the readonly builtin's scope in functions
This bug was first reported at https://github.com/att/ast/issues/881
'tdata.sh->prefix' is only set to the correct value when
'b_readonly' is called as 'export', which breaks 'readonly' in
functions because the correct scope isn't set. As a result, the
following example will only print a newline:
$ function show_bar { readonly foo=bar; echo $foo; }; show_bar
The fix is to move the required code out of the if statement for
'export', as it needs to be run for 'readonly' as well. This bugfix
is from https://github.com/att/ast/pull/906
src/cmd/ksh93/bltins/typeset.c:
- Set 'tdata.sh->prefix' to the correct value, otherwise 'readonly'
uses the wrong scope.
src/cmd/ksh93/tests/builtins.sh:
- Add the regression test from ksh2020, modified to run in a
subshell.
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/sh.1:
- Add documentation of 'readonly' vs. 'typeset -r' difference:
'readonly' does not create a function-local scope.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
The following set of commands ends with a memory fault under
certain circumstances because ksh attempts to free memory
twice, causing memory corruption:
$ testarray=(1 2)
$ compound testarray
$ unset testarray
$ eval testarray=
The fix is to make sure 'np->nvfun' is a valid pointer before
attempting to free memory in 'put_tree'. This patch is from
OpenSUSE: https://build.opensuse.org/package/view_file/shells/ksh/ksh93-nvtree-free.dif?expand=1
src/cmd/ksh93/sh/nvtree.c:
- Do not try to free memory when 'np->nvfun' and 'val'
are false.
src/cmd/ksh93/tests/comvar.sh:
- Add a regression test for the double free problem. The
reproducer must be run from an executable script
with 'ksh -c'.
Variables created with 'typeset -RF' were being treated as
short integers, even though they are actually floating point
values. As a result the following example will cause a crash:
$ typeset -RF foo=1
$ test "$foo"
This is fixed by checking for 'NV_DOUBLE' with 'nv_isattr',
which prevents ksh from treating floating point values as
short integers due to '== NV_INT16P' excluding 'NV_DOUBLE'.
This bugfix was backported from ksh93v- 2013-10-10-alpha.
src/cmd/ksh93/sh/array.c,
src/cmd/ksh93/sh/name.c,
src/cmd/ksh93/sh/nvdisc:
- Avoid treating floating point values as short integers by
checking for 'NV_DOUBLE' with 'nv_isattr'.
src/cmd/ksh93/tests/types.sh:
- Add a regression test for the 'typeset -RF' crash. The
crash cannot be replicated if 'typeset -RF' sets 'foo'
to zero.
ksh, even non-interactive, loads /etc/ksh.kshrc by default. On
some systems this can be a problem, e.g. OpenBSD, which installs a
default /etc/ksh.kshrc which is designed for its version of pdksh.
Quoth sh.1:
On systems that support a system wide /etc/ksh.kshrc
initialization file, if the filename generated by the expansion
of ENV begins with /./ or ././ the system wide initialization
file will not be executed.
src/cmd/ksh93/tests/shtests,
src/cmd/ksh93/tests/builtins.sh,
src/cmd/ksh93/tests/options.sh,
src/cmd/ksh93/tests/pty.sh:
- Instead of emptying or unsetting ENV, ensure it is exported with
a default value of /./dev/null so we skip loading the system-wide
profile and load an empty user profile.
- Where a specific ENV path was required for the tests, prefix it
with '/.' so it starts with '/./'.
The cd builtin was removing '.' from directory names when combined
with a preceding '../', which caused commands like 'cd ../.local'
to become 'cd ../local'. This patch fixes the problem by limiting
the extra handling to leading '..'. The bugfix comes from ksh93v-
2013-10-10-alpha, although this version is a shortened patch from
Solaris (as ksh93v- refactored a decent amount of the code for the
cd builtin).
src/cmd/ksh93/bltins/cd_pwd.c:
- cd should only check for leading '..', as trying to handle a lone
'.' only causes problems.
src/cmd/ksh93/tests/builtins.sh:
- Add a regression test for this problem based on the test present in
ksh93v- 2013-10-10-alpha.
Patch from Solaris:
https://github.com/oracle/solaris-userland/blob/860d27f/components/ksh93/patches/270-23319761.patch
Apparently some systems are still configured to use GMT instead of
UTC after all. This included our own GitHub CI runner config.
Oops. This made the previous commit fail to pass the CI test run.
We can't win this one, it's got to be either one or the other.
UTC is the international standard on which civil time is based.
GMT is often taken as synonymous for UTC, but in navigation,
it can differ from UTC by up to 0.9 seconds. Ref.:
https://en.wikipedia.org/w/index.php?title=Greenwich_Mean_Time&oldid=963422787
The more ambiguous term should not be the first preference.
src/cmd/ksh93/tests/builtins.sh:
- Before checking 'printf %T now' output against 'date' output,
change any ' GMT ' in the latter to ' UTC '.
.github/workflows/ci.yml:
- Set time zone to UTC, not GMT.
"UTC" is the modern name for what used to be "GMT", but ksh still
preferred GMT. On systems configured to use the UTC time zone, this
caused a 'printf %T' regression test failure in tests/builtins.sh
as the external 'data' utility will prefer UTC these days.
src/lib/libast/tm/tmdata.c:
- Reorder the name alternatives for UTC/GMT so that UTC is
the first preference.
src/cmd/ksh93/tests/builtins.sh:
- Report expected and actual values on 'printf %T' failure.
Related: #6
When specifying another ksh to test using e.g. KSH=/bin/ksh to test
the system's stock ksh, the pty command was not found because it
resides in the arch/*/bin directory. The main shtests script bases
its $PATH on $INSTALLROOT, but does not set INSTALLROOT itself if
the shell to test is outside the installation hierarchy. Setting it
would allow the pty.sh regresssion tests to run on a shell
specified on the command line.
bin/shtests:
- Set and export INSTALLROOT to our default installation hierarchy
if it wasn't already set on entry.
When neither '-o emacs' nor '-o vi' is active, there were a couple
of bugs with entering tab characters:
1. Tab completion was erroneously left active. The cause of this
was that raw Bourne edit mode is handled by ed_viread() in vi.c
on shells with wide character support, instead of the default
ed_read() in edit.c, and the former failed to check if vi mode
is active when processing tab characters.
2. When entering literal tab characters, the cursor was moved to
the right only one character, instead of the amount of
characters corresponding to the tab.
src/cmd/ksh93/edit/vi.c: getline():
- Before processing '\t' (tab) for command completion, check that
the 'vi' shell option (SH_VI) is active.
src/cmd/ksh93/edit/edit.c: ed_virt_to_phys():
- When translating literal tabs to on-terminal spaces and when
recalculating the cursor position, remove erroneous checks for
SH_VI; this is also needed in raw Bourne mode. According to my
own testing, this has no effect on emacs mode (knock on wood).
src/cmd/ksh93/tests/pty.sh:
- Add two regression tests. An odd race condition reveals itself in
either pty or in ksh's raw/Bourne edit mode; see comment in test.
Effect is we have to expect either literal tabs or tabs expanded
to spaces, until that is tracked down and fixed.
Fixes#43.
This commit fixes the bug reported in:
https://github.com/att/ast/issues/682
The following sequence fails in vi mode because ksh looks in the
wrong part of the 'virtual' buffer:
$ touch ~/testfile
$ ls ~/test<tab>
The fix is to change 'virtual[i]' to 'virtual[last_virt]' in the
bugged section of code. The other changes are to make sure listing
files in a directory with something like 'ls /etc/<tab>' calls the
code for Ctrl+L to preserve 'ls /etc/' rather than try (and fail)
to complete the directory name, producing 'ls /etc\n/'. This bugfix
was backported from ksh93v- 2013-10-10-alpha.
src/cmd/ksh93/edit/vi.c
- Backport the bugfix from ksh93v- 2013-10-10-alpha for this
problem.
src/cmd/ksh93/tests/pty.sh
- Add a regression test for this issue using pty, adjusted slightly
for a fake home directory in /tmp.
Somewhat notable changes in this commit:
- The 'set +r' bugfix (re: 74b41621) is now documented in the
changelog.
- Missing options have been added to the synopsis section of the
ksh man page.
- The minor formatting fix from https://github.com/ksh-community/ksh/pull/5
has been applied to the ksh man page.
- A few fixes from https://github.com/att/ast/commit/5e747cfb
have been applied to the ksh man page.
- The man page fixes from https://github.com/att/ast/pull/353
have been applied, being:
- An addition to document the behavior of 'set -H'.
- A fix for the cd section appending rksh93.
- A fix for some options being indented too far.
- Removal of a duplicate section documenting '-D'.
- Reordering the options for 'set' in alphabetical order.
- A minor fix for the documentation of 'ksh -i'.
src/cmd/ksh93/tests/bracket.sh:
- Disable tests for [[ -N ... ]] (test -N ...), because it is
expected to break on systems where $TMPDIR (or even the entire
root file system) is mounted with noatime for better performance.
Ref.: https://opensource.com/article/20/6/linux-noatime
(It also needs annoyingly long sleep times on older systems with
a 1-second timestamp granularity.)
Testing the behaviour of an external editor, even the standard one,
is outside the scope of the ksh regression tests.
src/cmd/ksh93/tests/pty.sh:
- Disable a test that invoked vi(1) and that failed, either
intermittently or consistently, on too many systems because
whatever vi(1) is installed locally doesn't write the string
"/tmp/" exactly as and/or when expected.
The output format is now identical to mksh's except for
the locale-dependent radix point ('.' or ',').
src/cmd/ksh93/bltins/misc.c:
- Output format tweak: pad seconds with initial zero if < 10.
- Use "too many operands" (e_toomanyops) error msg from 3ba4900e
if there are operands, instead of "bad syntax" (e_badsyntax).
- Consolidate repetitive calculating and printing code
into print_times().
- Get rid of some excessive variables.
src/cmd/ksh93/tests/builtins.sh:
- Update regression tests to match the above.
src/cmd/ksh93/data/builtins.c:
- Update sh_opttimes[] version string.
Well, that's what I get for backporting code without properly
checking it over. There was an elementary math error in how the
times builtin calculated seconds:
utime_sec = utime - utime_min;
which could cause output such as "1m98.38s" or "3m234.77s".
src/cmd/ksh93/bltins/misc.c: b_times():
- Use fmod(), i.e. floating point modulus, to calculate seconds.
When a nonexistent test script was given as an argument to
shtests, this was not counted as an error and shtests exited
successfully (with status 0).
src/cmd/ksh93/tests/shtests:
- Increase total error count if a test script is not found.
The code for handling process substitution with redirection was
never being run because IORAW is usually set when IOPROCSUB is
set. This commit fixes the problem by moving the required code
out of the !IORAW if statement. The following command now prints
'good' instead of writing 'ok' to a bizzare file:
$ ksh -c 'echo ok > >(sed s/ok/good/); wait'
good
This commit also fixes a bug that caused the process ID of the
asynchronous process to print when the shell was in interactive
mode. The following command no longer prints a process ID,
behaving like in Bash and zsh:
$ echo >(true)
/dev/fd/5
src/cmd/ksh93/sh/args.c:
- Temporarily turn off the interactive state while in a process
substitution to prevent the shell from printing the PID of
the asynchronous process.
src/cmd/ksh93/sh/io.c:
- Move the code for process substitution with redirection into
a separate if statement.
src/cmd/ksh93/tests/io.sh:
- Add two tests for both process substitution bugs fixed by this
commit.
src/cmd/ksh93/tests/shtests:
- Update shtests with a patch from Martijn Dekker to use
pretty-printing for the output from the times builtin (if it
is available).
Fixes#2
.github/workflows/ci.yml:
- Disable Mac build as the GitHub runners appear to be broken
(e.g. SIGCHLD fails, unlike on real Macs) and tend to hang.
- For the Linux build:
- Set GMT timezone for 'printf %T' tests in builtins.sh.
- Set the ulimit for open files to 1024 as the subshell.sh tests
need a lot of open files.
- As the runners lack the POSIX standard /dev/tty device, use the
script command to provide a fake /dev/tty for the bracket.sh
tests that use 'test -t $fd'.
Ref.: https://github.com/actions/runner/issues/241
This commit fixes the bug described in att/ast#32. The fix and
following explanation is from att/ast#467:
While copying variables from function's local scope to a new scope,
variable attributes were not copied. Such variables were not marked
to be exported in the new function. For e.g.
function f2 { env | grep -i "^foo"; }
function f1 { env | grep -i "^foo"; f2; }
foo=bar f1
prints 'foo=bar' only once, but it should print be twice.
src/cmd/ksh93/sh/xec.c:
- When variables from the local scope of a function are copied into
the scope of a nested function, the attributes of the variables
need to be copied as well.
src/cmd/ksh93/tests/functions.sh:
- Add regression tests from ksh2020 to check environment variables
passed to functions.
src/cmd/ksh93/tests/builtins.sh:
- The output of 'printf %T now' and the external 'date'
command aren't guaranteed to be the same unless $LC_ALL
is set to 'C'. Set LC_ALL in these command substitutions
to fix a spurious test failure on Linux.
Ksh was trying to use the 'pw' variable as a valid pointer even
when it was NULL. This is fixed by doing the error check for
'pw' before doing anything else in 'job_kill'.
This bugfix is from Red Hat:
44e0a643a9/f/SOURCES/ksh-20130214-fixkill.patchFixes#34
The 'stop' and 'suspend' default aliases are now converted into
regular built-in commands so that 'unalias -a' does not remove
them, 'suspend' can do some sanity checks, and something like
cmd=stop; $cmd $!
will now work.
src/cmd/ksh93/bltins/trap.c:
- b_kill(): Incorporate 'stop' functionality, which is simply
setting the same flag and variable as '-s STOP' would have done.
- b_suspend(): Add simple builtin function that sends SIGSTOP to
the main shell. Check for no operands, and refuse to suspend a
login shell (which would leave the user stuck with no way out).
Also check that 'kill' succeeds; if we're in an asynchronous
subshell, it is possible the main shell no longer exists.
src/cmd/ksh93/data/aliases.c:
- Remove "stop" and "suspend" default aliases. (Why were these
conditional upon SIGTSTP when they actually issued SIGSTOP?)
src/cmd/ksh93/include/builtins.h,
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/data/msg.c:
- Add declarations of "stop" and "suspend" regular built-ins.
- Add option strings (AST manual/--man pages) for them.
- Add e_toomanyops ("too many operands") reusable error message for
b_suspend(). Other new commands may want this at some point.
src/cmd/ksh93/sh.1:
- Remove "stop" and "suspend" default aliases.
- Document "stop" and "suspend" regular built-in commands.
This appears to be originating from:
2755 *) if test ! -d $INSTALLROOT
2756 then INSTALLROOT=$PACKAGEROOT;
where INSTALLROOT=PACKAGEROOT and 'clean' deletes everything under
INSTALLROOT thus deleting the entire git repo. This only applies when
there's no arch/$HOSTTYPE directory due to the condition above.
bin/package,
src/cmd/INIT/package.sh:
- Delete arch/$HOSTTYPE as stated in the documentation
for the clean action instead of $INSTALLROOT.
This fixes compiler warnings for implicit-ints:
warning: return type defaults to 'int' [-Wimplicit-int]
1854: sh_tcgetattr(int fd, struct termios *tty)
1863: sh_tcsetattr(int fd, int cmd, struct termios *tty)
cmd/ksh93/edit/edit.c:
- Set the return type explicitly to int and align with the prototype
declared in include/terminal.h.
After making PATH readonly in a virtual subshell (without otherwise
changing it, so the subshell is never forked), then the main shell
would erroneously fork into a background process immediately after
leaving the virtual subshell. This was caused by a bug in the
forking workaround that prevents changes in PATH in a virtual
subshell from clearing the parent shell's hash table.
src/cmd/ksh93/sh/name.c: nv_putval():
- If we're either setting or restoring PATH, do an additional check
for the NV_RDONLY flag, which means the function was told to
ignore the variable's readonly state. It is told to ignore that
when restoring the parent shell state after exiting a virtual
subshell. If we don't fork then, we don't fork the parent shell.
src/cmd/ksh93/tests/subshell.sh:
- Add regression test verifying that no forking happens when making
PATH readonly in a subshell.
Fixes#30.
src/cmd/ksh93/data/variables.c:
- Running 'unset .sh.lineno' creates a memory fault, so fix that
by giving it the NV_NOFREE attribute. This crash was happening
because ${.sh.lineno} is an integer that cannot be freed from
memory with free(3).
src/cmd/ksh93/sh/init.c:
- Tell _nv_unset to ignore NV_RDONLY when $RANDOM and $LINENO are
restored from the subshell scope. This is required to fully
restore the original state of these variables after a virtual
subshell finishes.
src/cmd/ksh93/bltins/typeset.c,
src/cmd/ksh93/sh/subshell.c:
- Disabled some optimizations for two instances of 'sh_assignok' to
fix 'readonly' in virtual subshells and '(unset .sh.level)' in
nested functions. This fixes the following variables when
'(readonly $varname); enum varname=' is run:
$_
${.sh.name}
${.sh.subscript}
${.sh.level}
The optimization in question prevents sh_assignok from saving the
original state of these variables by making the sh_assignok call
a no-op. Ksh needs the original state of a variable for it to be
properly restored after a virtual subshell has run, otherwise ksh
will simply carry over any new flags (being NV_RDONLY in this case)
from the subshell into the main shell.
src/cmd/ksh93/tests/variables.sh:
- Add regression tests from Martijn Dekker for setting special
variables as readonly in virtual subshells and for unsetting
special variables in general.
Fixes#4
src/cmd/INIT/ratz.c:
- Fix build warning:
src/cmd/INIT/ratz.c:4741:2: warning: case label value exceeds maximum value for
type
4741 | case 0241:
| ^~~~
The character literal in the switch expression was being treated
as a signed char while the case label 0241 is greater than 127,
resulting in this warning.
'whence -a' bases the path for tracked aliases on the user's
current working directory if an enabled ksh builtin of the same
name is also available. The following example will claim 'cat'
is in the user's current working directory:
$ whence -a cat
cat is a tracked alias for /usr/bin/cat
$ builtin cat
$ whence -a cat
cat is a shell builtin
cat is /usr/bin/cat
cat is a tracked alias for /current/working/directory/cat
This patch from ksh2020 fixes this problem by properly saving the
path of the tracked alias for use with 'whence -a', since
'path_pwd' (as implied by the function's name) only gets the users
current working directory, not the location of tracked aliases.
Ref.: https://github.com/att/ast/issues/1049
This bug was originally reported by David Morano about two decades
ago to the AST team: https://github.com/att/ast/issues/954
src/cmd/ksh93/bltins/whence.c:
- Print the actual path of a tracked alias, path_pwd doesn't
have this functionality.
src/cmd/ksh93/include/name.h:
- Add 'pathcomp' for saving the value of tracked aliases.
src/cmd/ksh93/sh/path.c:
- Save the value of tracked aliases for use by whence.
src/cmd/ksh93/tests/builtins.sh:
- Add a regression test for using 'whence -a' on tracked
aliases with a builtin equivalent.
src/cmd/ksh93/data/builtins.c:
- sh_optexec[], sh_optdirect[]: Move paragraphs on exit behaviour
to the EXIT STATUS section, where people would look for them.
- sh_optexec[]: Since we modified b_exec() to support 'redirect',
add "ksh93" to the credits to avoid blaming AT&T for our changes.
Applying the fix for 'unset -f' exposed a crashing bug in lookup()
in sh/nvdisc.c, which is the function for looking up discipline
functions. This is what caused tests/variables.sh to crash.
Ref.: https://github.com/ksh93/ksh/issues/23#issuecomment-645699614
src/cmd/ksh93/sh/nvdisc.c: lookup():
- To avoid segfault, check that the function pointer nq->nvalue.rp
is actually set before checking if nq->nvalue.rp->running==1.
src/cmd/ksh93/sh/xec.c,
src/cmd/ksh93/tests/functions.sh:
- Uncomment the 'unset -f' fix from b7932e87.
Resolves#21 (again).
The fix in sh/xec.c, which was backported from the ksh 93v- beta to
delay the actual removal of a running function that unsets itself,
caused a segfault in the variables.sh regression tests (see #23).
src/cmd/ksh93/sh/xec.c:
- Comment out the backported code pending a correct fix for #21.
Now both types of functions silently fail to unset themselves
(unless they're discipline functions).
src/cmd/ksh93/tests/functions.sh:
- Disable regression tests checking that the function was actually
unset, pending a correct fix for #21.
Resolves: #23Reopens: #21
src/cmd/ksh93/sh/name.c:
- Correct the check for when a function is currently running
to fix a segmentation fault that occurred when a POSIX
function tries to unset itself while it is running.
This bug fix was backported from ksh93v-.
src/cmd/ksh93/sh/xec.c:
- If a function tries to unset itself, unset the function
with '_nv_unset(np, NV_RDONLY)' to fix a silent failure.
This fix was also backported from ksh93v-.
src/cmd/ksh93/tests/functions.sh:
- Add four regression tests for when a function unsets itself.
Resolves#21
Note that shtests simply does a 'grep -c err_exit' and substracts 1
to count the number of regression tests in a test script. Not all
test scripts make temp dirs, so subtracting 2 instead won't do.
src/cmd/ksh93/tests/*.sh:
- Escape the err_exit call in the routine to create a temporary
directory so that it is not counted as a regression test.
That bypasses the alias, so we have to pass $LINENO manually.
Four added tests did not correctly report their line numbers
upon failure and were counted as one, because the err_exit
alias/function pair was called from a shell function.
Note that shtests simply does a 'grep -c err_exit' to count the
number of regression tests in a test script.
src/cmd/ksh93/tests/subshell.sh:
- check_hash_table():
- Take line number as 1st argument.
- Quote a character in err_exit to bypass the alias when calling
it, so we can pass on the argument for the line number. This
also stops this helper function from being counted as a test.
- When calling check_hash_table(), pass $LINENO.
- Add dummy err_exit comments to have the tests counted.
Ksh was not checking for `command` when running a special builtin,
which caused preceding invocation-local variable assignments to
become global. This is the reproducer from the att/ast#72:
$ foo=BUG command eval ':'
$ echo "$foo"
This no longer prints 'BUG', as ksh now makes sure the command builtin
is not running a special builtin before making invocation-local
variable assignments global.
src/cmd/ksh93/sh/xec.c:
- Backport the bugfix for BUG_CMDSPASGN from ksh93v- 2013-10-10-alpha.
src/cmd/ksh93/tests/builtins.sh:
- Add a regression test based on the reproducer in att/ast#72.
As previously reported in rhbz#1112306 (https://bugzilla.redhat.com/show_bug.cgi?id=1112306),
ksh may crash when receiving SIGCHLD because GCC's optimizer will fail to generate
`addl` and `sub` instructions to increment and decrement `job.in_critical` in the
`job_subsave` function. This bug *does* occur in GCC 10 with `-O2`, but not `-O1`;
it doesn't appear this bug has been fixed. As a reference, here is the relevant
debug assembly output of `job_subsave` when KSH is compiled with `CCFLAGS` set to
`-g -O1`:
0000000000034c97 <job_subsave>:
void *job_subsave(void)
{
34c97: 53 push %rbx
struct back_save *bp = new_of(struct back_save,0);
34c98: bf 18 00 00 00 mov $0x18,%edi
34c9d: e8 34 4a 0a 00 callq d96d6 <_ast_malloc>
34ca2: 48 89 c3 mov %rax,%rbx
job_lock();
34ca5: 83 05 3c 50 13 00 01 addl $0x1,0x13503c(%rip) # 169ce8 <job+0x28>
*bp = bck;
34cac: 66 0f 6f 05 4c 5a 13 movdqa 0x135a4c(%rip),%xmm0 # 16a700 <bck>
34cb3: 00
34cb4: 0f 11 00 movups %xmm0,(%rax)
34cb7: 48 8b 05 52 5a 13 00 mov 0x135a52(%rip),%rax # 16a710 <bck+0x10>
34cbe: 48 89 43 10 mov %rax,0x10(%rbx)
bp->prev = bck.prev;
34cc2: 48 8b 05 47 5a 13 00 mov 0x135a47(%rip),%rax # 16a710 <bck+0x10>
34cc9: 48 89 43 10 mov %rax,0x10(%rbx)
bck.count = 0;
34ccd: c7 05 29 5a 13 00 00 movl $0x0,0x135a29(%rip) # 16a700 <bck>
34cd4: 00 00 00
bck.list = 0;
34cd7: 48 c7 05 26 5a 13 00 movq $0x0,0x135a26(%rip) # 16a708 <bck+0x8>
34cde: 00 00 00 00
bck.prev = bp;
34ce2: 48 89 1d 27 5a 13 00 mov %rbx,0x135a27(%rip) # 16a710 <bck+0x10>
job_unlock();
34ce9: 8b 05 f9 4f 13 00 mov 0x134ff9(%rip),%eax # 169ce8 <job+0x28>
34cef: 83 e8 01 sub $0x1,%eax
34cf2: 89 05 f0 4f 13 00 mov %eax,0x134ff0(%rip) # 169ce8 <job+0x28>
34cf8: 75 2b jne 34d25 <job_subsave+0x8e>
34cfa: 8b 3d ec 4f 13 00 mov 0x134fec(%rip),%edi # 169cec <job+0x2c>
34d00: 85 ff test %edi,%edi
34d02: 74 21 je 34d25 <job_subsave+0x8e>
34d04: c7 05 da 4f 13 00 01 movl $0x1,0x134fda(%rip) # 169ce8 <job+0x28>
When `-O2` is used instead of `-O1`, the `addl` and `sub` instructions for
incrementing and decrementing the lock are removed. GCC instead generates a
broken `mov` instruction for `job_lock` and removes the initial `sub` instruction
in job_unlock (this is also seen in Red Hat's bug report):
job_lock();
*bp = bck;
37d7c: 66 0f 6f 05 7c 79 14 movdqa 0x14797c(%rip),%xmm0 # 17f700 <bck>
37d83: 00
struct back_save *bp = new_of(struct back_save,0);
37d84: 49 89 c4 mov %rax,%r12
job_lock();
37d87: 8b 05 5b 6f 14 00 mov 0x146f5b(%rip),%eax # 17ece8 <job+0x28>
...
job_unlock();
37dc6: 89 05 1c 6f 14 00 mov %eax,0x146f1c(%rip) # 17ece8 <job+0x28>
37dcc: 85 c0 test %eax,%eax
37dce: 75 2b jne 37dfb <job_subsave+0x8b>
The original patch works around this bug by using the legacy `__sync_fetch_and_add/sub`
GCC builtins. This forces GCC to generate instructions that change the lock with
`lock addl`, `lock xadd` and `lock subl`:
job_lock();
37d9f: f0 83 05 41 6f 14 00 lock addl $0x1,0x146f41(%rip) # 17ece8 <job+0x28>
37da6: 01
...
job_unlock();
37deb: f0 0f c1 05 f5 6e 14 lock xadd %eax,0x146ef5(%rip) # 17ece8 <job+0x28>
37df2: 00
37df3: 83 f8 01 cmp $0x1,%eax
37df6: 74 08 je 37e00 <job_subsave+0x70>
...
37e25: 74 11 je 37e38 <job_subsave+0xa8>
37e27: f0 83 2d b9 6e 14 00 lock subl $0x1,0x146eb9(%rip) # 17ece8 <job+0x28>
While this does work, it isn't portable. This patch implements a different
workaround for this compiler bug. If `job_lock` is put at the beginning of
`job_subsave`, GCC will generate the required `addl` and `sub` instructions:
job_lock();
37d67: 83 05 7a 5f 14 00 01 addl $0x1,0x145f7a(%rip) # 17dce8 <job+0x28>
...
job_unlock();
37dbb: 83 e8 01 sub $0x1,%eax
37dbe: 89 05 24 5f 14 00 mov %eax,0x145f24(%rip) # 17dce8 <job+0x28>
It is odd that moving a single line of code fixes this problem, although
GCC _should_ have generated these instructions from the original code anyway.
I'll note that this isn't the only way to get these instructions to generate.
The problem also seems to go away when inserting almost anything else inside
of the code for `job_subsave`. This is just a simple workaround for a strange
compiler bug.
This bug was previously reported in att/ast#37.
Ksh ignores `-r` when `read -r -d` is run because when
the bit for `D_FLAG` is set, the bit for `R_FLAG` is unset
as a side effect of setting `D_FLAG`. The following set
of commands fails to print a backslash:
$ printf '\\\000' | read -r -d ''
$ echo $REPLY
The fix for this bug is to set `D_FLAG` with `D_FLAG + 1`,
which prevents `R_FLAG` from being unset. This bugfix
has been backported from ksh93v- 2013-10-10-alpha.
src/cmd/ksh93/bltins/read.c:
- Set `D_FLAG` with `D_FLAG + 1` to prevent the bit for
`R_FLAG` from being unset.
src/cmd/ksh93/tests/builtins.sh:
- Add the regression test for `read -r -d` from ksh93v-.
Running shbench after undoing the incorrect subshell optimisation
showed that the performance of ${ subshare; }-type command
substitutions went down very slightly, but consistently.
The main purpose of using this ksh-specific type of command
substitution vs. a normal one is performance. Thus, it *is*
appropriate to eke every last bit of performance out of it that
we can, provided correctness is completely preserved.
It is also a type of command substitution where every change is
supposed to be shared with the main shell environment; only command
output is captured in a subshell-like fashion.
Thus, on the face of it, it would be a logical optimisation for
sh_assignok() to avoid bothering with saving a subshell context for
variables if we're in a subshare.
Lo and behold, applying it does not introduce any regress fails.
Here are my average results of the braces.ksh benchmark from
shbench <http://fossil.0branch.com/csb/tktnew> against stock
/bin/ksh 93u+ vs. current 93u+m (same compiler flags),
100 runs pre-optimisation and 100 runs post-optimisation:
Stock /bin/ksh: Pre-optimisation (at 3d38270b):
93u+: 0.743 secs 93u+m: 0.739 secs
Stock /bin/ksh: Post-optimisation (now):
93u+: 0.744 secs 93u+m: 0.726 secs
The left column shows only a small margin of error with 100 runs;
the right one shows a very small, but not insignificant difference.
However, these tests were not very rigorous with 100 runs each.
If anyone wants to do it properly, please report results to
korn-shell@googlegroups.com. I'm happy enough with this, though.
Thanks to Joerg van den Hoff for providing shbench, without
which it would not have occurred to me to try this.
src/cmd/ksh93/sh/subshell.c: sh_assignok():
- Don't bother if we're in a ${ subshare; }.
The fact that every ksh builtin command self-documents with the
options --help and --man (and others, see 'getopts --man'; but
these are the essential ones) is poorly known; the information is
buried somewhere deep in the sh.1 manual page, and is incomplete at
that. None of the terse usage messages displayed on error point the
user to these options. So let's fix that.
src/lib/libast/misc/optget.c:
- Change generic 'options' placeholder, used in all terse usage
messages, to 'options | --help | --man'.
src/cmd/ksh93/sh.1:
- Edit documentation of --man/-?, adding documentation on --help
which was completely undocumented. Refer to 'getopts --man' for
more advanced info.
- Separate these from the (important) documentation on special
builtins using a paragraph break.
