Fix BUG_CASELIT: pattern matching as literal string in 'case'
This fixes an undocumented 'case' pattern matching misbehaviour
(labelled BUG_CASELIT in modernish) that goes back to the original
Bourne shell, but wasn't discovered until 2018.
If a pattern doesn't match as a pattern, it's tried again as a
literal string. This breaks common validation use cases, such as:
n='[0-9]'
case $n in
( [0-9] ) echo "$n is a number" ;;
esac
would output "[0-9] is a number" as the literal string fallback
matches the pattern. As this misbehaviour was never documented
anywhere (not for Bourne, ksh88, or ksh93), and it was never
replicated in other shells (not even in ksh88 clones pdksh and
mksh), it is unlikely any scripts rely on it.
Of course, a literal string fallback, should it be needed, is
trivial to implement correctly without this breakage:
case $n in
( [0-9] | "[0-9]") echo "$n is a number or the number pattern" ;;
esac
src/cmd/ksh93/sh/xec.c:
- Remove trim_eq() function responsible for implementing the
misbehaviour described above.
NEWS:
- Added. Document this bugfix.
Ref.:
- The problem: thread starting at
https://www.mail-archive.com/austin-group-l@opengroup.org/msg02127.html
- The solution, thanks to George Koehler: comments/commits in
https://github.com/att/ast/issues/476
- Modernish BUG_CASELIT bug test & documentation:
https://github.com/modernish/modernish/commit/b2024ae3
(cherry picked from commit 8d6c8ce69884767a160c1e20049e77bdd849c248
with some extra edits to NEWS to upate the info for this reboot)
2020-06-11 15:14:31 +00:00
|
|
|
This documents significant changes in the 93u+m branch of AT&T ksh93.
|
|
|
|
For full details, see the git log at:
|
|
|
|
https://github.com/modernish/ksh
|
|
|
|
|
|
|
|
Any uppercase BUG_* names are modernish shell bug IDs.
|
|
|
|
|
Fix unsetting special vars in subshells (re: efa31503, 8b7f8f9b)
This fixes (or at least works around) a bug that caused special
variables such as PATH, LANG, LC_ALL, LINENO, etc. to lose their
effect after being unset in a subshell.
For example:
(unset PATH; PATH=/dev/null; ls); : wrongly ran 'ls'
(unset LC_ALL; LC_ALL=badlocale); : failed to print a diagnostic
This is yet another problem with non-forking/virtual subshells. If
you forced the subshell to fork (one way of doing this is using the
'ulimit' builtin, e.g. ulimit -t unlimited) before unsetting the
special variable, the problem vanished.
I've tried to localise the problem. I suspect the sh_assignok()
function, which is called from unall(), is to blame. This function
is supposed to make a copy of a variable node in the virtual
subshell's variable tree. Apparently, it fails to copy the
associated permanent discipline function settings (stored in the
np->nvfun->disc pointer) that gave these variables their special
effect, and which survive unset. However, my attempts to fix that
have been unsuccessful. If anyone can figure out a fix, please send
a patch/pull request!
Data point: This bug existed in 93u 2011-02-08, but did not yet
exist in M-1993-12-28-s+. So it is a regression.
Meanwhile, pending a proper fix, this commit adds a safe
workaround: it forces a non-forked subshell to fork before
unsetting such a special variable.
src/cmd/ksh93/bltins/typeset.c: unall():
- If we're in a non-forked, non-${ ...; } subshell, then before
unsetting any variable, check for variables with internal
trap/discipline functions, and call sh_subfork() if any are
found. To avoid crashing, this must be done before calling
sh_pushcontext(), so we need to loop through the args separately.
src/cmd/ksh93/tests/variables.sh:
- Remove the 'ulimit' that forced the fork; we do this in C now.
(cherry picked from commit 21b1a67156582e3cbd36936f4af908bb45211a4b)
2020-06-05 13:31:26 +00:00
|
|
|
2020-06-05:
|
|
|
|
|
|
|
|
- Fix a bug that caused special variables such as PATH, LANG, LC_ALL,
|
|
|
|
etc. to lose their effect after being unset in a subshell. For example:
|
|
|
|
(unset PATH; PATH=/dev/null; ls); : wrongly ran 'ls'
|
|
|
|
(unset LC_ALL; LC_ALL=badlocale); : failed to print a diagnostic
|
|
|
|
|
Fix BUG_KBGPID: $! was not updated under certain conditions
The $! special parameter was not set if a background job
(somecommand &) or co-process (somecommand |&) was launched as the
only command within a braces block with an attached redirection,
for example:
{
somecommand &
} >&2
With the bug, $! was unchanged; now it contains the PID of
somecommand.
Ref.: https://github.com/att/ast/issues/1357
src/cmd/ksh93/sh/parse.c: item():
- When processing redirections following a compound command, always
create a parent node with the TSETIO (I/O redirection) token.
Before this commit, if the last command was of type TFORK (and
the last command only tested as TFORK if the bg job or coprocess
was the only command in a braces block, because the ksh parser
optimises away the braces in that case), then the parent node was
created with the TFORK token instead.
I have no idea what David Korn's intention was with that, but
this is clearly very wrong. Creating another TFORK node when
parsing the redirection caused sh_exec() in sh/xec.c to execute
the redirection in an extra forked, non-background subshell.
Since redirections are executed before anything else, this
subshell is what then launched the background job between the
braces, so $! (a.k.a. shp->bckpid) was updated in that subshell
only, and never in the main shell. The extra subshell also
prevented the background job from being noticed by job control
on interactive shells.
So, the fix is simply to remove the broken test for TFORK.
src/cmd/ksh93/tests/variables.sh:
- Add regression tests for a bg job and a co-process as the only
command within a braces block with attached redirection.
(cherry picked from commit ffe5df30e69f7b596941a98498014d8e838861f2)
2020-06-04 03:19:59 +00:00
|
|
|
2020-06-04:
|
|
|
|
|
|
|
|
- Fix BUG_KBGPID: the $! special parameter was not set if a background job
|
|
|
|
(somecommand &) or co-process (somecommand |&) was launched as the only
|
|
|
|
command within a braces block with an attached redirection, for example:
|
|
|
|
{
|
|
|
|
somecommand &
|
|
|
|
} >&2
|
|
|
|
With the bug, $! was unchanged; now it contains the PID of somecommand.
|
|
|
|
|
2020-05-31 13:29:15 +00:00
|
|
|
2020-05-31:
|
|
|
|
|
|
|
|
- Fix a bug in autoloading functions. Directories in the path search list
|
|
|
|
which should be skipped (e.g. because they don't exist) did not interact
|
|
|
|
correctly with autoloaded functions, so that a function to autoload was
|
|
|
|
not always found correctly.
|
|
|
|
Details: https://github.com/att/ast/issues/1454
|
|
|
|
|
2020-05-30 14:21:30 +00:00
|
|
|
2020-05-30:
|
|
|
|
|
|
|
|
- Fix POSIX compliance of 'test'/'[' exit status on error. The command now
|
|
|
|
returns status 2 instead of 1 when given an invalid number or arithmetic
|
|
|
|
expression, e.g.:
|
|
|
|
[ 123 -eq 123x ]; echo $?
|
|
|
|
now outputs 2 instead of 1.
|
|
|
|
|
Fix redefining & unsetting functions in subshells (BUG_FNSUBSH)
Functions can now be correctly redefined and unset in subshell
environments (such as ( ... ), $(command substitutions), etc).
Before this fix, attempts to do this were silently ignored (!!!),
causing the wrong code (i.e.: the function by the same name from
the parent shell environment) to be executed.
Redefining and unsetting functions within "shared" command
substitutions of the form '${ ...; }' is also fixed.
Prior discussion: https://github.com/att/ast/issues/73
src/cmd/ksh93/sh/parse.c:
- A fix from George Koelher (URL above). He writes:
| The parser can set t->comnamp to the wrong function.
| Suppose that the shell has executed
| foo() { echo WRONG; }
| and is now parsing
| (foo() { echo ok; } && foo)
| The parser was setting t->comnamp to the wrong foo. [This
| fix] doesn't set t->comnamp unless it was a builtin. Now the
| subshell can't call t->comnamp, so it looks for foo and finds
| the ok foo in the subshell's function tree.
src/cmd/ksh93/bltins/typeset.c:
- Unsetting functions in a virtual/non-forked subshell still
doesn't work: nv_open() fails to find the function. To work
around this problem, make 'unset -f' fork the subshell into its
own process with sh_subfork().
- The workaround exposed another bug: if we unset a function in a
subshell tree that overrode a function by the same name in the
main shell, then nv_delete() exposes the function from the main
shell scope. Since 'unset -f' now always forks a subshell, the
fix is to simply walk though troot's parent views and delete any
such zombie functions as well. (Without this, the 4 'more fun'
tests in tests/subshell.sh fail.)
src/cmd/ksh93/sh/subshell.c: sh_subfuntree():
- Fix function (re)definitions and unsetting in "shared" command
substitutions of the form '${ commandlist; }' (i.e.: if
sp->shp->subshare is true). Though internally this is a weird
form of virtual subshell, the manual page says it does not
execute in a subshell (meaning, all changes must survive it), so
a subshell function tree must not be created for these.
src/cmd/ksh93/tests/subshell.sh:
- Add regression tests related to these bugfixes. Test unsetting
and redefining a function in all three forms of virtual subshell.
(cherry picked from commit dde387825ab1bbd9f2eafc5dc38d5fd0bf9c3652)
2020-05-29 07:27:20 +00:00
|
|
|
2020-05-29:
|
|
|
|
|
|
|
|
- Fix BUG_FNSUBSH: functions can now be correctly redefined and unset in
|
|
|
|
subshell environments (such as ( ... ), $(command substitutions), etc).
|
|
|
|
Before this fix, this was silently ignored, causing the function by the
|
|
|
|
same name from the parent shell environment to be executed instead.
|
|
|
|
fn() { echo mainsh; }
|
|
|
|
(fn() { echo subsh; }; fn); fn
|
|
|
|
This now correctly outputs "subsh mainsh" instead of "mainsh mainsh".
|
|
|
|
ls() { echo "ls executed"; }
|
|
|
|
(unset -f ls; ls); ls
|
|
|
|
This now correctly lists your directory and then prints "ls executed",
|
|
|
|
instead of printing "ls executed" twice.
|
|
|
|
|
Fix unsetting aliases in subshells
Aliases can now be correctly unset within subshell environments
(such as ( ... ), $(command substitutions), etc), as well as
non-subshell "shared" command substitutions (${ ...; }). Before,
attempts to unset aliases within these were silently ignored.
Prior discussion: https://github.com/att/ast/issues/108
Subshell alias trees are only referenced in a few places in the
code, *and* have always been broken, so this commit gets rid of the
whole notion of a subshell alias tree. Instead, there is now just
one flat alias tree, and subshells fork into a separate process
when aliases are set or unset within them. It is not really
conceivable that this could be a performance-sensitive operation,
or even a common one, so this is a clean fix with no downside.
src/cmd/ksh93/include/defs.h:
- Remove sh_subaliastree() definition.
src/cmd/ksh93/sh/subshell.c:
- Remove salias element (pointer to subshell alias tree) from
subshell struct.
- Remove sh_subaliastree() function.
- sh_subshell(): Remove alias subshell tree cleanup.
src/cmd/ksh93/bltins/typeset.c:
- b_alias(): If in subshell, fork before setting alias.
- b_unalias(): If in subshell, fork before unsetting alias.
- unall(): Remove sh_subaliastree() call.
src/cmd/ksh93/sh/name.c:
- nv_open(): Remove sh_subaliastree() call.
src/cmd/ksh93/tests/subshell.sh:
- Add regression tests for unsetting or redefining aliases within
subshells.
(cherry picked from commit 12a15605b9521a2564a6e657905705a060e89095)
2020-05-29 07:27:53 +00:00
|
|
|
- Fix a similar bug with aliases. These can now be correctly unset
|
|
|
|
in subshell environments.
|
|
|
|
|
2020-05-20 21:42:00 +00:00
|
|
|
2020-05-21:
|
|
|
|
|
|
|
|
- Fix truncating of files with the combined redirections '<>;file' and
|
|
|
|
'<#pattern'. The bug was caused by out-of-sync streams.
|
|
|
|
Details and discussion: https://github.com/att/ast/issues/61
|
|
|
|
|
2020-05-30 14:55:30 +00:00
|
|
|
- Patched code injection vulnerability CVE-2019-14868. As a result, you can
|
2020-05-21 12:27:51 +00:00
|
|
|
no longer use expressions in imported numeric environment variables; only
|
|
|
|
integer literals are allowed.
|
|
|
|
|
Fix bugs in testing if a parameter is set
This fixes three related bugs:
1. Expansions like ${var+set} remained static when used within a
'for', 'while' or 'until' loop; the expansions din't change
along with the state of the variable, so they could not be used
to check whether a variable is set within a loop if the state of
that variable changed in the course of the loop. (BUG_ISSETLOOP)
2. ${IFS+s} always yielded 's', and [[ -v IFS ]] always yielded
true, even if IFS is unset. (BUG_IFSISSET)
3. IFS was incorrectly exempt from '-u' ('-o nounset') checks.
src/cmd/ksh93/sh/macro.c: varsub():
- When getting a node pointer (np) to the parameter to test,
special-case IFS by checking if it has a value and not setting
the pointer if not. The node to IFS always exists, even after
'unset -v IFS', so before this fix it always followed the code
path for a parameter that is set. This fixes BUG_IFSISSET for
${IFS+s} and also fixes set -u (-o nounset) with IFS.
- Before using the 'nv_isnull' macro to check if a regular variable
is set, call nv_optimize() if needed. This fixes BUG_ISSETLOOP.
Idea from Kurtis Rader: https://github.com/att/ast/issues/1090
Of course this only works if SHOPT_OPTIMIZE==1 (the default),
but if not, then this bug is not triggered in the first place.
- Add some comments for future reference.
src/cmd/ksh93/bltins/test.c: test_unop():
- Fix BUG_IFSISSET for [[ -v IFS ]]. The nv_optimize() method
doesn't seem to have any effect here, so the only way that I can
figure out is to special-case IFS, nv_getval()'ing it to check if
IFS has a value in the current scope.
src/cmd/ksh93/tests/variables.sh:
- Add regression tests for checking if a varariable is set within a
loop, within and outside a function with that variable made local
(to check if the scope is honoured). Repeat these tests for a
regular variable and for IFS, for ${foo+set} and [[ -v foo ]].
(cherry picked from commit a2cf79cb98fa3e47eca85d9049d1d831636c9b16)
2020-05-20 14:50:43 +00:00
|
|
|
2020-05-20:
|
|
|
|
|
|
|
|
- Fix BUG_ISSETLOOP. Expansions like ${var+set} remained static when used
|
|
|
|
within a 'for', 'while' or 'until' loop; the expansions din't change along
|
|
|
|
with the state of the variable, so they could not be used to check whether a
|
|
|
|
variable is set within a loop if the state of that variable changed in the
|
|
|
|
course of the loop.
|
|
|
|
|
|
|
|
- Fix BUG_IFSISSET. ${IFS+s} always yielded 's', and [[ -v IFS ]] always
|
|
|
|
yielded true, even if IFS is unset. This applied to IFS only.
|
|
|
|
|
2020-05-19 14:49:56 +00:00
|
|
|
2020-05-19:
|
|
|
|
|
|
|
|
- Fix 'command -p'. The -p option causes the operating system's standard
|
|
|
|
utilities path (as output by 'getconf PATH') to be searched instead of $PATH.
|
|
|
|
Before this fix, this was broken on non-interactive shells as the internal
|
|
|
|
variable holding the default PATH value was not correctly initialised.
|
|
|
|
|
Fix 'test -t 1' in $(command substitutions)
Standard output (FD 1) tested as being on a terminal within a
command substitution, which makes no sense as the command
substitution is supposed to be catching standard output.
ksh -c 'v=$(echo begincomsub
[ -t 1 ] && echo oops
echo endcomsub)
echo "$v"'
This should not output "oops".
This is one of the many bugs with ksh93 virtual (non-forked)
subshells. On the abandoned Vashist/Rader ksh2020 branch, this bug
was fixed by changing quite a lot of code, which introduced and/or
exposed another bug:
https://github.com/att/ast/issues/1079
https://github.com/att/ast/commit/8e1e405e
https://github.com/att/ast/issues/1088
That issue was unresolved when the ksh2020 branch was abandoned.
The safer and more conservative fix is simply forcing the subshell
to fork if we're in a non-forked command substitution and testing
'-t 1'. It is hard to imagine a situation where this would cause a
noticable performance hit.
Note that this fix does not affect ksh93-specific "shared"
non-subshell ${ command substitutions; } which are executed in the
main shell environment, so that variables survive, etcetera.
'test -t 1' continues to wrongly return true there, but command
substitutions of that form cannot be forked because that would
defeat their purpose.
src/cmd/ksh93/bltins/test.c:
- Fix 'test -t 1', '[ -t 1 ]' and '[[ -t 1 ]]' by forking the
current subshell if it is a virtual/non-forked subshell
(shp->subshell), and a command substitution (shp->comsub), but
NOT a "shared" ${ command substitution; } (!shp->subshare).
src/cmd/ksh93/tests/bracket.sh:
- Add two regression tests for this issue, which were adapted from
the Vashist/Rader ksh2020 branch.
NEWS, src/cmd/ksh93/include/version.h:
- Update.
(cherry picked from commit b8ef05e457ead65b83417699b8dd8632f855e2fa)
2020-05-16 19:06:49 +00:00
|
|
|
2020-05-16:
|
|
|
|
|
|
|
|
- Fix 'test -t 1', '[ -t 1 ]', '[[ -t 1 ]]' in command substitutions.
|
|
|
|
Standard output (file descriptor 1) tested as being on a terminal within a
|
|
|
|
command substitution, which makes no sense as the command substitution is
|
|
|
|
supposed to be catching standard output.
|
|
|
|
v=$(echo begincomsub
|
|
|
|
[ -t 1 ] && echo oops
|
|
|
|
echo endcomsub)
|
|
|
|
echo "$v"
|
|
|
|
This now does not output "oops".
|
|
|
|
|
2020-05-14 10:36:16 +00:00
|
|
|
2020-05-14:
|
|
|
|
|
|
|
|
- Fix syncing history when print -s -f is used. For example, the
|
|
|
|
following now correctly adds a 'cd' command to the history:
|
|
|
|
print -s -f 'cd -- %q\n' "$PWD"
|
|
|
|
Ref.: https://github.com/att/ast/issues/425
|
|
|
|
https://github.com/att/ast/pull/442
|
|
|
|
|
2020-05-16 14:04:35 +00:00
|
|
|
- Fix BUG_PUTIOERR: Output builtins now correctly detect and report
|
|
|
|
input/output errors. This allows scripts to check for a nonzero exit
|
|
|
|
status on the 'print', 'printf' and 'echo' builtins and prevent possible
|
|
|
|
infinite loops if SIGPIPE is ignored.
|
|
|
|
|
2020-05-14 16:41:26 +00:00
|
|
|
- Add a convenient bin/run_ksh_tests script to the source tree that
|
|
|
|
sets up the necessary environment and runs the ksh regression tests.
|
|
|
|
|
Fix BUG_CASELIT: pattern matching as literal string in 'case'
This fixes an undocumented 'case' pattern matching misbehaviour
(labelled BUG_CASELIT in modernish) that goes back to the original
Bourne shell, but wasn't discovered until 2018.
If a pattern doesn't match as a pattern, it's tried again as a
literal string. This breaks common validation use cases, such as:
n='[0-9]'
case $n in
( [0-9] ) echo "$n is a number" ;;
esac
would output "[0-9] is a number" as the literal string fallback
matches the pattern. As this misbehaviour was never documented
anywhere (not for Bourne, ksh88, or ksh93), and it was never
replicated in other shells (not even in ksh88 clones pdksh and
mksh), it is unlikely any scripts rely on it.
Of course, a literal string fallback, should it be needed, is
trivial to implement correctly without this breakage:
case $n in
( [0-9] | "[0-9]") echo "$n is a number or the number pattern" ;;
esac
src/cmd/ksh93/sh/xec.c:
- Remove trim_eq() function responsible for implementing the
misbehaviour described above.
NEWS:
- Added. Document this bugfix.
Ref.:
- The problem: thread starting at
https://www.mail-archive.com/austin-group-l@opengroup.org/msg02127.html
- The solution, thanks to George Koehler: comments/commits in
https://github.com/att/ast/issues/476
- Modernish BUG_CASELIT bug test & documentation:
https://github.com/modernish/modernish/commit/b2024ae3
(cherry picked from commit 8d6c8ce69884767a160c1e20049e77bdd849c248
with some extra edits to NEWS to upate the info for this reboot)
2020-06-11 15:14:31 +00:00
|
|
|
2020-05-13:
|
|
|
|
|
|
|
|
- Fix BUG_CASELIT: an undocumented 'case' pattern matching misbehaviour that
|
|
|
|
goes back to the original Bourne shell, but wasn't discovered until 2018.
|
|
|
|
If a pattern doesn't match as a pattern, it was tried again as a literal
|
|
|
|
string. This broke common validation use cases, e.g.:
|
|
|
|
n='[0-9]'
|
|
|
|
case $n in
|
|
|
|
( [0-9] ) echo "$n is a number" ;;
|
|
|
|
esac
|
|
|
|
would output "[0-9] is a number" as the literal string fallback matches the
|
|
|
|
pattern. As this misbehaviour was never documented anywhere (not for Bourne,
|
|
|
|
ksh88, or ksh93), and it was never replicated in other shells (not even in
|
|
|
|
ksh88 clones pdksh and mksh), it is unlikely any scripts rely on it.
|
|
|
|
Of course, a literal string fallback, should it be needed, is trivial to
|
|
|
|
implement correctly without this breakage:
|
|
|
|
case $n in
|
|
|
|
( [0-9] | "[0-9]") echo "$n is a number or the number pattern" ;;
|
|
|
|
esac
|
2020-05-13 14:00:33 +00:00
|
|
|
Ref.: https://github.com/att/ast/issues/476
|
|
|
|
|
|
|
|
- Fix BUG_REDIRIO: ksh used to redirect standard output by default when no
|
|
|
|
file descriptor was specified with the rarely used '<>' reading/writing
|
|
|
|
redirection operator. It now redirects standard input by default, as POSIX
|
|
|
|
specifies and as all other POSIX shells do. To redirect standard output
|
|
|
|
for reading and writing, you now need '1<>'.
|
|
|
|
Ref.: https://github.com/att/ast/issues/75
|
|
|
|
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_07
|