A previous test left LC_ALL set, causing a spurious failure in one
of the newly added tests. The bug was masked because another test
conditional upon SHOPT_MULTIBYTE unset LC_ALL again. The fix is to
unset all locale variables before testing, just to be sure.
A side effect of the bug fixed in 2a835a2d caused the DEBUG trap
action to appear to be inherited by subshells, but in a buggy way
that could crash the shell. After the fix, the trap is reset in
subshells along with all the others, as it should be. Nonetheless,
as that bug was present for years, some have come to rely on it.
This commit implements that functionality properly. When the new
--functrace option is turned on, DEBUG trap actions are now
inherited by subshells as well as ksh function scopes. In addition,
since it makes logical sense, the new option also causes the
-x/--xtrace option's state to be inherited by ksh function scopes.
Note that changes made within the scope do not propagate upwards;
this is different from bash.
(I've opted against adding a -T short-form equivalent as on bash,
because -T was formerly a different option on 93u+ (see 63c55ad7)
and on mksh it has yet anohter a different meaning. To minimise
confusion, I think it's best to have the long-form name only.)
src/cmd/ksh93/include/shell.h,
src/cmd/ksh93/data/options.c:
- Add new "functrace" (SH_FUNCTRACE) long-form shell option.
src/cmd/ksh93/sh/subshell.c: sh_subshell():
- When functrace is on, copy the parent's DEBUG trap action into
the virtual subshell scope after resetting the trap actions.
src/cmd/ksh93/sh/xec.c: sh_funscope():
- When functrace is on and xtrace is on in the parent scope, turn
it on in the child scope.
- Same DEBUG trap action handling as in sh_subshell().
Resolves: https://github.com/ksh93/ksh/issues/162
After the previous commit, one inconsistency was left. Assignments
preceding special built-ins and POSIX functions (which persist past
the command :-/) caused pre-existing attributes of the respective
variables to be cleared.
$ (f() { typeset -p n; }; typeset -i n; n=3+4 f)
n=3+4
(expected output: 'typeset -i n=7')
This problem was introduced shortly before the release of ksh 93u+,
in version 2012-05-04, by adding these lines of code to the code
for processing preceding assignments in sh_exec():
src/cmd/ksh93/sh/xec.c:
1055: if(np)
1056: flgs |= NV_UNJUST;
So, for special and declaration commands and POSIX functions, the
NV_UNJUST flag is passed to nv_open(). In older ksh versions, this
flag cleared justify attributes only, but in early 2012 it was
repurposed to clear *all* attributes -- without changing the name
or the relevant comment in name.h, which are now both misleading.
The reason for setting this flag in xec.c was to deal with some
bugs in 'typeset'. Removing those two lines causes regressions:
attributes.sh[316]: FAIL: typeset -L should not preserve old attributes
attributes.sh[322]: FAIL: typeset -R should not preserve old attributes
attributes.sh[483]: FAIL: typeset -F after typeset -L fails
attributes.sh[488]: FAIL: integer attribute not cleared for subsequent typeset
Those are all typeset regressions, which suggests this fix was
relevant to typeset only. This is corroborated by the relevant
AT&T ksh93/RELEASE entry:
12-04-27 A bug in which old attributes were not cleared when
assigning a value using typeset has been fixed.
So, to fix this 2012 regression without reintroducing the typeset
regressions, we need to set the NV_UNJUST flag for invocations of
the typeset family of commands only. This is changed in xec.c.
While we're at it, we might as well rename that little-used flag to
something that reflects its current purpose: NV_UNATTR.
Reproducer:
$ typeset -i NUM=123
$ (NUM=3+4 env; :)|grep ^NUM=
NUM=3+4
$ (NUM=3+4 env)|grep ^NUM=
NUM=7
The correct output is NUM=7. This is also the output if ksh is
compiled with SHOPT_SPAWN disabled, suggesting that the problem is
here in sh_ntfork(), where the invocation-local environment list is
set using a sh_scope() call:
src/cmd/ksh93/sh/xec.c:
3496: if(t->com.comset)
3497: {
3498: scope++;
3499: sh_scope(t->com.comset,0);
3500: }
Analysis:
When ksh is compiled with SHOPT_SPAWN disabled, external commands
are always run using the regular forking mechanism. First the shell
forks, then in the fork, the preceding assignments list (if any)
are executed and exported in the main scope. Replacing global
variables is not a problem as the variables are exported and the
forked shell is then replaced by the external command using
execve(2).
But when using SHOPT_SPAWN/sh_ntfork(), this cannot be done as the
fork(2) use is replaced by posix_spawn(2) which does not copy the
parent process into the child, therefore it's not possible to
execute anything in the child before calling execve(2). Which means
the preceding assignments list must be processed in the parent, not
the child. Which makes overwriting global variables a no-no.
To avoid overwriting global variables, sh_ntfork() treats preceding
assignments like local variables in functions, which means they do
not inherit any attributes from the parent scope. That is why the
integer attribute is not honoured in the buggy reproducers.
And this is not just an issue for external commands. sh_scope() is
also used for assignments preceding a built-in command. Which is
logical, as those don't create a process at all.
src/cmd/ksh93/sh/xec.c:
1325: if(argp)
1326: {
1327: scope++;
1328: sh_scope(argp,0);
1329: }
Which means this bug exists for them as well, regardless of whether
SHOPT_SPAWN is compiled in.
$ /bin/ksh -c 'typeset -i NUM; NUM=3+4 command eval '\''echo $NUM'\'
3+4
(expected: 7, as on mksh and zsh)
So, the attributes from the parent scope will need to be copied
into the child scope. This should be done in nv_setlist() which is
called from sh_scope() with both the NV_EXPORT and NV_NOSCOPE flags
passed. Those flag bits are how we can recognise the need to copy
attributes.
Commit f6bc5c0 fixed a similar inconsistency with the check for the
read-only attribute. In fact, the bug fixed there was simply a
specific instance of this bug. The extra check for readonly was
because the readonly attribute was not copied to the temporary
local scope. So that fix is now replaced by the more general fix
for this bug.
src/cmd/ksh93/sh/name.c: nv_setlist():
- Introduce a 'vartree' local variable to avoid repetitive
'sh.prefix_root ? sh.prefix_root : sh.var_tree' expressions.
- Use the NV_EXPORT|NV_NOSCOPE flag combination to check if parent
scope attributes need to be copied to the temporary local scope
of an assignment preceding a command.
- If so, copy everything but the value itself: attributes (nvflag),
size parameter (nvsize), discipline function pointer (nvfun) and
the name pointer (nvname). The latter is needed because some
code, at least put_lang() in init.c, compares names by comparing
the pointers instead of the strings; copying the nvname pointer
avoids a regression in tests/locale.sh.
src/cmd/ksh93/sh/xec.c: local_exports():
- Fix a separate bug exporting attributes to a new ksh function
scope, which was previously masked by the other bug. The
attributes (nvflag) were copied *after* nv_putval()ing the value,
which is incorrect as the behaviour of nv_putval() is influenced
by the attributes. But here, we're copying the value too, so we
can simplify the whole function by using nv_clone() instead. This
may also fix other corner cases. (re: c1994b87)
Resolves: https://github.com/ksh93/ksh/issues/465
Those debug messages were written straight to standard error
without going through the shell's error/warning message mechanism.
They look like debug messages that aren't really supposed to appear
(no one ever sees them AFAIK).
This commit removes them, in some cases replacing them with abort()
calls so that a stack trace is generated if the unexpected thing
happens, instead of the shell continuing in an inconsistent state.
A systematic grepping of the extern function definitions in
src/cmd/ksh93/include/*.h revealed more functions that either don't
exist or are not used anywhere. Some of them have never seen any
use in the entire ksh93-history repo (i.e. since 1995). They were
also all undocumented, so it's unlikely third-party custom
built-ins rely on them.
Bug 1: as of 960a1a99, floating point literals were no longer
recognised when importing variables from the environment. The
attribute was still imported but the value reverted to zero:
$ (export LC_NUMERIC=C; typeset -xF5 num=7.75; \
ksh -c 'typeset -p num')
typeset -x -F 5 num=0.00000
Bug 2 (inherited from 93u+): The code that imported variable
attributes from the environment only checked '.' to determine
whether the float attribute should be set. It should check the
current radix point instead.
$ (export LC_NUMERIC=debug; typeset -xF5 num=7,75; \
ksh -c 'typeset -p num')
typeset -x -i num=0
...or, after fixing bug 1 only, the output is:
typeset -x -i num=75000
src/cmd/ksh93/sh/arith.c: sh_strnum():
- When importing untrusted env vars at init time, handle not only
"base#value" literals using strtonll, but also floating point
literals using strtold. This fixes the bug without reallowing
arbitary expressions. (re: 960a1a99)
- When not initialising, use sh.radixpoint (see f0386a87) instead
of '.' to help decide whether to evaluate an arith expression.
src/cmd/ksh93/sh/init.c: env_import_attributes():
- Use sh.radixpoint instead of '.' to check for a decimal fraction.
(This code is needed because doubles are exported as integers for
ksh88 compatibility; see attstore() in name.c.)
The GETDECIMAL(x) macro in src/cmd/ksh93/features/locale uses a
global static variable 'lp' to get the localeconv() results.
Several functions in ksh use local variables called 'lp'. It's dumb
luck that this hasn't conflicted yet; it's a bug waiting to happen.
It's also slightly inefficient as it calls localeconv() every time.
In addition there is a redundant 'sh.decomma' flag that is set to 1
if the radix point is a comma, but only once, by sh_init(). It is
not updated if the locale changes. That is not correct.
This commit gets rid of both and implements a new approach instead:
store the radix point in sh.radixpoint at init time in sh_init(),
and in the put_lang() locale discipline function, reinitialise
sh.radixpoint whenever LC_ALL or LC_NUMERIC changes. The rest of
the code can then simply use sh.radixpoint without worry.
The referenced commit introduced at least one bug: EXIT traps were
not triggered if a special builtin threw an error in a namespace
that is within a virtual subshell.
src/cmd/ksh93/sh/xec.c: sh_exec(): TNSPACE:
- When an error occurs, siglongjmp to the previous saved
environment; it is not correct to sh_exit directly.
src/cmd/ksh93/tests/namespace.sh:
- Remove the forkign workaround and the TODO where I incorrectly
blamed this problem on the virtual subshell mechanism.
typeset -g allows directly manipulating the attributes of variables
at the global level from any context. This feature already exists
on bash 4.2 and later.
mksh (R50+), yash and zsh have this flag as well, but it's slightly
different: it ignores the current local scope, but a parent local
scope from a calling function may still be used -- whereas on bash,
'-g' always refers to the global scope. Since ksh93 uses static
scoping (see III.Q28 at <http://kornshell.com/doc/faq.html>), only
the bash behaviour makes sense here.
Note that the implementation needs to be done both in nv_setlist()
(name.c) and in b_typeset() (typeset.c) because assignments are
executed before the typeset built-in itself. Hence also the
pre-parsing of typeset options in sh_exec().
src/cmd/ksh93/include/nval.h:
- Add new NV_GLOBAL bit flag, using a previously unused bit that
still falls within the 32-bit integer range.
src/cmd/ksh93/sh/xec.c: sh_exec():
- When pre-parsing typeset flags, make -g pass the NV_GLOBAL flag
to the nv_setlist() call that processes shell assignments prior
to running the command.
src/cmd/ksh93/sh/name.c: nv_setlist():
- When the NV_GLOBAL bit flag is passed, save the current variable
tree pointer (sh.var_tree) as well as the current namespace
(sh.namespace) and temporarily set the former to the global
variable tree (sh.var_base) and the latter to NULL. This makes
assignments global and ignores namesapces.
src/cmd/ksh93/bltins/typeset.c:
- b_typeset():
- Use NV_GLOBAL bit flag for -g.
- Allow combining -n with -g, permitting 'typeset -gn var' or
'nameref -g var' to create a global nameref from a function.
- Do not allow a nonsensical use of -g when using nested typeset
to define member variables of 'typeset -T' types. (Type method
functions can still use it as normal.)
- setall():
- If NV_GLOBAL is passed, use sh.var_base and deactivate
sh.namespace as in nv_setlist(). This allows attributes
to be set correctly for global variables.
src/cmd/ksh93/tests/{functions,namespace}.sh:
- Add regression tests based on reproducers for problems found
by @hyenias in preliminary versions of this feature.
Resolves: https://github.com/ksh93/ksh/issues/479
Reproducer:
$ namespace test { x=123; typeset -g x=456; }
$ echo $x ${.test.x}
456 123
$ namespace test { typeset -Q; }
arch/darwin.i386-64/bin/ksh: typeset: -Q: unknown option
[usage message snipped for brevity]
$ echo $x ${.test.x}
123 123 <== expected: 123 456
$ x=789
$ echo $x ${.test.x}
789 789 <== expected: 789 456
$ # look at that, we never left the namespace...
When prefixing the erroneous 'typeset' with 'command', the problem
does not occur. 'command' disables the properties of special
built-ins such as exit on error. So, when a special built-in exits
on error, the parent scope is not properly resotred.
This bug exists in every ksh93 version with SHOPT_NAMESPACE so far.
src/cmd/ksh93/sh/xec.c: sh_exec():
- Before entering a namespace, use sh_pushcontext and sigsetjmp to
make sure we return here if sh_exit() is called, e.g. when a
special builtin throws an error, to ensure the parent scope
(oldnspace) is restored.
Thanks to @hyenias for making me aware of this bug.
Discussion: https://github.com/ksh93/ksh/issues/479#issuecomment-1140468965
Reproducer:
$ (IFS=$'\t\t'; val=$'\tone\t\ttwo\t'; set --posix; \
set -- $val; echo $#; set --noposix; set -- $val; echo $#)
2
4 <== OK
$ (IFS=$'\t\t'; val=$'\tone\t\ttwo\t'; set --posix; \
set -- $val; echo $#; set --default; set -- $val; echo $#)
2
2 <== bug
The output of the seconnd command line should be like the first.
When POSIX mode is turned off using 'set --noposix' (or 'set +o
posix'), sh.ifstable is invalidated as it needs to be repopulated
on the next field split to restore ksh-specific special handling of
a repeated $IFS whitespace character as non-whitespace. However,
when 'set --default' is used, this does not happen, which is a bug.
src/cmd/ksh93/sh/args.c: sh_argopts():
- While processing --default, when turning off SH_POSIX, call
sh_invalidate_ifs() to invalidate sh.ifstable.
The typeset output for -L/-R/-Z seems to be wrong when the input
has leading/trailing spaces. This started occurring after the
dynamic buffer size changes introduced in name.c as part of the
fix for <https://github.com/ksh93/ksh/issues/142>.
Test script:
typeset -L8 s_date1=" 22/02/09 08:25:01"; echo "$s_date1"
typeset -R10 s_date1="22/02/09 08:25:01 "; echo "$s_date1"
typeset -Z10 s_date1="22/02/09 08:25:01 "; echo "$s_date1"
Actual output:
22/02/0
08:25:01
0008:25:01
Expected output:
22/02/09
9 08:25:01
9 08:25:01
src/cmd/ksh93/sh/name.c: nv_newattr():
- Simplify allocation code, replacing the earlier dynamic buffer
size calculation with just the greater of the strlen and size.
Resolves: https://github.com/ksh93/ksh/issues/476
Co-authored-by: George Lijo <george.lijo@gmail.com>
On some systems, including at least Solaris 10.1 and QNX 6.5.0, the
regression tests below occurred. This is because, on these systems,
'cd .' always fails with 'no such file or directory', regardless of
flags, if the present working directory no longer exists. This is a
legitimate variation in UNIX-like systems so the tests should be
compatible.
test builtins begins at 2022-05-22+13:08:28
/usr/local/src/ksh/src/cmd/ksh93/tests/builtins.sh[1499]: cd: .: [No such file or directory]
builtins.sh[1501]: FAIL: cd -P without -e exits with error status if $PWD doesn't exist (expected 0, got 1)
/usr/local/src/ksh/src/cmd/ksh93/tests/builtins.sh[1504]: cd: .: [No such file or directory]
builtins.sh[1506]: FAIL: cd -eP doesn't fail if $PWD doesn't exist (expected 1, got 2)
test builtins failed at 2022-05-22+13:08:37 with exit code 2 [ 287 tests 2 errors ]
src/cmd/ksh93/tests/builtins.sh:
- Delete the 'cd -P .' test for a nonexistent PWD.
- For the 'cd -eP .' test for a nonexistent PWD, redirect standard
error to /dev/null and also accept exit status 2, which we would
expect with the '-e' flag if a 'no such file or directory' error
is thrown.
In command substitutions of the $(standard) and ${ shared state; }
form, backslash line continuation is broken.
Reproducer:
echo $(
echo one two\
three
)
Actual output (ksh93, all versions):
one two\ three
Expected output (every other shell, POSIX spec):
one twothree
src/cmd/ksh93/sh/lex.c: sh_lex(): case S_REG:
- Do not skip new-line joining if we're currently processing a
command substitution of one of these forms (i.e., if the
lp->lexd.dolparen level is > 0).
Background info/analysis:
comsub() is called from sh_lex() when S_PAR is the current state.
In src/cmd/ksh93/data/lexstates.c, we see that S_PAR is reached in
the ST_DOL state table at index 40. Decimal 40 is ( in ASCII. So,
the previous skipping of characters was done according to the
ST_DOL state table, and the character that stopped it was (. This
means we have $(.
Alternatively, comsub() may be called from sh_lex() by jumping to
the do_comsub label. In brief, that is the case when we have ${.
Regardless of which it is from the two, comsub() is now called from
sh_lex(). In comsub(), lp->lexd.dolparen is incremented at the
beginning and decremented at the end. Between them, we see that
sh_lex() is called. So, lp->lexd.dolparen in sh_lex() indicates the
depth of nesting $( or ${ statements we're in. Thus, it is also the
number of comsub() invocations seen in a backtrace taken in
sh_lex().
The codepath for `...` is different (and never had this bug).
Co-authored by: Martijn Dekker <martijn@inlv.org>
Resolves: https://github.com/ksh93/ksh/issues/367
The following reproducer causes a spurious syntax error:
foo="`: "("`"
The nested double quotes are not recognised correctly, causing a
syntax error at the '('. Removing the outer double quotes (which
are unnecessary) is a workaround, but it's still a bug as every
other shell accepts this. This bug has been present since the
original Bourne shell.
src/cmd/ksh93/sh/lex.c: sh_lex(): case S_QUOTE:
- If the current character is '"' and we're in a `...` command
substitution (ingrave is true), then do not switch to the old
mode but keep using the ST_QUOTE state table.
Thanks to @JohnoKing for the report and to @atheik for the fix.
Co-authored by: Martijn Dekker <martijn@inlv.org>
Resolves: https://github.com/ksh93/ksh/issues/352
The memory leak only occurred when an \f...\f string was used
outside a braces block ('{'...'}' a.k.a. GO...OG). This shows
the way to a more correct and elegant fix.
Co-authored by: Martijn Dekker <martijn@inlv.org>
In vi.c, there is a potential use of strcpy(3) on overlapping
strings, which is undefined behaviour:
> SHOPT_MULTIBYTE == 0
>
> # define gencpy(a,b) strcpy((char*)(a),(char*)(b))
>
> .
> .
> .
>
> if( mode != 'y' )
> {
> gencpy(cp,cp+nchars);
Thanks to Heiko Berges for the report.
src/cmd/ksh93/edit/{edit,emacs,vi}.c:
- Change almost all use of strcpy(3) to libast strcopy(), which
is a simple implementation that does not have a problem with
overlapping strings. Note that the return value is different
(it returns a pointer to the terminating '\0') but that is not
relevant in any of these cases.
- Same for strncpy(3) => libast strncopy().
src/lib/libast/string/strcopy.c:
- Backport a couple of cosmetic tweaks from the 93v- beta.
The QNX system at polarhome.com seems to be back up, at least
temporarily, though polarhome has officially shut down. This
allowed me to test ksh on QNX again, discovering that the build
was broken since reworking the standards macros handling.
src/lib/libast/features/standards:
- On QNX, define _QNX_SOURCE=1 to enable all extensions and
_FILE_OFFSET_BITS=64 to enable large file support with standard
library function names.
src/lib/libast/misc/optget.c: textout(): case ']':
- Before returning, call pop() to free any \f...\f info items that
are left. Note that this is a safe no-op if the pointer is null.
Resolves: https://github.com/ksh93/ksh/issues/407
Co-authored-by: Martijn Dekker <martijn@inlv.org>
This fixes another corner case bug in the horror show that is the
test/[ comand.
Reproducer:
$ ksh --posix -c 'test X -a -n'
ksh: test: argument expected
Every other shell returns 0 (success) as, POSIXly, this is a test
for the strings 'X' and '-n' both being non-empty, combined with
the binary -a (logical and) operator. Instead, '-n' was taken as a
unary primary operator with a missing argument, which is incorrect.
POSIX reference:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html
> 3 arguments:
> * If $2 is a binary primary, perform the binary test of $1 and $3.
src/cmd/ksh93/bltins/test.c:
- e3(): If the final argument begins with a dash, always treat it
as a test for a non-empty string, therefore return true. Do not
limit this to "new flags" only.
src/cmd/ksh93/tests/posix.sh:
- Added. These are tests for every aspect of the POSIX mode.
ksh has a little-known field splitting feature that conflicts with
POSIX: if a single-byte whitespace character (cf. isspace(3)) is
repated in $IFS, then field splitting is done as if that character
wasn't a whitespace character. An exmaple with the tab character:
$ (IFS=$'\t'; val=$'\tone\t\ttwo\t'; set -- $val; echo $#)
2
$ (IFS=$'\t\t'; val=$'\tone\t\ttwo\t'; set -- $val; echo $#)
4
The latter being the same as, for example
$ (IFS=':'; val='1️⃣2️⃣'; set -- $val; echo $#)
4
However, this is incompatible with the POSIX spec and with every
other shell except zsh, in which repeating a character in IFS does
not have any effect. So the POSIX mode must disable this.
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/sh/init.c:
- Add sh_invalidate_ifs() function that invalidates the IFS state
table by setting the ifsnp discipline struct member to NULL,
which will cause the next get_ifs() call to regenerate it.
- get_ifs(): Treat a repeated char as S_DELIM even if whitespace,
unless --posix is on.
src/cmd/ksh93/sh/args.c:
- sh_argopts(): Call sh_invalidate_ifs() when enabling or disabling
the POSIX option. This is needed to make the change in field
splitting behaviour take immediate effect instead of taking
effect at the next assignment to IFS.
b_enum() contains a check that exactly one argument is given:
237: if (error_info.errors || !*argv || *(argv + 1))
But the subsequent argument handling loop will happily deal with
multiple arguments:
246: while(cp = *argv++)
Every other declaration command supports multiple arguments and I
see no reason why enum shouldn't. Simply removing the '*(argv + 1)'
check allows 'enum' to create more than one type per invocation.
src/cmd/ksh93/bltins/enum.c:
- b_enum(): Remove check for >1 args as described above.
- Update documentation to describe the behaviour of enumeration
types in arithmetic expressions and to add an example: a bool
type with two enumeration values 'false' (0) and 'true' (1).
That type is predefined in ksh 93v- and 2020. We're not going
to do that in 93u+m but it's good to document the possibility.
src/cmd/ksh93/sh.1:
- Make changes parallel to the enum.c self-doc update.
Only notable changes listed below.
**/Mamfile:
- Do not bother redirecting standard error for 'cmp -s' to
/dev/null. Normally, 'cmp -s' on Linux, macOS, *BSD, or Solaris
do not not print any message. If it does, something unusual is
going on and I would want to see the message.
- Since we now require a POSIX shell, we can use '!'.
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/sh/init.c:
- Remove SH_TYPE_PROFILE symbol, unused after the removal of the
SHOPT_PFSH code. (re: eabd6453)
src/cmd/ksh93/sh/io.c:
- piperead(), slowread(): Replace redundant sffileno() calls by
the variables already containing their results. (re: 50d342e4)
src/cmd/ksh93/bltins/mkservice.c,
rc/lib/libcmd/vmstate.c:
- If these aren't compiled, define a stub function to silence the
ranlib(1) warning that the .o file does not contain symbols.
Changes:
- Fixed two xtrace test failures introduced in commit cfc8744c.
- The definition of _use_ntfork_tcpgrp in xec.c is now dependent on
SHOPT_SPAWN being defined (re: 8e9ed5be).
- Removed many unnecessary newlines and fixed various typos.
package now uses mamake to run all available regression tests
(currently iffe, mamake and ksh93) instead of just the ksh tests.
However, as a consequence, passing options or other arguments to
the shtests script via bin/package is no longer possible -- run
bin/shtests directly for that.
src/Mamfile, src/*/Mamfile:
- Make the 'test' virtual target execute subdirectories by
including the 'install' and 'all' virtual targets within it.
src/cmd/ksh93/Mamfile:
- Simplify and update the script in the 'test' virtual target.
src/cmd/builtin/Mamfile:
- Add dummy 'test' target to avoid an error.
bin/package, src/cmd/INIT/package.sh:
- Run 'mamake test' from the arch/*/src directory. This is the one
that must be the initial working directory for mamake, though the
Mamfiles aren't here; mamake finds them via the VPATH variable.
- Update self-doc.
- tests/*.sh: Backported many additional regression tests and test
fixes from the alpha and beta releases of ksh93v-.
- tests/alias.sh: Avoid trying to add vi to the hash table, as some
platforms do not provide a vi(1) implementation installed as part
of the default system. This fixes a regression test failure I was
getting in one of my Linux virtual machines.
- tests/builtins.sh: Fixed a bug in one of the regression tests that
caused an incorrect total error count if any of the tests failed.
- tests/sh_match.sh: Fixed a regression test failure on DragonFly
BSD caused by the diff command printing an extra 'No differences
encountered' line.
This commit fixes an infinite loop introduced in commit 0863a8eb
that caused ksh to enter an infinite loop if posix_spawn failed
to start a new process after setting the terminal process group.
Reproducer (warning: it will cause ksh to crash Wayland sessions
and drives up CPU usage by a ton):
$ /tmp/this/file/does/not/exist
/usr/bin/ksh: /tmp/this/file/does/not/exist: not found
$ <Press enter>
(ksh now prints $PS1 in a loop until killed with SIGKILL)
The first bug fixed is the infinite loop that occurs when
posix_spawn fails to execute a command. This was fixed by setting
the terminal process group to the main interactive shell.
The second bug fixed is related to the signal handling of the
SIGTTIN, SIGTTOU and SIGTSTP signals. In sh_ntfork() these signals
are set to their default signal handlers (SIG_DFL) before running
a command. The signal handlers were only restored to SIG_IGN
(ignore signal) when sh_ntfork() successfully ran a command.
This could cause a SIGTTOU lockup under strace when a command
failed to execute in an interactive shell, while also being one
cause of the infinite loop.
src/cmd/ksh93/sh/xec.c: sh_ntfork():
- Restore the terminal process group if posix_spawn failed to
launch a new process. This is necessary because posix_spawn will
set the terminal process group before it attempts to run a
command and doesn't restore it on failure.
This change turns off O_NONBLOCK for stdin if a previously ran
program left it on so that interactive programs that expect it
to be off work properly.
src/cmd/ksh93/sh/io.c: slowread():
- Turn off O_NONBLOCK for stdin if it is on.
Fixes: https://github.com/ksh93/ksh/issues/469
The previous fix for the += operator introduced a use-after-free
bug that could result in a variable pointing to random garbage:
$ foo=bar
$ foo+=_foo true
$ typeset -p foo
foo=V V
The use after free issue occurs because when nv_clone creates a
copy of $foo in the true command's invocation-local scope, it does
not duplicate the string $foo points to. As a result, the $foo
variable in the parent scope points to the same string as $foo in
the invocation-local scope, which causes the use after free bug
when cloned $foo variable is freed from memory.
src/cmd/ksh93/sh/nvdisc.c:
- To fix the use after free bug, allow nv_clone to duplicate the
string with memdup or strdup when no flags are passed.
src/cmd/ksh93/tests/variables.sh:
- Add a regression test for using the += operator with regular
commands.
src/cmd/ksh93/tests/leaks.sh:
- Add a regression test to ensure the bugfix doesn't introduce any
memory leaks.
Of course, we should not free the 'cp' pointer when we still need
to use it.
Resolves: https://github.com/ksh93/ksh/issues/467
Thanks to @atheik for the report.
dcl_dehacktivate() actually left a bit of inconsistent state beyond
the current line because it did not clear the dummy builtins tree.
This caused assignments to variables of types whose definitions
were parsed but not executed to throw a "not found" error instead
of a syntax error, even beyond the current line.
There is also an opportunity for an optimisation. We do not need to
initialise and maintain the dummy builtins tree if we're never
going to use it. So move that to the check_typedef() function right
before the sh_addbuiltin() call: check there if the tree exists and
if not create it and open a view. If no 'typeset -T' or 'enum'
type definition command is executed, the tree is never created.
dcl_hacktivate() now basically does nothing until the tree is
needed, but it does still count the recursion level and install the
error_info.exit hook because we need this to dcl_dehacktivate() at
the correct time when the tree does exist.
dcl_dehacktivate() is amended to clear the tree -- except if we're
running shcomp, as shcomp parses the script line by line without
executing anything, so we need the dummies to persist beyond the
sh_parse() invocation for the entire script. (Note that we do not
need this workaround for dot scripts or noexec mode, as the script
is entirely parsed in a single sh_parse() call in those cases.)
Reproducer (symptoms on at least macOS and FreeBSD):
$ mkfifo f
$ echo foo > f
(press Ctrl+Z)
^Zksh: f: cannot create [Interrupted system call]
Abort
The shell either aborts (dev builds) or crashes with 'Illegal
instruction' (release builds). This is consistent with
UNREACHABLE() being reached.
Backtrace:
0 libsystem_kernel.dylib __kill + 10
1 ksh sh_done + 836 (fault.c:678)
2 ksh sh_fault + 1324
3 libsystem_platform.dylib _sigtramp + 29
4 dyld ImageLoaderMachOCompressed::resolve(ImageLoader::LinkContext const&, char const*, unsigned ch
5 libsystem_c.dylib abort + 127
6 ksh sh_redirect + 3576 (io.c:1356)
7 ksh sh_exec + 7231 (xec.c:1308)
8 ksh exfile + 3247 (main.c:607)
9 ksh sh_main + 3551 (main.c:368)
10 ksh main + 38 (pmain.c:45)
11 libdyld.dylib start + 1
This means that UNREACHABLE() is actually reached here:
ksh/src/cmd/ksh93/sh/io.c
1351: if((fd=sh_open(tname?tname:fname,o_mode,RW_ALL)) <0)
1352: {
1353: errormsg(SH_DICT,ERROR_system(1),((o_mode&O_CREAT)?e_create:e_open),fname);
1354: UNREACHABLE();
1355: }
The cause is that, in the following section of code in sh_fault():
ksh/src/cmd/ksh93/sh/fault.c
183: #ifdef SIGTSTP
184: if(sig==SIGTSTP)
185: {
186: sh.trapnote |= SH_SIGTSTP;
187: if(pp->mode==SH_JMPCMD && sh_isstate(SH_STOPOK))
188: {
189: sigrelease(sig);
190: sh_exit(SH_EXITSIG);
191: return;
192: }
193: }
194: #endif /* SIGTSTP */
...sh_exit() is not getting called and the function will not return
because the SH_STOPOK bit is not set while the shell is blocked
waiting to write to a FIFO.
Even if sh_exit() did get called, that would not fix it, because
that function also checks for the SH_STOPOK bit and returns without
doing a longjmp if the signal is SIGTSTP and the SH_STOPOK bit is
not set. That is direct the reason why UNREACHABLE() was raeched:
errormsg() does call sh_exit() but sh_exit() then does not longjmp.
src/cmd/ksh93/sh/fault.c: sh_fault():
- To avoid the crash, we simply need to return from sh_fault() if
SH_STOPOK is off, so that the code path does not continue, no
error message is given on Ctrl+Z, UNREACHABLE() is not reached,
and the shell resumes waiting on trying to write to the FIFO.
The sh.trapnote flag should not be set if we're not going to
process the signal. This makes ksh behave like all other shells.
Resolves: https://github.com/ksh93/ksh/issues/464
Reproducer:
$ ksh -c 'unset PWD; (cd /); :'
Memory fault
The shell crashes because b_cd() is testing the value of the PWD
variable without checking if there is one.
src/cmd/ksh93/sh/path.c: path_pwd():
- Never return an unfreeable pointer to e_dot; always return a
freeable pointer. This fixes another corner-case crashing bug.
- Make sure the PWD variable gets assigned a value if it doesn't
have one, even if it's the "." fallback. However, if the PWD is
inaccessible but we did inherit a $PWD value that starts with a
/, then use the existing $PWD value as this will help the shell
fail gracefully.
src/cmd/ksh93/bltins/cd_pwd.c:
- b_cd(): When checking if the PWD is valid, use the sh.pwd copy
instead of the PWD variable. This fixes the crash above.
- b_cd(): Since path_pwd() now always returns a freeable value,
free sh.pwd unconditionally before setting the new value.
- b_pwd(): Not only check that path_pwd() returns a value starting
with a slash, but also verify it with test_inode() and error out
if it's wrong. This makes the 'pwd' command useful for checking
that the PWD is currently accessible.
src/cmd/ksh93/data/msg.c:
- Change e_pwd error message for accuracy and clarity.
This backports two minor additions to the 'read' built-in from ksh
93v-: '-a' is now the same as '-A' and '-u p' is the same as '-p'.
This is for compatibility with some 93v- or ksh2020 scripts.
Note that their change to the '-p' option to support both prompts
and reading from the coprocess was *not* backported because we
found it to be broken and unfixable. Discussoin at:
https://github.com/ksh93/ksh/issues/463
src/cmd/ksh93/bltins/read.c: b_read():
- Backport as described above.
- Rename the misleadingly named 'name' variable to 'prompt'.
It points to the prompt string, not to a variable name.
src/cmd/ksh93/data/builtins.c: sh_optpwd[]:
- Add -a as an alterative to -A. All that is needed is adding '|a'
and optget(3) will automatically convert it to 'A'.
- Change -u from a '#' (numeric) to ':' option to support 'p'. Note
that b_read() now needs a corresponding strtol() to convert file
descriptor strings to numbers where applicable.
- Tweaks.
src/cmd/ksh93/sh.1:
- Update accordingly.
- Tidy up the unreadable mess that was the 'read' documentation.
The options are now shown in a list.
From README:
FILESCAN on Experimental option that allows fast reading of files
using while < file;do ...; done and allowing fields in
each line to be accessed as positional parameters.
As SHOPT_FILESCAN has been enabled by default since ksh 93l
2001-06-01, the filescan loop is now documented in the manual page
and the compile-time option is no longer considered experimental.
We must disable this at runtime if --posix is active because it
breaks a portable use case: POSIXly, 'while <file; do stuff; done'
repeatedly excutes 'stuff' while 'file' can successfully be opened
for reading, without actually reading from 'file'.
This also backports a bugfix from the 93v- beta. Reproducer:
$ echo 'one two three' >foo
$ while <foo; do printf '[%s] ' "$@"; echo; done
[one two three]
Expected output:
[one] [two] [three]
The bug is that "$@" acts like "$*", joining all the positional
parameters into one word though it should be generating one word
for each.
src/cmd/ksh93/sh/macro.c: varsub():
- Backport fix for the bug described above. I do not understand the
opaque macro.c code well enough yet to usefully describe the fix.
src/cmd/ksh93/sh/xec.c: sh_exec():
- Improved sanity check for filescan loop: do not recognise it if
the simple command includes variable assignments, more than one
redirection, or an output or append redirection.
- Disable filescan loops if --posix is active.
- Another 93v- fix: handle interrupts (errno==EINTR) when closing
the input file.
As so often with SHOPT_* compile-time options, not all the relevant
code was actually conditional on the option macro. After removing
SHOPT_ENV, the arguments to sh_putenv() and env_delete() macro
calls are dead code and, after removing that, the sh.env variable
is unused.
src/cmd/ksh93/{sh/path.c,include/shell.h}:
- The sh.defpathlist variable is never set once, which makes every
use of this variable unnecessary (as it's always null). This
commit removes sh.defpathlist while also fixing a possible memory
leak (there was another location where defpathinit() was invoked
without saving the returned pointer, causing a memory leak).
In UTF-8 locales, ksh breaks when a KEYBD trap is active, even a
dummy no-op one like 'trap : KEYBD'. Entering multi-byte characters
fails (the input is interrupted and a new prompt is displayed) and
pasting content with multi-byte characters produces corrupted
results.
The cause is that the KEYBD trap code is not multibyte-ready.
Unfortunately nobody yet understands the edit.c code well enough
to implement a proper fix. Pending that, this commit implements
a workaround that at least avoids breaking the shell.
src/cmd/ksh93/edit/edit.c: ed_getchar():
- When a multi-byte locale is active, do not trigger the the KEYBD
trap except for ASCII characters (1-127).
Resolves: https://github.com/ksh93/ksh/issues/307
macOS 12.2.1 doesn't seem to like the -M, -v or -d ulimit options:
sh_match.sh[502]: FAIL: test_xmlfragment1/0/testfile1.xml:
Expected empty stderr, got $'test1_script.sh[2]: ulimit: 1048576:
limit exceeded [Invalid argument]\ntest1_script.sh[3]: ulimit:
1048576: limit exceeded [Invalid argument]\ntest1_script.sh[4]:
ulimit: 1048576: limit exceeded [Invalid argument]'
The 'Invalid argument' addition is caused by errno==EINVAL and
suggests the OS either doesn't support setting this limit, or
support for it was somehow disabled.
src/cmd/ksh93/tests/sh_match.sh:
- Redirect standard error for ulimit commands to 2>/dev/null. If
they fail it's pretty inconsequential and it's not related to
actual ${.sh.match} testing at all.
Resolves: https://github.com/ksh93/ksh/issues/459
Thanks to @posguy99 for the report.
Sfio may theoretically be compiled with threads support using a
separate AT&T library called Vthread, also by Kiem-Phong Vo. That
library was never shipped in the AST distribution, though. It is
only available with a standalone version of Sfio.
The only standalone Sfio version with Vthread that I've found is
from 2005, mirrored at <https://github.com/lichray/sfio>. More
recent versions never seem to have made it out of the defunct AT&T
software download site.
Even if they weren't, the rest of libast doesn't support threads,
and at this point it never will, so for our purposes the Sfio
threads code is never going to be usable. Meanwhile, macros such as
SFMTXENTER and SFMTXRETURN make the code a lot harder to read. And
not quite all threading code is disabled; some of it is dead code
that is getting compiled in.
Chances are that code now won't work properly in any case as we've
not had any chance to test it as we were making changes. Bit rot
has surely set in by now.
So this commit expands all the sfio/stdio threading-related macros
to their non-threads fallbacks (which is null for most of them, but
not all), deletes dead mutex-related code and struct fields, and
removes the related documentation from the sfio.3 man page. Unless
I did something wrong, there should be no change in behaviour.
The -f/--noglob shell option is documented simply as: "Disables
pathname expansion." But after 'set -f' on an interactive shell,
command completion and file name completion also stop working. This
is because they internally use the pathname expansion mechanism.
But it is not documented anywhere that 'set -f' disables
completion; it's just a side effect of an implementation detail.
Though ksh has always acted like this, I think it should change
because it's not useful or expected behaviour. Other shells like
bash, yash or zsh don't act like this.
src/cmd/ksh93/sh/expand.c,
src/cmd/ksh93/sh/macro.c:
- Allow the SH_COMPLETE (command completion) or SH_FCOMPLETE
(file name completion) state bit to override SH_NOGLOB in
path_generate() and in sh_macexpand().
It probably won't make a difference since the 'sleep' is run in the
background, but let's change 'sleep .5 &' back to the original
'sleep 1 &' from the 93u+ 2012-08-01 version and see what happens.
See: https://github.com/ksh93/ksh/issues/344#issuecomment-982219206
The >&- redirection subshell leak fixed in 6304dfce still existed
for shared-state ${ command substitutions; } a.k.a. subshares,
which cannot be forked.
I previously noticed that sh_subsavefd() saves the FD state even
for subshares. That seems logically incorrect as subshares share
their state with the invoking environment by definition.
src/cmd/ksh93/sh/subshell.c: sh_subsavefd():
- Sure enough, adding a check for !sh.subshare fixes the bug.
- Use the sh.subshell counter and not the subshell data pointer to
check for a virtual subshell. If the shell is reinitialised in a
fork to execute a new script (see 0af81992), any parent virtual
subshell data is currently not cleared as it is locally scoped to
subshell.c, so that check would be incorrect then.
src/cmd/ksh93/sh/io.c: sh_redirect:
- Remove now-redundant (and actually incorrectly placed) check for
sh.subshare added in fb755163.
Reproducer:
exec 9>&1
( { exec 9>&1; } 9>&- )
echo "test" >&9 # => 9: cannot open [Bad file descriptor]
The 9>&- incorrectly persists beyond the { } block that it
was attached to *and* beyond the ( ) subshell. This is yet another
bug with non-forking subshells; forking it with something like
'ulimit -t unlimited' works around the bug.
In over a year we have not been able to find a real fix, but I came
up with a workaround that forks a virtual subshell whenever it
executes a code block with a >&- or <&- redirection attached. That
use case is obscure enough that it should not cause any performance
regression except in very rare corner cases.
src/cmd/ksh93/sh/xec.c: sh_exec(): TSETIO:
- This is where redirections attached to code blocks are handled.
Check for a >&- or <&- redirection using bit flaggery from
shnodes.h and fork if we're executing such in a virtual subshell.
Resolves: https://github.com/ksh93/ksh/issues/161
Thanks to @ko1nksm for the bug report.
