macOS 12.2.1 doesn't seem to like the -M, -v or -d ulimit options:
sh_match.sh[502]: FAIL: test_xmlfragment1/0/testfile1.xml:
Expected empty stderr, got $'test1_script.sh[2]: ulimit: 1048576:
limit exceeded [Invalid argument]\ntest1_script.sh[3]: ulimit:
1048576: limit exceeded [Invalid argument]\ntest1_script.sh[4]:
ulimit: 1048576: limit exceeded [Invalid argument]'
The 'Invalid argument' addition is caused by errno==EINVAL and
suggests the OS either doesn't support setting this limit, or
support for it was somehow disabled.
src/cmd/ksh93/tests/sh_match.sh:
- Redirect standard error for ulimit commands to 2>/dev/null. If
they fail it's pretty inconsequential and it's not related to
actual ${.sh.match} testing at all.
Resolves: https://github.com/ksh93/ksh/issues/459
Thanks to @posguy99 for the report.
The -f/--noglob shell option is documented simply as: "Disables
pathname expansion." But after 'set -f' on an interactive shell,
command completion and file name completion also stop working. This
is because they internally use the pathname expansion mechanism.
But it is not documented anywhere that 'set -f' disables
completion; it's just a side effect of an implementation detail.
Though ksh has always acted like this, I think it should change
because it's not useful or expected behaviour. Other shells like
bash, yash or zsh don't act like this.
src/cmd/ksh93/sh/expand.c,
src/cmd/ksh93/sh/macro.c:
- Allow the SH_COMPLETE (command completion) or SH_FCOMPLETE
(file name completion) state bit to override SH_NOGLOB in
path_generate() and in sh_macexpand().
It probably won't make a difference since the 'sleep' is run in the
background, but let's change 'sleep .5 &' back to the original
'sleep 1 &' from the 93u+ 2012-08-01 version and see what happens.
See: https://github.com/ksh93/ksh/issues/344#issuecomment-982219206
The >&- redirection subshell leak fixed in 6304dfce still existed
for shared-state ${ command substitutions; } a.k.a. subshares,
which cannot be forked.
I previously noticed that sh_subsavefd() saves the FD state even
for subshares. That seems logically incorrect as subshares share
their state with the invoking environment by definition.
src/cmd/ksh93/sh/subshell.c: sh_subsavefd():
- Sure enough, adding a check for !sh.subshare fixes the bug.
- Use the sh.subshell counter and not the subshell data pointer to
check for a virtual subshell. If the shell is reinitialised in a
fork to execute a new script (see 0af81992), any parent virtual
subshell data is currently not cleared as it is locally scoped to
subshell.c, so that check would be incorrect then.
src/cmd/ksh93/sh/io.c: sh_redirect:
- Remove now-redundant (and actually incorrectly placed) check for
sh.subshare added in fb755163.
Reproducer:
exec 9>&1
( { exec 9>&1; } 9>&- )
echo "test" >&9 # => 9: cannot open [Bad file descriptor]
The 9>&- incorrectly persists beyond the { } block that it
was attached to *and* beyond the ( ) subshell. This is yet another
bug with non-forking subshells; forking it with something like
'ulimit -t unlimited' works around the bug.
In over a year we have not been able to find a real fix, but I came
up with a workaround that forks a virtual subshell whenever it
executes a code block with a >&- or <&- redirection attached. That
use case is obscure enough that it should not cause any performance
regression except in very rare corner cases.
src/cmd/ksh93/sh/xec.c: sh_exec(): TSETIO:
- This is where redirections attached to code blocks are handled.
Check for a >&- or <&- redirection using bit flaggery from
shnodes.h and fork if we're executing such in a virtual subshell.
Resolves: https://github.com/ksh93/ksh/issues/161
Thanks to @ko1nksm for the bug report.
Notable changes:
- Remove SHOPT_PFSH compile-time option and associated code.
This was meant to work with Solaris rights profiles, see:
https://docs.oracle.com/cd/E23824_01/html/821-1461/profiles-1.html#REFMAN1profiles-1
But it has been obsolete for years as Solaris stopped using
it in its shipped ksh several OS versions ago, preferring a
library-based wrapper around ksh and other shells.
Nonetheless I experimented with the option on Solaris 11.4.
Result: no external command will run; output of unitialised
memory in error message. So it's already fallen victim to bit
rot. There's nothing interesting here, so just get rid.
- Remove SHOPT_TYPEDEF compile-time option (but keep the code!).
Turning it off caused the build to fail. It may be possible to
fix it, but the type definition code is integral to ksh now (e.g.
'enum' depends on much of it) so it makes no sense to disable it.
This was removed in the ksh 93v- beta version as well.
- Remove nv_close() calls and remove nv_close() documentation from
the nval.3 man page. This function is a dummy, present without
any changes since the beginning of the ast-open-archive repo in
1995. The comment was: "Currently this is a dummy, but someday
will be needed for reference counting". 27 or more years later,
it's time to admit it's never going to happen. (And of course,
nv_close() calls were not being used with anything resembling
consistency.)
- Add a null nv_close() macro to nval.h for compatibility with
third party code that follows the old documentation.
- Add a few missing regression tests.
This commit backports all of the relevant .sh.match bugfixes from
ksh93v-. Most of the .sh.match rewrite is from versions 2012-08-24
and 2012-10-04, with patches from later releases of 93v- and
ksh2020 also applied. Note that there are still some remaining bugs
in .sh.match, although now the total count of .sh.match bugs should
be less that before.
These are the relevant changes in the ksh93v- changelog that were
backported:
12-08-07 .sh.match no longer gets set for patterns in PS4 during
set -x.
12-08-10 Rewrote .sh.match expansions fixing several bugs and
improving performance.
12-08-22 .sh.match now handles subpatterns that had no matches with
${var//pattern} correctly.
12-08-21 A bug in setting .sh.match after ${var//pattern/string}
when string is empty has been fixed.
12-08-21 A bug in setting .sh.match after [[ string == pattern ]]
has been fixed.
12-08-31 A bug that could cause a core dump after
typeset -m var=.sh.match has been fixed.
12-09-10 Fixed a bug in typeset -m the .sh.match is being renamed.
12-09-07 Fixed a bug in .sh.match code that coud cause the shell
to quitely
13-02-21 The 12-01-16 bug fix prevented .sh.match from being used
in the replacement string. The previous code was restored
and a different fix which prevented .sh.match from being
computed for nested replacement has been used instead.
13-05-28 Fixed two bug for typeset -c and typeset -m for variable
.sh.match.
Changes:
- The SHOPT_2DMATCH option has been removed. This was already the
default behavior previously, and now it's documented in the man
page.
- init.c: Backported the sh_setmatch() rewrite from 93v- 2012-08-24
and 2012-10-04.
- Backported the libast 93v- strngrpmatch() function, as the
.sh.match rewrite requires this API.
- Backported the sh_match regression tests from ksh93v-, with many
other sh_match tests backported from ksh2020. Much of the sh_match
script is based on code from Roland Mainz:
https://marc.info/?l=ast-developers&m=134606574109162&w=2https://marc.info/?l=ast-developers&m=134490505607093
- tests/{substring,treemove}.sh: Backported other relevant .sh.match
fixes, with tests added to the substring and treemove test scripts.
- tests/types.sh: One of the (now reverted) memory leak bugfixes
introduced a CI test failure in this script, so for that test the
error message has been improved.
- string/strmatch.c: The original ksh93v- code for the strngrpmatch()
changes introduced a crash that could occur because strlen would
be used on a null pointer. This has been fixed by avoiding strlen
if the string is null.
One nice side effect of these changes is a considerable performance
improvement in the shbench[1] gsub benchmark (results from 20
iterations with CCFLAGS=-Os):
--------------------------------------------------
name /tmp/ksh-current /tmp/ksh-matchfixes
--------------------------------------------------
gsub.ksh 0.883 [0.822-0.959] 0.457 [0.442-0.505]
--------------------------------------------------
Despite all of the many fixes and improvements in the backported
93v- .sh.match code, there are a few remaining bugs:
- .sh.match is printed with a default [0] subscript (see also
https://github.com/ksh93/ksh/issues/308#issuecomment-1025016088):
$ arch/*/bin/ksh -c 'echo ${!.sh.match}'
.sh.match[0]
This bug appears to have been introduced by the changes from
ksh93v- 2012-08-24.
- The wrong variable name is given for 'parameter not set' errors
(from https://marc.info/?l=ast-developers&m=134489094602596):
$ arch/*/bin/ksh -u
$ x=1234
$ true "${x//~(X)([012])|([345])/}"
$ compound co
$ typeset -m co.array=.sh.match
$ printf "%q\n" "${co.array[2][0]}"
arch/linux.i386-64/bin/ksh: co.array[2][(null)]: parameter not set
- .sh.match leaks out of subshells. Further information and a
reproducer can be found here:
https://marc.info/?l=ast-developers&m=136292897330187
[1]: https://github.com/ksh-community/shbench
When executing a script without a hashbang path like #!/bin/ksh,
ksh forks itself, longjmps back to sh_main(), and then (among other
things) calling sh_reinit() which is the function that tries to
reinitialise as much of the shell as it can. This is its way of
ensuring the child script is run in ksh and not some other shell.
However, this appraoch is incredibly buggy. Among other things,
changes in built-in commands and custom type definitions survived
the reinitialisation, "exporting" variables didn't work properly,
and the hash table and ${.sh.stats} weren't reset. As a result,
depending on what the invoking script did, the invoked script could
easily fail or malfunction.
It is not actually possible to reinitialise the shell correctly,
because some of the shell state is in locally scoped static
variables that cannot simply be reinitialised. There are probably
huge memory leaks with this approach as well. At some point, all
this is going to need a total redesign. Clearly, the only reliable
way involves execve(2) and a start from scratch.
For now though, this seems to fix the known bugs at least. I'm sure
there are more to be discovered.
This commit makes another change: instead of the -h/trackall option
(which has been a no-op for decades), the posix option is now
inherited by the child script. Since there is no hashbang path from
which to decide whether the shell should run in POSIX mode nor not,
the best guess is probably the invoking script's setting.
src/cmd/ksh93/sh/init.c: sh_reinit():
- Keep the SH_INIT state on during the entire procedure.
- Delete remaining non-exported, non-default variables.
- Remove attributes from exported variables. In POSIX mode, remove
all attributes; otherwise, only remove readonly.
- Unset discipline function pointers for variables.
- Delete all custom types.
- Delete all functions and built-ins, then reinitialise the built-ins
table from scatch.
- Free the alias values before clearing the alias table.
- Same with hash table entries (tracked aliases).
- Reset statistics.
- Inherit SH_POSIX instead of SH_TRACKALL.
- Call user init function last, not somewhere in the middle.
src/cmd/ksh93/sh/name.c: sh_envnolocal():
- Be sure to preserve the export attribute of kept variables.
Resolves: https://github.com/ksh93/ksh/issues/350
This bugfix was backported from ksh93v- 2012-10-04. The bug fixed
by this change is one that causes 'typeset -p' to omit the -C flag
when listing compound arrays belonging to a type:
$ typeset -T Foo_t=(compound -a bar)
$ Foo_t baz
$ typeset -p baz.bar
typeset -a baz.bar='' # This should be 'typeset -C -a'
src/cmd/ksh93/sh/nvtype.c:
- Backport change from 93v- 2012-10-04 that sets the array nvalue to
a pointer named Null (which is "") in nv_mktype(), then to Empty
in fixnode().
- Change the Null name from the 93v- code to AltEmpty to avoid
misleading code readers into thinking that it's a null pointer.
src/cmd/ksh93/tests/types.sh:
- Backport the relevant 93v- changes to the types regression tests.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
In ksh93v- 2012-10-04 the following bugfix is noted in the changelog
(this fix was most likely part of ksh93v- 2012-09-27, although that
version is not archived anywhere):
12-09-21 A bug in which the output of a two dimensional sparse
indexed array would cause the second subscript be treated
as an associative array when read back in has been fixed.
Elements that are sparse indexed arrays now are prefixed
type "typeset -a".
Below is a before and after of this change:
# Before
$ typeset -a foo[1][2]=bar
$ typeset -p foo
typeset -a foo=([1]=([2]=bar) )
# After
$ typeset -a foo[1][2]=bar
$ typeset -p foo
typeset -a foo=(typeset -a [1]=([2]=bar) )
src/cmd/ksh93/sh/*.c:
- Backport changes from ksh93v- to print 'typeset -a' before sparse
indexed arrays and properly handle 'typeset -a' in reinput
commands from 'typeset -p'.
src/cmd/ksh93/tests:
- Add two regression tests to arrays.sh for this change.
- Update the existing regression tests for compatibility with the
new printed typeset output.
Special builtins are undeleteable for a reason. But 'enum' and
'typeset -T' allow overriding them, causing an inconsistent state.
@JohnoKing writes:
| The behavior is rather buggy, as it appears to successfully
| override normal builtins but fails to delete the special
| builtins, leading to scenarios where both the original builtin
| and type are run:
|
| $ typeset -T eval=(typeset BAD; typeset TYPE) # This should have failed
| $ eval foo=BAD
| /usr/bin/ksh: eval: line 1: foo: not found
| $ enum trap=(BAD TYPE) # This also should have failed
| $ trap foo=BAD
| /usr/bin/ksh: trap: condition(s) required
| $ enum umask=(BAD TYPE)
| $ umask foo=BAD
| $ echo $foo
| BAD
|
| # Examples of general bugginess
| $ trap bar=TYPE
| /usr/bin/ksh: trap: condition(s) required
| $ echo $bar
| TYPE
| $ eval var=TYPE
| /usr/bin/ksh: eval: line 1: var: not found
| $ echo $var
| TYPE
This commit fixes the following:
The 'enum' and 'typeset -T' commands are no longer allowed to
override and replace special built-in commands, except for type
definition commands previously created by these commands; these
are already (dis)allowed elsewhere.
A command like 'typeset -T foo_t' without any assignments no longer
creates an incompletely defined 'foo_t' built-in comamnd. Instead,
it is now silently ignored for backwards compatibility. This did
have a regression test checking for it, but I'm changing it because
that's just not a valid use case. An incomplete type definition
command does nothing useful and only crashes the shell when run.
src/cmd/ksh93/bltins/enum.c: b_enum():
- Do not allow overriding non-type special built-ins.
src/cmd/ksh93/sh/name.c: nv_setlist():
- Do not allow 'typeset -T' to override non-type special built-ins.
To avoid an inconsistent state, this must be checked for while
processing the assignments list before typeset is really invoked.
src/cmd/ksh93/bltins_typeset.c: b_typeset():
- Only create a type command if sh.envlist is set, i.e., if some
shell assignment(s) were passed to the 'typeset -T' command.
Progresses: https://github.com/ksh93/ksh/issues/350
$ unset foo
$ echo ${foo[42]=bar}
(empty line)
Instead of the empty line, 'bar' was expected. As foo[42] was
unset, the conditional assignment should have worked.
$ unset foo
$ : ${foo[42]?error: unset}
(no output)
The expansion should have thrown an error with the given message.
This bug was introduced in ksh 93t 2008-10-01. Thanks to @JohnoKing
for finding the breaking change.
Analysis: The problem was experimenally determined to be in in the
following lines of nv_putsub(). If the array member is unset (i.e.
null), the value is set to the empty string instead:
src/cmd/ksh93/sh/array.c
1250: else
1251: ap->val[size].cp = Empty;
It makes some sense: if there is a value (even an empty one), the
variable is set and these expansions should behave accordingly.
Sure enough, deleting these lines fixes the bug, but at the expense
of introducing a lot of other array-related regressions. So we need
a way to special-case the affected expansions.
Where to do this? If we replace line 1251 with an abort(3) call, we
get this stack trace:
0 libsystem_kernel.dylib __pthread_kill + 10
1 libsystem_pthread.dylib pthread_kill + 284
2 libsystem_c.dylib abort + 127
3 ksh nv_putsub + 1411 (array.c:1255)
4 ksh nv_endsubscript + 940 (array.c:1547)
5 ksh nv_create + 4732 (name.c:1066)
6 ksh nv_open + 1951 (name.c:1425)
7 ksh varsub + 4934 (macro.c:1322)
[rest omitted]
The special-casing needs to be done on line 1250 of array.c, but
flagged in varsub() which processes these expansions. So, varsub()
calls nv_open() calls nv_create() calls nv_endsubscript() calls
nv_putsub(). That's a fairly deep call stack, so passing an extra
flag argument does not seem doable. I did try an approach using a
couple of new bit flags passed via these functions' flags and mode
parameters, but the way this code base uses bit flags is so
intricate, it does not seem to be possible to add or change
anything without unwanted side effects in all sorts of places.
So the only fix I can think of adds yet another global flag
variable for a very special case. It's ugly, but it works.
An elegant fix would probably involve a fairly comprehensive
redesign, which is simply not going to happen.
src/cmd/ksh93/include/shell.h:
- Add global sh.cond_expan flag.
src/cmd/ksh93/sh/array.c: nv_putsub():
- Do not set value to empty string if sh.cond_expan is set.
src/cmd/ksh93/sh/macro.c: varsub():
- Set sh.cond_expan flag while calling nv_open() for one of the
affected expansions.
- Minor refactoring for legibility and to make the fix fit better.
- SSOT: Instead of repeating string "REPLY", use the node's nvname.
- Do not pointlessly add an extra 0 byte when saving id for error
message; sfstruse() already adds this.
Thanks to @oguz-ismail for the bug report.
Resolves: https://github.com/ksh93/ksh/issues/383
Variables with a dot in their name, such as those declared in
namespace { ... } blocks, are usually stored in a separate tree
with their actual names not containing any dots. But under some
circumstances, including at least direct assignment of a
non-preexisting dot variable, dot variables are stored in the main
sh.var_tree with names actually containing dots. With allexport
active, those could end up exported to the environment. This bug
was also present in previous release versions of ksh.
src/cmd/ksh93/sh/name.c: pushnam():
- Check for a dot in the name before pushing a variable to export.
This fix was backported from ksh 93v- 2012-10-04.
src/cmd/ksh93/sh/nvtree.c: nv_outnode():
- If the array is supposed to be empty, do not continue. This
avoids outputting a nonexistent [0]= element for empty arrays.
Resolves: https://github.com/ksh93/ksh/issues/420
Co-authored-by: Martijn Dekker <martijn@inlv.org>
All variables that are assigned a value should be exported while
the allexport shell option is enabled. This works in most cases,
but variables assigned to with ${var:=foo} or $((var=123)) aren't
exported while allexport is on.
src/cmd/ksh93/sh/name.c:
- nv_putval(): This is the central assignment function; all forms
of variable assignment end up here. So this is the best place
to check for SH_ALLEXPORT and turn on the export attribute.
- nv_setlist(): Remove allexport handling, now redundant.
src/cmd/ksh93/bltins/read.c: sh_readline():
- Remove allexport handling, now redundant.
src/cmd/ksh93/sh/main.c: sh_main():
- nv_putval() is used to initialize PS4 and IFS using nv_putval();
this is after an -a/--allexport specified on the ksh command
line has been processed, so temporarily turn that off.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
If the VISUAL or EDITOR environment variable is not set to a value
matching *[Vv][Ii]* or *macs* at initialisation time, then ksh does
not turn on any line editor.
This is user-hostile. New users on Unix-like systems typically have
a simple editor like nano preconfigured as their default, or may
not have the VISUAL or EDITOR variable set at all. So if they try
ksh, they find themselves without basic functionality such as arrow
keys and probably go straight back to bash.
The emacs line editor is by far the most widely used, especially
among new users, so ksh should default to that. Most other shells
already do this.
src/cmd/ksh93/sh/main.c: sh_main():
- On an interactive shell, if on editor was turned on based on
$VISUAL or $EDITOR, turn on emacs before reading input.
Reproducer:
$ ksh -c 'bash -c '\''kill -s INT $$'\''; echo "$?, continuing"'
Expected result: output "258, continuing"; exit status 0.
Actual result: no output; exit status 258. The child process sent
SIGINT only to itself and not to the process group, so the parent
script was wrongly interrupted.
Every shell except ksh93 produces the expected result. ksh93 also
gave the expected result before version 2008-01-31 93s+, which
introduced the code below.
Analysis: The problem is in these lines of code in xec.c,
sh_exec(), TFORK case, parent branch of fork:
1649: if(!sh_isstate(SH_MONITOR))
1650: {
1651: if(!(sh.sigflag[SIGINT]&(SH_SIGFAULT|SH_SIGOFF)))
1652: sh_sigtrap(SIGINT);
1653: sh.trapnote |= SH_SIGIGNORE;
1654: }
[...pipe and I/O handling, wait for command to finish...]
1667: if(!sh_isstate(SH_MONITOR))
1668: {
1669: sh.trapnote &= ~SH_SIGIGNORE;
1700: if(sh.exitval == (SH_EXITSIG|SIGINT))
1701: kill(sh.current_pid,SIGINT);
1702: }
When a user presses Ctrl+C, SIGINT is sent to the entire process
group. If job control is fully off (i.e., !sh_isstate(SH_MONITOR)),
then the process group includes the parent script. Therefore, in a
script such as
$ ksh -c 'bash -c '\''read x'\''; echo "$?, continuing"'
when the user presses Ctrl+C while bash waits for 'read x' input,
the parent ksh script should be interrupted as well.
Now, the code above ignores SIGINT while bash is running. (This is
done using special-casing in sh_fault() to handle that SH_SIGIGNORE
flag for SIGINT.) So, when Ctrl+C interrupts the process group, the
parent script is not getting interrupted as it should.
To compensate for that, the code then detects, using sh.exitval
(the child process' exit status), whether the child process was
killed by SIGINT. If so, it simply assumes that the signal was
meant for the process group including the parent script, so it
reissues SIGINT to itself after unignoring it.
But, as we can see from the broken reproducer above, that
assumption is not valid. Scripts are perfectly free to send SIGINT
to themselves only, and that must work as expected.
src/cmd/ksh93/sh/xec.c: sh_exec(): TFORK: parent branch:
- Instead of ignoring SIGINT, sigblock() it, which delays handling
the signal until sigrelease(). (Note that these are macros
defined in src/cmd/ksh93/features/sigfeatures according to OS
capabilities.)
- This makes reissuing SIGINT redundant, so delete that, which
fixes the bug.
src/cmd/ksh93/sh/fault.c:
- Nothing now sets the SH_SIGIGNORE flag in sh.trapnote, so remove
special-casing added in 2008-01-31 93s+.
This commit applies various accumulated bugfixes:
- Applied some fixes for compiler warnings based off of the
following pull requests (with whitespace changes excluded to
avoid inflating the size of the diff):
https://github.com/att/ast/pull/281https://github.com/att/ast/pull/283https://github.com/att/ast/pull/303https://github.com/att/ast/pull/304
- clone_type(): Two separate variables in this function share the
same name. A bugfix from ksh93v- 2013-05-24 was backported to
avoid conflict issues.
- Backported a minor error message improvement from ksh2020 for
when the user attempts to run too many coprocesses.
- Backported a ksh2020 bugfix for a file descriptor leak:
58bc8b56
- Backported ksh2020 bugfixes for unused variable and pointless
assignment lint warnings:
47650fe0df209c0d5e417b00
- Applied a few minor improvements to libast from graphviz:
e7c03541969a7cde
- dtmemory(): Mark unused parameters with NOT_USED(). Based on:
6ac3ad99
- Applied a few documentation/comment tweaks for the NEWS file,
printf -v and spawnveg.
- Added a missing regression test for using the rm builtin's -f
option without additional arguments (this was fixed in
ksh93u+ 2012-02-14).
Attempting to use array subscript expansion with variables that
aren't set currently causes a spurious syntax error (in ksh93u+ and
older commits the reproducer crashes):
$ ksh -c 'echo ${foo[${bar}..${baz}]}' # Shouldn't print anything
ksh: : arithmetic syntax error
src/cmd/ksh93/sh/macro.c:
- Backport a parser bugfix from ksh93v- 2012-08-24 that avoids
setting mp->dotdot until the copyto() function's loop is
finished.
src/cmd/ksh93/tests/arrays.sh:
- Add regression tests for this bug.
Notable changes:
src/cmd/ksh93/include/fault.h:
- Get rid of the superflous sh pointer argument in the
sh_pushcontext() and sh_popcontext() macros. (re: 2d3ec8b6)
src/cmd/ksh93/tests/io.sh:
- Tweak a process substitution test to allow up to a second for
unused process substitution processes to disappear. On the Alpine
Linux console (at least the musl libc version), this is needed to
avoid a test failure as long as no GUI is active. As soon as you
start X11, this phenomenon disappears, even on the console. Very
strange, but also very probably not ksh's fault.
src/cmd/ksh93/tests/shtests:
- Instead of just SIGCONT and SIGPIPE, set all signals to default,
just to be sure to avoid spurious test failures due to signals
that were ignored on entry. (It made no difference to the
aforementioned Alpine Linux test failure, so ignored signals had
nothing to do with that -- but still a good idea.)
.github/workflows/ci.yml:
- On the GitHub CI runs, when testing with SHOPTs disabled, disable
SHOPT_SPAWN as well, which tests that everything still works
correctly with the regular fork(2) method.
COPYRIGHT:
- Remove duplicate of BSD license.
src/cmd/ksh93/data/variables.sh: shtab_variables[]:
- Remove unused "CSWIDTH" entry. All use of it (including the
matching CSWIDTHNOD macro) was removed in version 2003-04-22.
src/cmd/ksh93/tests/variables.sh:
- For the tests on the shtab_variables[] variables, read the
variable names straight from variables.c instead of synching
the list in the test script, which would surely be forgotten.
src/cmd/ksh93/tests/*.sh:
- Fix a number of mistaken tries to count errors from a subshell.
- Fix miscellaneous minor breakage and typos.
So far all ksh versions accept event numbers referring to
nonexistent history events in history expansion (-H/-o histexpand),
e.g. !9999 is accepted even if the history file has no item 9999.
These expansions seem to show random content from the history file,
sometimes including binary data. Of course an "event not found"
error should have been thrown instead.
hist_expand() in hexpand.c calls hist_seek() (from history.c)
without any bounds checking except verifying the history event
number is greater than zero. This commit adds a bounds check
to hist_seek() itself as it's called from three other places
in history.c, so perhaps this fixes a few other bugs as well.
src/cmd/ksh93/edit/history.c: hist_seek():
- Use the hist_min() and hist_max() macros provided in history.h
to check bounds. Note that hist_max() yields the number of the
command line currently being entered, so the maximum for seeking
purposes is actually its result minus 1.
src/lib/libast/features/standards:
- Do not emit #defines for the typ u_long test which is only used
as a heuristic in subsequent tests in this file. (Note that 'set'
can set and unset any iffe command-line --option at runtime.)
- Remove definition of _ISOC99_SOURCE macro. This is another old
GNU thing; feature_test_macros(7) says invoking the compiler with
the option -std=c99 has the same effect. But modern GCC has C11
with GNU extensions as the default, which is fine. If a
particular standard is desired, pass a -std=... flag in $CC.
src/cmd/ksh93/features/rlimits:
- Remove overlooked Linux *64* types/functions hackery.
After defining standards macros it caused a build failure
on at least one version of Void Linux (but not 5.15.14_1).
Thanks to @JohnoKing for the report.
src/cmd/ksh93/sh/subshell.c,
src/lib/libdll/dllnext.c:
- Remove now-redundant local definitions of _GNU_SOURCE and
__EXTENSIONS__ macros.
src/cmd/ksh93/tests/builtins.sh:
- Fix broken sed invocation (re: 41829efa).
The more notable ones are:
src/lib/libast/features/standards:
- Do not redefine _GNU_SOURCE and _FILE_OFFSET_BITS if already
defined from $CCFLAGS. Thanks to @hyanias for the heads-up.
(re: 289dd46c)
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/include/shell.h,
src/cmd/ksh93/sh/args.c,
src/cmd/ksh93/sh/name.c:
- Remove -T test code activation option. It was basically unused.
The only thing it did was intentionally introduce a memory leak
in table_unset() if the 4th bit in the option argument was set.
A search in ast-open-history reveals a few more trivial test uses
that were later deleted, but nothing interesting.
src/cmd/ksh93/tests/{basic,path}.sh:
- Skip a couple of tests on AIX avoid hangs, at least one of which
is not ksh's fault. Thanks to @HansH111 for the report.
src/cmd/ksh93/tests/builtins.sh:
- Change one awk use to a more portable sed invocation to placate
systems with ancient awk commands, such as AIX. (re: de795e1f)
These are minor things I accumulated over the last month or so.
Notable changes:
src/lib/libast/features/api,
src/lib/libast/misc/state.c,
src/lib/libast/comp/conf.tab,
src/cmd/ksh93/include/defs.h:
- Bump internal libast version to 20220101L. We've made a few
additions to the API, at least pathicase (see 71934570, ca3ec200)
and astconf_long (see c2ac69b2), so this should have been done
already. This also updates '/opt/ast/bin/getconf _AST_VERSION'.
- Use AST_VERSION instead of outdated _AST_VERSION.
- In state.c, use AST_VERSION instead of hardcoding the version.
src/cmd/ksh93/sh/xec.c:
- Remove 'restorefd' variable, unused as of 42becab6.
- Remove 'cmdrecurse' function and SH_RUNPROG macro; this was once
used by a few libcmd commands, but ast-open-archive reveals it's
unused as of ast 1999-12-25.
src/cmd/ksh93/sh/*.c:
- Where available, use e_dot instead of "." for consistency; it is
defined as an extern so we might as well use it.
src/cmd/ksh93/tests/*.sh:
- When reporting signal names in fails, include the SIG prefix.
- Fix a broken process hang test in subshell.sh.
src/lib/libast/man/sfdisc.3:
- Removed. The interfaces described here never made it out of AT&T;
they do not exist in any libast version in ast-open-archive.
Resolves: https://github.com/ksh93/ksh/issues/426
In the times(3) fallback for the time keyword (which can be enabled
in xec.c by undefining _lib_getrusage and timeofday), ksh will
print the obtained time incorrectly if TIMEFORMAT is set to use a
precision level of three:
$ TIMEFORMAT=$'\nreal\t%3lR'
$ time sleep .080
real 0m00.008s # Should be '00.080s'
This commit corrects that issue by using 10^precision to get the
correct fractional scaling. Note that the fallback still doesn't
support a true precision level of three (times(3) alone doesn't
support it), so this in effect pads a zero to the end of the output
when the precision level is three.
Additional change to tests/builtins.sh:
- While fixing the above issue I found out that ksh93v- broke
support for passing microseconds to the sleep builtin in the form
of <num>U. I've added a regression test for that bug to ensure it
isn't backported to ksh93u+m by accident.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
The code used to fork subshells when creating/changing aliases will
always fork, even when the alias tree isn't changed:
$ echo $(unalias --man 2> /dev/null; echo $$ ${.sh.pid})
375097 375107
$ alias foo=bar; echo $(alias -p foo; echo $$ ${.sh.pid})
alias foo=bar 375097 375110
This is a bit inefficient, so this commit avoids forking a subshell
unless at least one change is made to the alias table.
src/cmd/ksh93/bltins/typeset.c:
- b_alias(), b_unalias(): Remove sh_subfork() calls.
- setall(): When creating an alias (name contains '='), fork a
virtual subshell before calling nv_open() to add the node.
- unall():
- When unsetting all aliases (-a), fork subshell before dtclear().
- When unsetting one alias, fork subshell before nv_delete().
- Move sh_pushcontext() and sh_popcontext() expansions so that
sh_subfork() is not in between them, as that would cause
program flow corruption or a crash.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
The referenced commit introduced a bug that caused command
substitutions to hang, writing infinite zero bytes, when
redirecting standard output on a built-in comand that forks the
command substitution subshell.
The bug was caused by removing the fork when redirecting standard
output in a non-permanent manner. However, simply reintroducing the
fork causes multiple regressions that we had fixed in the meantime.
Thankfully, it looks like this forking workaround is only necessary
when redirecting the output of built-ins. It appears that moving
workaround from io.c to the built-ins handling code in sh_exec() in
xec.c, right before calling sh_redirect(), allows reintroducing the
forking workaround for non-permanent redirections without causing
other regressions.
It would be better if the underlying cause of the hang were fixed
so the workaround becomes unnecessary, but I don't think that is
going to happen any time soon (AT&T didn't manage, either).
src/cmd/ksh93/sh/io.c: sh_redirect():
- Remove forking workaround for redirecting stdout in a comsub.
src/cmd/ksh93/sh/xec.c: sh_exec(): TCOM: built-ins handling code:
- Reimplement the workaround here.
Resolves: https://github.com/ksh93/ksh/issues/416
Attempting to complete file names in vi mode using tab completion can
fail if the last character on the command line is a space. Reproducer
(note that this bug doesn't occur in emacs mode):
$ set -o vi
$ mkdir '/tmp/foo bar'
$ test -d /tmp/foo\ <Tab>
src/cmd/ksh93/edit/vi.c:
- Don't disable tab completion or reset the tab count just because the
last character on the command line is a space. This bugfix was
backported from ksh93v- 2014-06-06.
src/cmd/ksh93/tests/pty.sh:
- Add a regression test for the tab completion bug.
This commit makes various different improvements to the documentation:
- sh.1: Backported (with changes) mandoc warning fixes from ksh2020
for the ksh93(1) man page: <https://github.com/att/ast/pull/1406>
- Removed unnecessary spaces at the end of lines to fix a few other
mandoc warnings.
- Fixed various typos and capitalization errors in the documentation.
- ANNOUNCE: Document the addition of the ${.sh.pid} variable
(re: 9de65210).
- libast/man/str*: Update the man pages for the libast str* functions
to improve how accurately each function is described.
- ksh93/README: Update regression test/compatibility notes to include
OpenBSD 7.0, FreeBSD 13.0 and WSL running Ubuntu 20.04.
- Change a few places to store the return value from strlen in a
size_t variable rather than signed int.
- comp/setlocale.c: To avoid confusion of two separate variables named
lang, the function local variable has been renamed to langidx.
This fixes the use after free issue that caused typeset -m to crash on
older versions of OpenBSD and under ASan. The problem that was causing
the failure was that the ap pointer wasn't set to null after the memory
associated with it was freed. This commit backports a bugfix from
ksh93v- 2013-06-28 that sets ap to null before freeing the associated
memory and adds a check that makes sure ap is still a valid pointer
before calling array_unscope().
tests/types.sh changes:
- Avoid redirecting stderr to /dev/null, as this test shouldn't print
anything to stderr.
- Apply error message improvement from
https://github.com/ksh93/ksh/issues/231#issue-834252084.
tests/arrays.sh change:
- Apply error message improvement from
https://github.com/ksh93/ksh/issues/229#issue-834240645 (re: 7c7fde75).
Resolves: https://github.com/ksh93/ksh/issues/231
After 'unset CDPATH', CDPATH continued to work as if nothing
happened. Unsetting it should be a valid way to deactivate it.
This bug is in every ksh93 version.
src/cmd/ksh93/bltins/cd_pwd.c: b_cd():
- Fix a manifest logic error: first check if CDPATH (CDPNOD) is
unset before assigning to 'cdpath', not the other way around.
Setting the 'cdpath' pointer is what activates the CDPATH search.
This commit primarily makes changes that allow the regression tests to
run without any of the /opt/ast/bin builtins compiled into ksh. It also
makes a minor improvement to one of the tests in locale.sh by
shellquoting an error message.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
On modern operating systems, memory management is non-deterministic
(i.e. random, unpredictable) to varying degrees. This makes testing
for memory leaks a nightmare as the OS may decide to randomly grow
a process's memory allocation at any time for no apparent reason,
causing intermittent test failures that do not represent real
memory leaks. So far, the leaks test tried to cope with this by
using a large number of iterations plus a certain amount of bytes
of tolerance per iteration. This was inefficient and on some
systems still did not fully eliminate intermittent test failures.
This commit introduces a new testing algorithm that is designed to
cope with a large degree of unpredictability. Instead of a fixed
number of test iterations, it defines a maximum (16384), dividing
them in blocks of 128 iterations. It also defines a minimum number
of sequential "good" iteration blocks, counted if memory usage did
not increase from one block to the next. That minimum number is set
to 16. The theory is that if we can get 16 "good" iteration blocks
in a row, we can safely assume it's not a real memory leak, break
the loop, and consider the test succeeded. That "good" sequence is
allowed to occur at any point in the loop, creating a high built-in
tolerance for non-deterministic shenanigans. It also speeds up the
tests, as successful tests can bow out at 16 * 128 == 2048
iterations if they're lucky. If the OS decides to randomly grow the
memory heap, it may take more tries, but almost (?) certainly not
more than the maximum 16384 (128 blocks). If the counter reaches
that, then we assume a memory leak and throw a test failure.
We're also no longer testing with byte granularity in any case; the
randomness of memory management makes that pointless. All getmem()
function versions now return kibibytes (1024 bytes).
This should eliminate the need for workarounds such as initial
iterations to "steady the state" or a tolerance of a certain number
of bytes. I've experimentally determined the exact values
(max_iter, block_iter, min_good_blocks) that seem to work reliably
on all systems I've tested. They are easy to tweak if necessary.
To make all this manageable, this commit hides all the supporting
code in a triplet of aliases (TEST, DO, DONE) that, when used
correctly, create a grammatically robust shell code block: you can
add redirections, pipe into it, etc. as expected. This makes the
actual tests a great deal easier to read as well.
src/cmd/ksh93/tests/pty.sh:
- Implement new leaks testing framework as described and convert
all the tests to it.
- Mark known leaks with a 'known' variable. Print non-fail warnings
for all known leaks, but skip the tests by default. Test them
only if DEBUG is exported. This is better than commenting them
out as we will no longer be tempted to forget about these.
- Move the test for large command substitutions to subshell.sh --
it's not in fact a leak test; instead, it checks that command
substitutions don't lose data.
src/cmd/ksh93/tests/_common: err_exit():
- Since we're printing more warnings, clearly mark all test
failures with 'FAIL:' to make them stand out.
src/cmd/ksh93/tests/shtests.
src/cmd/ksh93/tests/pty.sh:
- Special-case leaks.sh for counting tests; grep ^TEST.
- Special-case pty.sh as well while we're at it by grepping tst()
calls. Remove all the dummy '# err_exit #' comments from pty.sh
as they are now no longer used for counting the tests.
The variable 'i' had already been used for a non-numeric purpose,
so when declaring it integer in a subshell it's necessary to
initialise it with value 0 or an arithmetic error is shown (which
does not interrupt the test or make it fail).
For the record, the errpr was mine, not Johnothan's.
This commit backports the whence '-t' option from ksh93v-. The '-t'
option is useful when one needs to identify the type of a command.
The '-t' flag was added by ksh93v- for compatibility with Bash.
It should be noted the ksh93v- patch had one bug, which this commit
fixes. Path-bound builtins from /opt/ast/bin were classified as
files if loaded from /opt/ast/bin in the PATH. Reproducer:
$ PATH=/opt/ast/bin whence -t cat
file
src/cmd/ksh93/bltins/whence.c:
- Simplify the bitmask values for the command and whence builtin
flags.
- Add the -t flag to the whence and type builtins. To prevent bugs,
-t will always override -v if both of those flags were passed.
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/sh.1:
- Add documentation for the new -t option.
Depending on the OS, the heredoc.sh regression tests, and possibly
others, still crashed with the -x option (xtrace) on.
Analysis: The lexer crashes in lex_advance(). Something has caused
an inconsistent lexer state, and it happened earlier on, so the
backtrace is useless for figuring out where that happened.
But I think I've found it. It's the sh_mactry() call here:
src/cmd/ksh93/sh/xec.c, lines 2800 to 2807 in f7213f03
2800: if(!(cp=nv_getval(sh_scoped(shp,PS4NOD))))
2801: cp = "+ ";
2802: else
2803: {
2804: sh_offoption(SH_XTRACE);
2805: cp = sh_mactry(shp,cp);
2806: sh_onoption(SH_XTRACE);
2807: }
sh_mactry() needs to parse the contents of $PS4 to perform
expansions and command substitutions in it, which involves the
lexer. If that happens in a here-document, the lexer is in the C
function call stack, in the middle of parsing the here-document.
Result: inconsistent lexer state. Solution: save and restore lexer
state in sh_mactry().
After this commit, all regression tests should pass with the
'-x'/'--xtrace' option in use, with no errors or crashes.
Note for backporters: this fix depends both on on d7cada7b and on
the consistency fix for the Lex_t type's size applied in a7ed5d9f.
src/cmd/ksh93/include/shlex.h:
- Cosmetic fix: remove a copied & pasted backslash. (re: a7ed5d9f)
src/cmd/ksh93/sh/macro.c: sh_mactry():
- Save and restore the lexer state before letting sh_mactrim()
indirectly parse and execute code.
src/cmd/ksh93/tests/*.sh:
- Turn off xtrace in various command substitutions that contain
2>&1 redirections, so that the xtrace output is not caught by
the command substitutions, causing tests to fail incorrectly.
- Turn off xtrace for a few code blocks with 2>&1 redirections,
stopping xtrace output from being written to standard output.
Resolves: https://github.com/ksh93/ksh/issues/306 (again)
This commit was originally intended to fix just one bug with shcomp's
handling of 'alias -p', but while fixing that I found a large number
of related issues in the alias command's -p, -t and -x options. The
current patch provides bugfixes for all of the bugs listed below:
1) Listing aliases in a script with 'alias -p' or 'alias' broke
shcomp's bytecode output:
https://github.com/ksh93/ksh/issues/87#issuecomment-813819122
2) Listing individual aliases with the -p option doesn't work:
$ alias foo=bar bar=foo
$ alias foo
foo=bar
$ alias -p foo # No output
3) Listing specific tracked aliases with -pt does not display them
in a reusable format, but rather adds another tracked alias:
$ hash -r cat vi
$ alias -pt vi # No output
$ alias -pt rm
$ alias -t
cat=/usr/bin/cat
rm=/usr/bin/rm
vi=/usr/bin/vi
4) Listing all tracked aliases with -pt does not output them in a
reusable format (the resulting command printed only creates a
normal alias, which is different from a tracked alias):
$ hash -r cat
$ alias -pt
alias cat=/usr/bin/cat # Expected 'alias -t cat'
5) Listing a non-existent alias with -p doesn't cause an error:
$ unalias -a
$ alias -p notanalias # No output
$ echo $?
0
$ alias notanalias
notanalias: alias not found
$ echo $?
1
$ hash -r
$ alias -pt notacommand # No output
$ echo $?
0
6) Attempting to list 256 non-existent aliases results in exit
status zero:
$ unalias -a
$ alias $(awk -v ORS= 'BEGIN { for(i=0;i<256;i++) print "x "; }')
x: alias not found
--cut error message--
$ echo $?
0
Changes:
- typeset.c: Avoid printing anything while shcomp is compiling a
script. This is needed because the alias command is run by shcomp
to prevent parsing issues.
- b_alias(): To avoid adding tracked aliases with -pt, set
tdata.aflag to '+' so that setall() and other related functions
only list tracked aliases.
- b_alias(): Set tdata.pflag to 1 so that setall() and other
functions recognize -p was passed.
- print_value(): Add support for listing specific aliases with
'alias -p'.
- setall(): To avoid any issues with zombie tracked aliases (see also
the regression tests) ignore tracked alias nodes marked with the
NV_NOALIAS attribute. This bit is set for tracked alias nodes by
the nv_rehash() function.
- setall(): For backward compatibility, continue incrementing the
exit status for each invalid alias and tracked alias passed. This
was already how alias behaved when listing aliases without -p, so
using -p shouldn't cause a change in behavior:
$ unalias -a
$ alias foo bar
foo: alias not found
bar: alias not found
$ echo $?
2
To fix bug 6, the exit status is set to one if an enforced 8-bit
exit status would be zero.
- print_namval(): Set the prefix to 'alias -t' so that listing
tracked aliases with 'alias -pt' works correctly.
- data/msg.c and include/name.h: Add an error message for when
'alias -pt' doesn't find a tracked alias.
- tests/alias.sh: Add a ton of regression tests for the bugs fixed in
this commit.
- sh/args.c: A process substitution run in a profile script may print
its PID as if it was a command spawned with '&'. Reproducer:
$ cat /tmp/env
true >(false)
$ ENV=/tmp/env ksh
[1] 730227
$
This bug is fixed by turning off the SH_PROFILE state while running
a process substitution.
- sh/subshell.c: The SH_INTERACTIVE fix in 3525535e renders the extra
check for SH_PROFILE redundant, so it has been removed.
- tests/io.sh: Update the procsub PIDs test to also check the result
after using process substitution in a profile script.
- Redirect error output from the ulimit builtin (re: 3e58851f).
- Fix the test failure for 'cd -eP' on illumos by making a directory
symlink first, then removing the symlink after cd.
- Fix the test failure for 'getconf -l' on illumos by quoting
strings with the -q option.
- astconf.c: Only quote strings if the -q option was passed.
- Improve error messages from intermittently failing types.sh tests
Inserting the _common script instead of sourcing it caused all test
failures in shcomp runs to be reported with the number of lines in
_common added.
src/cmd/ksh93/shtests:
- Only incorporate the aliases from _common; dot/source the rest of
the code as normal. Replace the first few lines with the aliases
to avoid affecting $LINENO; they are comments anyway.
Announcing: KornShell 93u+m 1.0.0-beta.2
https://github.com/ksh93/ksh
In May 2020, when every KornShell (ksh93) development project was
abandoned, development was rebooted in a new fork based on the last
stable AT&T version: ksh 93u+. This new fork is called ksh 93u+m as a
permanent nod to its origin. We're restarting it at version 1.0. Seven
months after the first beta, the second one is ready. Please test this
second beta and report any bugs you find, or help us fix known bugs.
We're now the default ksh93 in some OS distributions, at least Debian
and Slackware! Even though we don't think it's stable release quality
yet, the consensus seems to be that 93u+m is already much better than
the last AT&T release.
Main developers: Martijn Dekker, Johnothan King, hyenias
Contributors: Andy Fiddaman, Anuradha Weeraman, Chase, Gordon Woodhull,
Govind Kamat, Harald van Dijk, Lev Kujawski, Marc Wilson, Ryan Schmidt,
Sterling Jensen
HOW TO GET IT
Please download the source code tarball from our GitHub releases page:
https://github.com/ksh93/ksh/releases
To build, follow the instructions in README.md or src/cmd/ksh93/README.
HOW TO GET INVOLVED
To report a bug, please open an issue at our GitHub page (see above).
Alternatively, email me at martijn@inlv.org with your report.
To get involved in development, read the brief policy information in
README.md and then jump right in with a pull request or email a patch.
See the TODO file in the top-level directory for a to-do list.
*** MAIN CHANGES between 1.0.0-beta.1 and 1.0.0-beta.2 ***
New features in built-in commands:
- 'cd' now supports an -e option that, when combined with -P, verifies
that $PWD is correct after changing directories; this helps detect
access permission problems. See:
https://www.austingroupbugs.net/view.php?id=253
- 'printf' now supports a -v option as in bash. This assigns formatted
output directly to variables, which is very fast and will not strip
final newline (\n) characters.
- The 'return' command, when used to return from a function, can now
return any status value in the 32-bit signed integer range, like on
zsh. However, due to a traditional Unix kernel limitation, $? is
still trimmed to its least significant 8 bits whenever leaving a
(sub)shell environment.
- 'test'/'[' now supports all the same operators as [[ (including =~,
\<, \>) except for the different 'and'/'or' operators. Note that
'test'/'[' remains deprecated due to its unfixable pitfalls;
[[ ... ]] is recommended instead.
Shell language changes:
- Several improvements were made to the --noexec shell code linter.
- Arithmetic expressions in native ksh mode no longer interpret a
number with a leading zero as octal in any context. Use 8#octalnumber
instead (e.g. 8#400 == 256). Arithmetic expressions now also behave
identically within and outside ((...)) and $((...)).
- POSIX compatibility mode fixes (only applicable with the --posix shell
option on):
- A leading zero is now consistently recognised as introducing an octal
number in all arithmetic contexts.
- $((inf)) and $((nan)) are now interpreted as regular variables.
- The '.' built-in no longer runs ksh functions and now only runs
files.
Bugs fixed:
- '.' and '..' are now once again completed by tab completion.
- If SIGINT is set to ignore, the interactive shell no longer exits on
Ctrl+C.
- ksh now builds and runs on Apple's new M1 hardware.
- The 'return' and 'exit' commands no longer risk triggering actual
signals by returning or exiting with a status > 256.
- Ksh no longer behaves badly when parsing a type definition command
('typeset -T' or 'enum') without executing it or when executing it in
a subshell. Types can now safely be defined in subshells and defined
conditionally as in 'if condition; then enum ...; fi'.
- Discipline functions, especially those applied to PS2 or .sh.tilde,
will no longer crash your shell upon being interrupted or throwing an
error.
- Fixed a bug that could corrupt output if standard output is closed
upon initialising the shell.
- Fixed a bug in the [[ ... ]] compound command: the '!' logical
negation operator now correctly negates another '!', e.g.,
[[ ! ! 1 -eq 1 ]] now returns 0/true. Note that this has always been
the case for 'test'/'['.
- Fixed SHLVL so that replacing ksh by itself (exec ksh) will not
increase it.
- Arithmetic expressions are no longer allowed to assign out-of-range
values to variables of types declared with enum.
- The 'time' keyword no longer makes the --errexit shell option
ineffective.
- Various bugs in libcmd built-in commands (those bound to the
/opt/ast/bin path by default) have been fixed.
- Various other crashing bugs have been fixed.
Fixes for the shcomp byte code compiler:
- shcomp is now able to compile scripts that define types using enum.
- shcomp now refuses to mess up your terminal by writing bytecode
to it.
*** MAIN CHANGES between ksh 93u+ 2012-08-01 and 93u+m 1.0.0-beta.1 ***
Hundreds of bugs have been fixed, including many serious/critical bugs.
This includes upstreamed patches from OpenSUSE, Red Hat, and Solaris, fixes
backported from the abandoned 93v- beta and ksh2020 fork, as well as many
new fixes from the community. See the NEWS file for more information, and
the git commit log for complete documentation of every fix. Incompatible
changes have been minimised, but not at the expense of fixing bugs. For a
list of potentially incompatible changes, see src/cmd/ksh93/COMPATIBILITY.
Though there was a "no new features, bugfixes only" policy, some new
features were found necessary, either to fix serious design flaws or to
complete functionality that was evidently intended, but not finished.
Below is a summary of these new features.
New command line editor features:
- The forward-delete and End keys are now handled as expected in the
emacs and vi built-in line editors.
- In the vi and emacs line editors, repeat count parameters can now also
be used for the arrow keys and the forward-delete key. E.g., in emacs
mode, <ESC> 7 <left-arrow> will now move the cursor seven positions to
the left. In vi control mode, this would be entered as: 7 <left-arrow>.
New shell language features:
- The &>file redirection shorthand (for >file 2>&1) is now available for
all scripts and interactive sessions and not only for profile/login
scripts, bringing ksh 93u+m in line with mksh, bash, and zsh.
- File name generation (a.k.a. pathname expansion, a.k.a. globbing) now
never matches the special navigational names '.' (current directory)
and '..' (parent directory). This change makes a pattern like .*
useful; it now matches all hidden files (dotfiles) in the current
directory, without the harmful inclusion of '.' and '..'.
- Tilde expansion can now be extended or modified by defining a
.sh.tilde.get or .sh.tilde.set discipline function. This replaces a
2004 undocumented attempt to add this functionality via a .sh.tilde
command, which never worked and crashed the shell. See the manual for
details on the new method.
New features in built-in commands:
- Usage error messages now show the --help/--man self-documentation options.
- Path-bound built-ins (such as /opt/ast/bin/cat) can now be executed by
invoking the canonical path, so the following will now work as expected:
$ /opt/ast/bin/cat --version
version cat (AT&T Research) 2012-05-31
- 'command -x' now looks for external commands only, skipping built-ins.
In addition, its xargs-like functionality no longer freezes the shell on
Linux and macOS, making it effectively a new feature on these systems.
- 'redirect' now checks if all arguments are valid redirections before
performing them. If an error occurs, it issues an error message instead
of terminating the shell.
- 'suspend' now refuses to suspend a login shell, as there is probably no
parent shell to return to and the login session would freeze.
- 'times' now gives high precision output in a POSIX compliant format.
- 'typeset' now gives an informative error message if an incompatible
combination of options is given.
- 'whence -v/-a' now reports the location of autoloadable functions.
New features in shell options:
- A new --globcasedetect shell option is added on OSs where we can
check for a case-insensitive file system (currently Windows/Cygwin,
macOS, Linux and QNX 7.0+). When this option is turned on, file name
generation (globbing), as well as file name tab completion on
interactive shells, automatically become case-insensitive on file
systems where the difference between upper and lower case is ignored
for file names. This is transparently determined for each directory, so
a path pattern that spans multiple file systems can be part
case-sensitive and part case-insensitive.
- A new --nobackslashctrl shell option disables the special escaping
behaviour of the backslash character in the emacs and vi built-in
editors. Particularly in the emacs editor, this makes it much easier to
go backward, insert a forgotten backslash into a command, and then
continue editing without having your next cursor key replace your
backslash with garbage. Note that Ctrl+V (or whatever other character
was set using 'stty lnext') always escapes all control characters in
either editing mode.
- A new --posix shell option has been added to ksh 93u+m that makes the
ksh language more compatible with other shells by following the POSIX
standard more closely. See the manual page for details. It is enabled by
default if ksh is invoked as sh, otherwise it is disabled by default.
- Enhancement to -G/--globstar: symbolic links to directories are now
followed if they match a normal (non-**) glob pattern. For example, if
'/lnk' is a symlink to a directory, '/lnk/**' and '/l?k/**' now work as
you would expect.
Strings compared in [[ with the > and < operators should be compared
lexically. This does not work when the strings are single digits, as
the parser interprets it as a syntax error:
$ [[ 10<2 ]] # 10 lexically sorts before 2
$ echo $?
0
$ [[ 1<2 ]]
/usr/bin/ksh: syntax error: `<' unexpected
$ echo $?
3
src/cmd/ksh93/sh/lex.c:
- Don't interpret numbers next to > and < as a redirection while
inside of [[. This bugfix was backported from ksh93v- 2014-06-25.
src/cmd/ksh93/tests/bracket.sh:
- Add regression tests for the > and < operators.
New version. I'm pretty sure the problems that forced me to revert
it earlier are fixed.
This commit mitigates the effects of the hack explained in the
referenced commit so that dummy built-in command nodes added by the
parser for declaration/assignment purposes do not leak out into the
execution level, except in a relatively harmless corner case.
Something like
if false; then
typeset -T Foo_t=(integer -i bar)
fi
will no longer leave a broken dummy Foo_t declaration command. The
same applies to declaration commands created with enum.
The corner case remaining is:
$ ksh -c 'false && enum E_t=(a b c); E_t -a x=(b b a c)'
ksh: E_t: not found
Since the 'enum' command is not executed, this should have thrown
a syntax error on the 'E_t -a' declaration:
ksh: syntax error at line 1: `(' unexpected
This is because the -c script is parsed entirely before being
executed, so E_t is recognised as a declaration built-in at parse
time. However, the 'not found' error shows that it was successfully
eliminated at execution time, so the inconsistent state will no
longer persist.
This fix now allows another fix to be effective as well: since
built-ins do not know about virtual subshells, fork a virtual
subshell into a real subshell before adding any built-ins.
src/cmd/ksh93/sh/parse.c:
- Add a pair of functions, dcl_hactivate() and dcl_dehacktivate(),
that (de)activate an internal declaration built-ins tree into
which check_typedef() can pre-add dummy type declaration command
nodes. A viewpath from the main built-ins tree to this internal
tree is added, unifying the two for search purposes and causing
new nodes to be added to the internal tree. When parsing is done,
we close that viewpath. This hides those pre-added nodes at
execution time. Since the parser is sometimes called recursively
(e.g. for command substitutions), keep track of this and only
activate and deactivate at the first level.
(Fixed compared to previous version of this commit: calling
dcl_dehacktivate() when the recursion level is already zero is
now a harmless no-op. Since this only occurs in error handling
conditions, who cares.)
- We also need to catch errors. This is done by setting libast's
error_info.exit variable to a dcl_exit() function that tidies up
and then passes control to the original (usually sh_exit()).
(Fixed compared to previous version of this commit: dcl_exit()
immediately deactivates the hack, no matter the recursion level,
and restores the regular sh_exit(). This is the right thing to
do when we're in the process of erroring out.)
- sh_cmd(): This is the most central function in the parser. You'd
think it was sh_parse(), but $(modern)-form command substitutions
use sh_dolparen() instead. Both call sh_cmd(). So let's simply
add a dcl_hacktivate() call at the beginning and a
dcl_deactivate() call at the end.
- assign(): This function calls path_search(), which among many
other things executes an FPATH search, which may execute
arbitrary code at parse time (!!!). So, regardless of recursion
level, forcibly dehacktivate() to avoid those ugly parser side
effects returning in that context.
src/cmd/ksh93/bltins/enum.c: b_enum():
- Fork a virtual subshell before adding a built-in.
src/cmd/ksh93/sh/xec.c: sh_exec():
- Fork a virtual subshell when detecting typeset's -T option.
Improves fix to https://github.com/ksh93/ksh/issues/256
The killpg(getpgrp(),SIGINT) call added to ed_getchar() in that
commit caused the interactive shell to exit on ^C even if SIGINT is
being ignored. We cannot revert or remove that call without
breaking job control. This commit applies a new fix instead.
Reproducers fixed by this commit:
SIGINT ignored by child:
$ PS1='childshell$ ' ksh
childshell$ trap '' INT
childshell$ (press Ctrl+C)
$
SIGINT ignored by parent:
$ (trap '' INT; ENV=/./dev/null PS1='childshell$ ' ksh)
childshell$ (press Ctrl+C)
$
SIGINT ignored by parent, trapped in child:
$ (trap '' INT; ENV=/./dev/null PS1='childshell$ ' ksh)
childshell$ trap 'echo test' INT
childshell$ (press Ctrl+C)
$
I've experimentally determined that, under these conditions, the
SFIO stream error state is set to 256 == 0400 == SH_EXITSIG.
src/cmd/ksh93/sh/main.c: exfile():
- On EOF or error, do not return (exiting the shell) if the shell
state is interactive and if sferror(iop)==SH_EXITSIG.
- Refactor that block a little to make the new check fit in nicely.
src/cmd/ksh93/tests/pty.sh:
- Test the above three reproducers.
Fixes: https://github.com/ksh93/ksh/issues/343
