SHOPT_KIA enables the -R option that generates a cross-reference
database from a script. However, no tool to analyse this database
is shipped or seems to be available anywhere (in spite of multiple
people looking for one), and the format is very opaque. No usage
examples are known or findable on the internet. This seems like it
should not be compiled in by default, although we'll keep the code
in case some way to use it is found.
src/cmd/ksh93/SHOPT.sh:
- Disable SHOPT_KIA by default by removing the default 1 value.
src/cmd/ksh93/sh/args.c, src/cmd/ksh93/sh/parse.c:
- Fix a couple of preprocessor logic bugs that made it impossible
to compile ksh without SHOPT_KIA.
src/cmd/ksh93/data/builtins.c:
- Fix typo in -R doc in ksh --man (in case SHOPT_KIA is enabled).
src/cmd/ksh93/sh.1:
- Since sh.1 is not generated dynamically, remove the -R doc.
When building old code for debugging purposes (e.g. when doing 'git
bisect' runs), it's best to use the current build system even with
the old code, because the old build system was very broken. E.g.:
git checkout (some old commit)
git checkout master bin src/cmd/INIT # use new build system
bin/package make
However, that became impossible in 6cc2f6a0 because the new
SHOPT.sh script was unconditionally sourced. The error caused the
script to exit because '.' is a special builtin.
bin/package, src/cmd/INIT/package.sh:
- If src/cmd/ksh93/SHOPT.sh doesn't exist, issue a warning instasd
of trying to source it.
The build system is adapted to make SHOPT_* compile-time options
editable without nmake. We can now easily change ksh's compile-time
options by editing src/cmd/ksh93/SHOPT.sh. The bin/package script
is adapted to turn these into compile flags. This resolves the most
important drawback of not using nmake.
Also, mamake now has support for indented Mam (Make Abstract
Machine) code. Only one type of block (make...done) is supported in
Mamfiles, so they are easy to indent automatically. A script to
(re)do this is included.
Since nmake is not going to be restored (it has too many problems
that no one is interested in fixing), this at least makes mamake
significantly easier to work with.
The Makefiles are deleted. They may still be handy for reference to
understand the Mamfiles, but they haven't actually matched the
Mamfiles for a while -- and you can still look in the git history.
Deleting them requires some adaptations to bin/package and mamake.c
because, even though they do not use those files, they still looked
for them to decide whether to build code in a directory.
Finally, this commit incorporates some #pragmas for clang to
suppress annoying warnings about the coding style used in this
historic code base. (gcc does not complain so much.)
src/cmd/ksh93/SHOPT.sh:
- Added.
bin/package, src/cmd/INIT/package.sh:
- cd into our own directory in case we were run from another dir.
- $makefiles: only look for Mamfiles.
- Add ksh compile-options via KSH_SHOPTFLAGS. Include SHOPT.sh.
- make_recurse(): Do not write a missing Makefile.
- finalize environment: Look for Mamfiles instead of Makefiles.
src/cmd/INIT/mamake.c:
- Tell clang to suppress annoying warnings about coding style.
- Update version string and self-documentation.
- input(): Add support for indented Mam code by skipping initial
whitespace on each input line.
- files[]: Instead of looking for various of Makefiles to decide
where to build, only look for Mamfiles.
src/Makefile, src/cmd/INIT/Makefile, src/cmd/Makefile,
src/cmd/builtin/Makefile, src/cmd/ksh93/Makefile, src/lib/Makefile,
src/lib/libast/Makefile, src/lib/libcmd/Makefile,
src/lib/libdll/Makefile, src/lib/libsum/Makefile:
- Removed.
src/Mamfile, src/cmd/INIT/Mamfile, src/cmd/Mamfile,
src/cmd/builtin/Mamfile, src/cmd/ksh93/Mamfile, src/lib/Mamfile,
src/lib/libast/Mamfile, src/lib/libcmd/Mamfile,
src/lib/libdll/Mamfile, src/lib/libsum/Mamfile:
- Indent the code with tabs.
- In ksh93/Mamfile, add ${KSH_SHOPT_FLAGS} to every $CC command.
- In ksh93/Mamfile, add "prev SHOPT.sh" for every *.o file
so they are rebuilt whenever SHOPT.sh changes.
bin/Mamfile_indent:
- Added, in case someone wants to re-indent a Mamfile.
src/cmd/INIT/proto.c, src/cmd/INIT/ratz.c, src/cmd/INIT/release.c,
src/lib/libast/features/common, src/lib/libast/include/ast.h:
- Tell clang to suppress annoying warnings about coding style that
it disapproves of (mainly concerning the use of parentheses).
src/cmd/INIT/cc.darwin, src/cmd/INIT/cc.freebsd,
src/cmd/INIT/cc.openbsd:
- Remove now-redundant clang warning suppression flags.
Resolves: https://github.com/ksh93/ksh/issues/60
What is this for? See cefe087d
src/cmd/ksh93/Mamfile:
- Make iffe generate a test for the presence of setproctitle(3).
src/cmd/ksh93/sh/main.c:
- Include setproctitle test result.
- Re-enable fixargs() for FreeBSD and DragonFly BSD.
Disable it for UnixWare.
- fixargs(): Add _lib_setproctitle version. Keep it simple with a
128-character buffer array -- should be plenty for 'ps' output.
- fixargs(): Fix an off-by-one in zeroing the rest of the buffer.
src/cmd/ksh93/tests/basic.sh:
- Update the relevant regression test to run on FreeBSD/DragonFly
and tolerate the "ksh: " prefix added by setproctitle(3).
src/cmd/ksh93/include/version.h:
- Centrally define the 93u+m copyright (SH_RELEASE_CPYR) for adding
to the original AT&T copyright in 'ksh --man' and 'shcomp --man'.
- Centrally define the binary header version number for bytecode
generated by shcomp: SHCOMP_HDR_VERSION.
- Bump SHCOMP_HDR_VERSION from 3 to 4. Converting all the preset
aliases to builtin commands has caused new bytecode to be
incompatible with old ksh. (However, old bytecode runs fine on
93u+m, because shcomp pre-expands the preset aliases.)
src/cmd/ksh93/sh/shcomp.c:
- Instead of keeping its own version date (not changed since 2003),
use the same version string as ksh itself (SH_RELEASE).
- Use SH_RELEASE_CPYR for the extra 93u+m copyright string.
- Use SHCOMP_HDR_VERSION for the bytecode header.
src/cmd/ksh93/sh/parse.c: sh_parse():
- Use SHCOMP_HDR_VERSION for the bytecode version check.
src/cmd/ksh93/data/builtins.c: opt_ksh[]:
- Use SH_RELEASE_CPYR for the extra 93u+m copyright string.
src/cmd/ksh93/COMPATIBILITY:
- Mention that 93u+m shcomp bytecode won't run on older ksh.
- Document changes in printf %T (re: 9526b3fa).
src/cmd/ksh93/README:
- Mention that we run on UnixWare (with major regressions).
https://github.com/ksh93/ksh/pull/159#issuecomment-764667929
This adds informative error messages if incompatible options are
given. It also documents the exclusive -m, -n and -T options on
separate usage lines, as was already done with -f. The usage
message for incompatible options now looks something like this:
| $ ksh -c 'typeset -L10 -F -f -i foo'
| ksh: typeset: -i/-F/-E/-X cannot be used with -L/-R/-Z
| ksh: typeset: -f cannot be used with other options
| Usage: typeset [-bflmnprstuxACHS] [-a[type]] [-i[base]] [-E[n]]
| [-F[n]] [-L[n]] [-M[mapping]] [-R[n]] [-X[n]]
| [-h string] [-T[tname]] [-Z[n]] [name[=value]...]
| Or: typeset -f [name...]
| Or: typeset -m [name=name...]
| Or: typeset -n [name=name...]
| Or: typeset -T [tname[=(type definition)]...]
| Help: typeset [ --help | --man ] 2>&1
(see also the previous commit, e21a053e)
Unfortunately the first "Usage" line has some redundancies with the
"Or:" lines showing separate usages. It doesn't seem to be possible
to avoid this; it's a flaw in how libast generates everything
(usage, help, manual) from one huge getopt(3) string. I still think
the three added "Or:" lines are an improvement as it wasn't
previously shown that these options need to be used on their own.
src/cmd/ksh93/bltins/typeset.c: b_typeset():
- Instead of only showing a generic usage message, add an
informative error message if incompatible options were given.
- Conflicting options detection was failing because NV_LJUST and
NV_EXPNOTE have the same bitmask value. Use a new 'isadjust'
flag for -L/-R/-Z to remember if one of these was set.
- Detect conflict between -L/-R/-Z and a float option, not just -i.
src/cmd/ksh93/include/name.h, src/cmd/ksh93/data/msg.c:
- Add the two new error messages for incompatible options.
src/cmd/ksh93/data/builtins.c: sh_opttypeset[]:
- Add a space after 'float' in in "[+float?\btypeset -lE\b]" as
this makes 'float' appear on its own line, improving formatting.
- Show -m, -n, -T on separate usage lines like -f, as none of these
can be combined with other options.
- Remove "cannot be combined with other options" from -m and -n
descriptions, as that should now be clear from the separate usage
lines -- and even if not, the error message is now informative.
src/cmd/ksh93/sh.1, src/cmd/ksh93/COMPATIBILITY:
- Update.
src/cmd/ksh93/tests/types.sh:
- Remove obsolete test: 'typeset -RF' is no longer accepted.
(It crashed in 93u+, so this is not an incompatibility...)
Resolves: https://github.com/ksh93/ksh/issues/48
For example, this changes 'typeset -Q' (a bad option) from:
| ksh: typeset: -Q: unknown option
| Usage: typeset [-bflmnprstuxACHS] [-a[type]] [-i[base]] [-E[n]]
| [-F[n]] [-L[n]] [-M[mapping]] [-R[n]] [-X[n]]
| [-h string] [-T[tname]] [-Z[n]] [name[=value]...]
| Or: typeset [ options ] -f [name...]
to:
| ksh: typeset: -Q: unknown option
| Usage: typeset [-bflmnprstuxACHS] [-a[type]] [-i[base]] [-E[n]]
| [-F[n]] [-L[n]] [-M[mapping]] [-R[n]] [-X[n]]
| [-h string] [-T[tname]] [-Z[n]] [name[=value]...]
| Or: typeset -f [name...]
| Help: typeset [ --help | --man ] 2>&1
src/lib/libast/misc/optget.c: args():
- Revert the changes done in 6916a873 and ae92cd89. The --help and
--man labels weren't added consistently (they did not show up in
the example above) whereas they did show up unnecessarily in the
manual page itself.
- In the usage section and usage messges, only show an [ options ]
label on the first usage line; don't redundantly repeat on second
and further ("Or:") lines.
- In usage and --help (but not --man), add a new "Help:" line
telling the user about the --help and --man options. This
replaces the reverted changes. Show the 2>&1 redirection as a
reminder that you need to do this to pipe it into a pager, as
everything is written to standard error!
- Add some comments clarifying what I think this code does...
src/cmd/ksh93/tests/builtins.sh:
- Update to match changes in getopts usage output.
src/cmd/ksh93/bltins/typeset.c:
- The new_argv[] array was one item too short (should be argc+2).
- Use AST stakalloc(3) to allocate it instead of a dynamic array;
this restores compatibility with ISO C90.
src/lib/libast/features/standards, src/cmd/INIT/cc.unixware.i386:
- Add support for UnixWare.
- Do not define any standards macros on this system, as on FreeBSD
and DragonFly BSD.
This fixes the following:
trap ':' DEBUG
r=$(exit 123)
echo $? # Expected 123, but actually 0.
Thanks to Koichi Nakashima for the report and reproducer.
src/cmd/ksh93/sh/fault.c: sh_trap():
- Restore the saved current exit status (exitval) for all traps.
Do not except the DEBUG trap from doing that. I've no idea why
this exception was made, but it's not correct.
src/cmd/ksh93/tests/basic.sh:
- Add tests.
Makes progress on: https://github.com/ksh93/ksh/issues/155
Commit d1483150 did not fully fix#153.
Test case from Harald van Dijk that was still failing:
$ mkdir test
$ cd test
$ rmdir $PWD
$ mkdir $PWD
$ ksh -c "(cd /); pwd"
/
Forking a virtual subshell in that case is needed to avoid ending
up in a directory that replaced the PWD, because it will not be
possible for a process to change back to the original directory.
src/cmd/ksh93/bltins/cd_pwd.c:
- When deciding whether to fork, instead of attempting to opendir
the PWD, compare the inodes $PWD and "." to determine if $PWD
still actually refers to the current directory. This uses the
test_inode() function which is also used by 'test foo -ef bar'.
src/cmd/ksh93/tests/subshell.sh:
- Add test based on the above.
Progresses: https://github.com/ksh93/ksh/issues/153
src/cmd/ksh93/tests/variables.sh:
- Fork the subshell with the test that includes unsetting LINENO
and changing its type. Otherwise, some side effect of that leaks
out of the subshell, messing up $LINENO. This is a bug, but it's
low priority -- we may get to it someday. Marked with a TODO.
- Do the LC_* tests in their own subshell. Skip them if changing
LANG to an invalid value does not produce a diagnostic message.
This occurs on OpenBSD and Alpine Linux (with musl libc). It
looks like their C libraries do not verify the locale, so
failures here are not a ksh problem; skip the tests in that case.
AIX on ibm.risc comes with a broken version of ksh88 as /bin/sh
where the following causes breakage in the parser (spurious syntax
errors):
(set -o posix) 2>/dev/null && set -o posix
However, prefixing it with 'command' (while keeping the subshell)
circumvents the problem. So, why not.
(command set -o posix) 2>/dev/null && set -o posix
Solaris Studio 12.5 cc seems to produce incorrect code at -O2
(a.k.a. -xO2) optimisation level; integer variables initialise at
random values, and the behaviour of the shell is so incorrect it
can't even run the test scripts. It does not support -Os so that
is skipped for that compiler. At -O it works fine.
src/cmd/INIT/make.probe:
- By default, only try -Os and -O optimisation flags.
This commit also further mitigates the problems with restoring an
inaccessible or nonexistent PWD on exiting a virtual subshell.
Harald van Dijk writes:
> On a build of ksh with -fsanitize=undefined to help diagnose
> problems:
>
> $ mkdir deleted
> $ cd deleted
> $ rmdir ../deleted
> $ ksh -c '(cd /; (cd /)); :'
> /home/harald/ksh/src/cmd/ksh93/sh/subshell.c:561:22: runtime
> error: null pointer passed as argument 1, which is declared to
> never be null
> Segmentation fault (core dumped)
>
> Note that it segfaults the same with default compilation flags,
> but it does not print out the useful extra message. The code
> assumes that pwd is non-null and passes it to strcmp without
> checking, but it will be null if the current directory cannot be
> determined, for instance because it has been deleted.
src/cmd/ksh93/sh/subshell.c: sh_subshell():
- Avoid the null pointer dereference reported above.
src/cmd/ksh93/bltins/cd_pwd.c: b_cd():
- Fork a virtual subshell even on systems with fchdir(2) if the
present working directory tests as inaccessible on invoking 'cd';
it may no longer exist and fchdir would fail to get a handle.
(For the test we have to opendir(3) the full path to the PWD and
not ".", as the latter may succeed even if the PWD is gone.)
src/cmd/ksh93/data/builtins.c:
- Update 'cd' version string.
Fixes: https://github.com/ksh93/ksh/issues/153
Related: https://github.com/ksh93/ksh/issues/141
src/cmd/ksh93/sh/main.c: sh_main():
- Reading the code makes it obvious that the shp->comdiv-- decrease
in the 'else' block is never reached unless that pointer is still
null, in which case it makes no sense to decrease it. Must be
some kind of missed leftover from old code. Remove the decrease.
This change is backported from the abandoned ksh 93v- beta.
src/cmd/ksh93/sh/subshell.c: sh_subsavefd():
- Do not subtract 1 from fd, as this would cause a negative shift
operand for stdin (fd==0).
If iffe re-ran a test because the source test script changed, but
the result file is unchanged, it didn't update the timestamp of the
result, so the source script remained newer. The build system then
kept pointlessly re-running the test on each rebuild. If a central
test script such as src/lib/libast/features/standards was changed,
this had cascading effects, e.g., causing libast to be rebuilt over
and over as I recompiled small changes elsewhere. Until now, my
workaround was to delete the entire 'arch' directory and start
over. Hopefully that will now no longer be needed.
src/cmd/INIT/iffe.sh:
- If a test output file has not changed after rerunning a test that
has changed, still update the output file's timestamp using
touch(1) to signal that the test has already been run.
Instead, we now link to the libm system math library where needed
by adding -lm to the relevant compile commands in the Mamfiles.
This is not needed on every system but never does any harm.
(This adds more custom edits to the Mamfiles, which were originally
generated from the nmake Makefiles. This takes us further from
restoring nmake, but that already wasn't going to happen anyway,
due to its many problems... the path forward will be to translate
the Mamfiles to some other, current make system such as GNU make.)
bin/package, src/cmd/INIT/package.sh:
- Remove LDFLAGS=-lm hack for DragonFly BSD, NetBSD and Solaris.
src/cmd/builtin/Mamfile,
src/cmd/ksh93/Mamfile,
src/lib/libdll/Mamfile:
- Add -lm where linking failed on any of the three mentioned OSs.
src/lib/libdll/features/dll:
- In the output test program, add missing #include <math.h>, fixing
unknown identifier errors on NetBSD (ldexp, ldexpl).
src/cmd/builtin/features/pty:
- Add missing #include <stdio.h> to make printf work on all systems
(this is just a feature test, no need to bother with sfio here).
src/lib/libast/features/stdio:
- Undef __FILE_T to avoid interference from system headers on QNX.
(There are still other problems preventing a build on QNX 6.5.0.
The shipped version of gcc seems to be broken.)
This now makes ksh build on DragonFly BSD.
bin/package, src/cmd/INIT/package.sh:
- DragonFly also needs the -lm hack for LDFLAGS.
src/cmd/ksh93/sh/main.c, src/cmd/ksh93/tests/basic.sh:
- fixargs() doesn't work on DragonFly either
(re: 9b7c392a, 159fb9ee, cefe087d).
The following are backported from:
https://github.com/att/ast/issues/26#issuecomment-313927854https://github.com/att/ast/pull/19
src/lib/libast/comp/setlocale.c:
- Add missing #include <errno.h> since errno is used.
src/lib/libast/features/standards:
- Do not set any standards macros (_POSIX_SOURCE etc) on FreeBSD or
DragonflyBSD; they disable too much functionality on those.
src/lib/libast/features/wchar:
- Set _STDFILE_DECLARED on DragonFly, too.
src/lib/libast/include/sfio.h, src/lib/libast/include/sfio_t.h,
src/lib/libast/sfio/_sfopen.c, src/lib/libast/sfio/sfclrlock.c,
src/lib/libast/sfio/sfhdr.h, src/lib/libast/sfio/sfnew.c,
src/lib/libast/sfio/sfset.c:
- Rename SF_* macros to SFIO_* to avoid a conflict with system
headers.
src/lib/libast/string/strexpr.c:
- Rename error() to err() to avoid a conflict.
~- and ~+ are ksh93-specific tilde expansions that expand to
$OLDPWD and $PWD, respectively. On some systems, $OLDPWD is not set
on entry to the test script, because it is not exported to the
environment. This made it unset before any 'cd' was executed,
which (correctly) disabled ~- expansion.
src/cmd/ksh93/tests/basic.sh:
- Before testing 'cd ~-', make sure $OLDPWD is set by cd'ing to
/dev first (a directory guaranteed by POSIX).
Solaris /bin/ksh disables the SHOPT_PFSH compile option ("solaris
exec_attr(4) profile execution") with a patch. Since this option
applies to Solaris and variants only, let's upstream that change.
(Solaris now provides pfksh93 as a wrapper around ksh93, and does
the same for other shells, so profiling functionality is no longer
ksh-specific.)
If you want to re-enable it, add -DSHOPT_PFSH to your $CCFLAGS.
Original patch:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/150-CR7168611.patch
src/cmd/ksh93/Makefile:
- Add note that edits in Makefile are ineffective as we do not ship
nmake.
- Disable SHOPT_PFSH, cosmetically.
src/cmd/ksh93/Mamfile:
- Remove -DSHOPT_PFSH from all compiler commands.
bin/package, src/cmd/INIT/package.sh:
- It can depend on the compiler flags passed whether the compiler
produces code for a 64-bit architecture, so pass $CCFLAGS to
the compiler when testing whether it creates 64-bit object code.
README.md:
- Copy-edit of build instructions.
bin/package, src/cmd/INIT/package.sh:
- CCFLAGS overwrites the autodetected optimisation flags (e.g. -Os)
if set. Unfortunately, that also happened when we added something
to CCFLAGS for a release build or to add an extra flag needed by
Solaris. The fix is to use a new flags variable (KSH_RELFLAGS)
instead. This needs to be done in a different place as it needs
to be added to the mamake command as an assignment argument.
- Remove the Solaris CCFLAGS hack; see features/common below.
src/*/*/Mamfile:
- Add ${KSH_RELFLAGS} to all the compiler commands.
src/lib/libast/features/common:
- Enable POSIX standard on Solaris (i.e.: if __sun is defined) by
defining _XPG6 directly in the feature test that generates
ast_std.h, which is indirectly included by everything. This
removes the need to pass -D_XPG6 via CCFLAGS. (Doing so
automatically with gcc was not otherwise possible.)
src/cmd/INIT/cc.sol11.*:
- No longer pass -D_XPG6, as per above.
bin/package, src/cmd/INIT/package.sh:
- The code for detecting a 64-bit object file was seriously broken:
the temporary file name could contain '64' because it included $$,
the current PID, and 64-bit was detected if the output of 'file'
(which includes the complete file name) contained '64'. Fix by
removing the file name from 'file' output before testing.
- Also refactor that code a bit and remove the nonsensical test if
/bin/sh is a 64-bit binary, which is neither here nor there. It's
what the compiler produces that we need to care about.
src/cmd/INIT/make.probe:
- probe_optimize: Also try -O2 and -O, for compilers (such as
Solaris Studio cc) that do not support -Os.
- Use more robust code to loop through possible optimiser flags.
The versions from the Solaris patch require $CC_EXPLICIT to be set,
which is specific to the internal Solaris build environment.
src/cmd/INIT/cc.sol11.*:
- Cope without $CC_EXPLICIT set in environment; fall back to $CC
and if that is not set either, detect whether to use cc or gcc.
- Set appropriate flags for cc (Solaris Studio) or gcc, including
the necessary -D_XPG6 flag, without which ksh crashes on Solaris.
bin/package, src/cmd/INIT/package.sh:
- Update hack to add the -D_XPG6 flag so it applies to gcc only
(note: the src/cmd/INIT/cc.* scripts are never used for gcc).
src/cmd/ksh93/sh/main.c: fixargs():
- Erase the entire length of the command arguments buffer (the
space from argv[0] until environ[0]) so that remnants of longer
command arguments aren't left in 'ps' output when executing a
hashbang-ess script with a shorter command line.
- Disable fixargs() on FreeBSD. It has never had any effect on that
system; apparently it either requires another method to rewrite
arguments for 'ps' output purposes (which?) or it's not possible.
src/cmd/ksh93/tests/basic.sh:
- Skip the test if running on FreeBSD.
This reverts commit 600cb182.
$cc may be a system compiler binary, it is not necessarily a
src/cmd/INIT/cc.* wrapper script; so prefixing 'sh' is wrong.
This applies a patch from Solaris:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/160-CR7175995.patch
There is no public information on why it's needed, but it seems
sensible on the face of it. Using a file called '.profile' in the
PWD on login, without a directory path, is redundant at best, since
"$HOME/.profile" (e_profile, see data/msg.c) is already used. And
if the PWD is not $HOME at login time, it seems to me there are
serious problems and the last thing you want is to read some
random and probably dodgy '.profile' from the PWD.
src/cmd/ksh93/sh/init.c: sh_init(): login_files[]:
- Remove redundant/problematic ".profile" entry.
This change was pulled in from:
https://raw.githubusercontent.com/oracle/solaris-userland/master/components/ksh93/patches/185-Bug17714341.patch
No public information about the reasons for this change is
available, but it seems reasonable to trust that the Solaris people
found a legitimate need for it.
src/cmd/ksh93/bltins/cd_pwd.c: b_cd():
- When determining the old PWD before 'cd', do not trust shp->pwd
but get and validate the current PWD using path_pwd().
$KSH_VERSION is initialised as a nameref to ${.sh.version}, but it
was not realiable as it could be overridden from the environment.
Some scripts do version checking so this would allow influencing
their execution.
This fix is inspired by the following Solaris patch:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/200-17435456.patch
but a different approach was needed, because the code has changed
(see 960a1a99).
src/cmd/ksh93/sh/init.c: env_init():
- Refuse to import $KSH_VERSION. Using strncmp(3) might be crude,
but it's effective and I can't figure out another way.
ksh 93u+ has a subshell leak bug where a variable exported in
function in a subshell is also visible in a different subshell.
This is long fixed in 93u+m, but there wasn't a regression test for
this particular bug yet, so this commit adds one.
This fixes the following bug filed with Solaris: "22964338 ksh93
appears to send SIGHUP to unrelated processes on occasion". It is
fixed by applying this patch by Lijo George from the Solaris repo:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/260-22964338.patch
The ksh2020 upstream rejected this, but if it's in production use
in Solaris, Solaris, it's probably good enough for 93u+m. If any
breakage is left, it can be fixed later.
https://github.com/att/ast/pull/1
src/cmd/ksh93/include/jobs.h,
src/cmd/ksh93/sh/fault.c,
src/cmd/ksh93/sh/jobs.c:
- Use a new job_hup() function instead of job_kill() to send SIGHUP
to job processes on termination. The new function checks if a job
is in fact still live before issuing SIGHUP to it.
This pulls a new version of sh_iosafefd() from:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/285-30771135.patch
It was written by Kurtis Rader for ksh2020:
https://github.com/att/ast/issues/198https://github.com/att/ast/pull/199
It is presumably better than the Red Hat version and also comes
with more regression test cases (although it still doesn't fix
modernish BUG_CSUBSTDO, which remains in the TODO file).
This commit does not go along with other peripheral changes from
that patch, i.e. a different name and location of this function.
src/cmd/ksh93/sh/io.c:
- Replace sh_iosafefd() as above.
src/cmd/ksh93/tests/subshell.sh:
- Add and tweak tests from the patch.
This change is pulled from here:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/280-23332860.patch
Info and reproducers:
https://github.com/att/ast/issues/36
In a -c script (like ksh -c 'commands'), the last command
misredirects standard output if an EXIT or ERR trap is set.
This appears to be a side effect of the optimisation that
runs the last command without forking.
This applies a patch by George Lijo that flags these specific
cases and disables the optimisation.
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/bltins/trap.c,
src/cmd/ksh93/sh/init.c,
src/cmd/ksh93/sh/main.c,
src/cmd/ksh93/sh/xec.c:
- Apply patch as above.
src/cmd/ksh93/tests/io.sh:
- Add the reproducers from the bug report as regression tests.
This change is pulled from here:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/275-20855453.patchhttps://github.com/att/ast/issues/30
George Lijo wrote on 17 Feb 2017:
> Here's a reproducible testcase on a Solaris11 host running
> ksh93u+(2012-08-01).
> $ cat a.sh
> #!/bin/sh
>
> AAA="aaa"
> echo 'insert character'
> BBB=`echo ${AAA} | sed "s/aaa/bbb/g"`
> logger "variable BBB = ${BBB}"
>
> $ cat t.sh
> #!/bin/ksh
>
> sleep 10
> /bin/ksh ./a.sh
> exit 0
>
> $
>
> $ ./t.sh
>
> The expected result is:
>
> Apr 9 12:43:34 lab user: [ID 702911 user.notice] variable BBB = bbb
>
> because variable "BBB" is supposed to be set to 'bbb' in a.sh.
>
> But if the parent shell is terminated, the variable is wrongly set.
>
> user@xxxxx$ telnet lab
> ...
> $ ./t.sh & <--- Run t.sh in background.
> [1] 2067
> $ logout <--- CTRL + D to exit while t.sh is running.
> Connection to lab closed by foreign host.
>
> Again, access the system and check the output:
>
> user@xxxxx$ telnet lab
> ...
> $ tail -f /var/adm/messages
> :
> Apr 9 12:47:47 lab user: [ID 702911 user.notice] variable BBB =
> insert character <--- !!!
> Apr 9 12:47:47 lab bbb
> <--- !!!
>
> Thus the variable is wrongly set. (The previous echo string was
> not cleared.)
>
> The issue happens because the EIO error during the logout is not
> handled properly.
src/cmd/ksh93/sh/io.c,
src/lib/libast/include/error.h:
- Amend the ERROR_PIPE() macro to check for EIO as well as EPIPE
and ECONNRESET.
This change is pulled from here:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/240-22461939.patch
Information:
https://github.com/att/ast/issues/6
George Lijo wrote on 14 Mar 2016:
> I observed this issue in a Solaris 11 system on ksh2012-08-01
> [...]. The issue can be reproduced if we add Asian locales to
> ibus (such as Korean). In the ksh93 shell prompt, input some
> Asian character. ksh promptly dumps core [...].
>
> The coredump happens at the following line no 320 in
> src/cmd/ksh93/edit/emacs.c
> if(c!='\t' && c!=ESC && !isdigit(c)).
>
> I referred the vi.c code and added the digit(c) macro, i.e.
> ((c&~STRIP)==0 && isdigit(c)) and replaced the isdigit(c) usage
> with the "digit(c)" macro.
This change is pulled from here:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/211-21547336.patch
src/cmd/ksh93/bltins/cd_pwd.c: b_cd():
- The functions path_pwd() and path_relative() in sh/path.c may
return a pointer to e_dot[] (".") as a fallback if they fail to
determine a path. This is a string in read-only memory
(data/msg.c), so must not be freed. A pointer to that string may
end up in sh.pwd (== shp->pwd), so b_cd() needs a check for that.
This change is pulled from here:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/110-CR7061011.patch
Unfortunately there is no publicly available documentation on why
this change was needed. We just have to assume the Solaris people
knew what they were doing. ksh2020 upstreamed this too (as well as
all the other Solaris patches applied here).
src/cmd/ksh93/sh/nvdisc.c: setdisc():
- If no <event> is known for <np>, return a null pointer instead
of a pointer to the empty string.
The forking fix implemented in 102868f8 and 9d428f8f, which stops
the main shell's hash table from being cleared if PATH is changed
in a subshell, can cause a significant performance penalty for
certain scripts that do something like
( PATH=... command foo )
in a subshell, especially if done repeatedly. This is because the
hash table is cleared (and hence a subshell forks) even for
temporary PATH assignments preceding commands.
It also just plain doesn't work. For instance:
$ hash -r; (ls) >/dev/null; hash
ls=/bin/ls
Simply running an external command in a subshell caches the path in
the hash table that is shared with a main shell. To remedy this, we
would have to fork the subshell before forking any external
command. And that would be an unacceptable performance regression.
Virtual subshells do not need to fork when changing PATH if they
get their own hash tables. This commit adds these. The code for
alias subshell trees (which was removed in ec888867 because they
were broken and unneeded) provided the beginning of a template for
their implementation.
src/cmd/ksh93/sh/subshell.c:
- struct subshell: Add strack pointer to subshell hash table.
- Add sh_subtracktree(): return pointer to subshell hash table.
- sh_subfuntree(): Refactor a bit for legibility.
- sh_subshell(): Add code for cleaning up subshell hash table.
src/cmd/ksh93/sh/name.c:
- nv_putval(): Remove code to fork a subshell upon resetting PATH.
- nv_rehash(): When in a subshell, invalidate a hash table entry
for a subshell by creating the subshell scope if needed, then
giving that entry the NV_NOALIAS attribute to invalidate it.
src/cmd/ksh93/sh/path.c: path_search():
- To set a tracked alias/hash table entry, use sh_subtracktree()
and pass the HASH_NOSCOPE flag to nv_search() so that any new
entries are added to the current subshell table (if any) and do
not influence any parent scopes.
src/cmd/ksh93/bltins/typeset.c: b_alias():
- b_alias(): For hash table entries, use sh_subtracktree() instead
of forking a subshell. Keep forking for normal aliases.
- setall(): To set a tracked alias/hash table entry, pass the
HASH_NOSCOPE flag to nv_search() so that any new entries are
added to the current subshell table (if any) and do not influence
any parent scopes.
src/cmd/ksh93/sh/init.c: put_restricted():
- Update code for clearing the hash table (when changing $PATH) to
use sh_subtracktree().
src/cmd/ksh93/bltins/cd_pwd.c:
- When invalidating path name bindings to relative paths, use the
subshell hash tree if applicable by calling sh_subtracktree().
- rehash(): Call nv_rehash() instead of _nv_unset()ting the hash
table entry; this is needed to work correctly in subshells.
src/cmd/ksh93/tests/leaks.sh:
- Add leak tests for various PATH-related operations in the main
shell and in a virtual subshell.
- Several pre-existing memory leaks are exposed by the new tests
(I've confirmed these in 93u+). The tests are disabled and marked
TODO for now, as these bugs have not yet been fixed.
src/cmd/ksh93/tests/subshell.sh:
- Update.
Resolves: https://github.com/ksh93/ksh/issues/66
The SHOPT_2DMATCH code block in sh_setmatch() modifies the 'ap'
pointer, which is initialised as nv_arrayptr(SH_MATCHNOD). This
caused a (rarely occurring) segfault in the following line near the
end of the function:
ap->nelem -= x;
as this line assumed that 'ap' still had the initial value.
src/cmd/ksh93/sh/init.c: sh_setmatch():
- On init, save ap in ap_save and use ap_save instead of ap where
it should be pointing to SH_MATCHNOD. This also allows removing
two redundant nv_arrayptr(SH_MATCHNOD) calls, slightly increasing
the efficiency of this function.
Scripts that use floating point shell arithmetic confusingly fail
with 'arithmetic syntax error' when running in a locale that uses
',' as the radix point, because '.' is generally assumed by
scripts. The error message is confounding as the problem is not a
syntax error but a locale that is incompatible with the script.
src/cmd/ksh93/sh/arith.c:
- If the locale's radix point is not '.' but a '.' is found in its
place, issue an informative error message that instructs setting
LC_NUMERIC=C.
Resolves: https://github.com/ksh93/ksh/issues/145
@stephane-chazelas writes:
> Per POSIX[*], cd should skip the $CDPATH processing if the first
> component of the directory given to cd is . or ...
>
> Yet, with ksh93u+m 2021-01-03 at least, while that's OK with ..,
> it's not with . with or without the posix option:
>
> $ CDPATH=/ ./ksh -o posix -c 'cd -P ./etc && pwd'
> /etc
> /etc
>
> It seems to be a regression introduced with ksh93u+ as I can't
> reproduce it with ksh93u or any version prior to that. I can also
> reproduce in u+, v- and the ksh2020 from the Ubuntu 20.04
> package.
src/cmd/ksh93/bltins/cd_pwd.c: b_cd():
- Skip $CDPATH processing not only if the path is absolute, but
also if the initial path component is '.' or '..' (in the latter
case the $CDPATH processing was done but appeared to be a no-op).
src/cmd/ksh93/tests/builtins.sh:
- Add regression test.
[*] https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/utilities/cd.html
Fixes: https://github.com/ksh93/ksh/issues/151
As of this commit, ksh 93u+m has a standard semantic version number
<https://semver.org/>, beginning with 1.0.0-alpha. This is added to
the version string in a way that should be compatible with scripts
parsing ${.sh.version} or $(ksh --version). This addition does not
replace the release date and does not affect $((.sh.version)).
For non-release builds, the version string will be:
FORK/VERSION+HASH YYYY-MM-DD
e.g.: 93u+m/1.0.0-alpha+41ef7f76 2021-01-03
For release builds, it will be:
FORK/VERSION YYYY-MM-DD
e.g.: 93u+m/1.0.0 2021-01-03
It is now automatically decided by bin/package whether to build a
release or development build. When building from a directory that
is not a git repository, or if the current git branch name starts
with a number (e.g. '1.0'), the release build is enabled; otherwise
a development build is the default. This is arranged by adding -D
flags to $CCFLAGS as described below. These flags are prepended to
$CCFLAGS, so they can be overridden by adding your own -D or -U
flags via the environment.
In addition, AST vmalloc is disabled for release builds as of this
commit, forcing the use of the OS's standard malloc(3). In 2021,
this is generally more reliable, faster, and more economical with
memory than AST vmalloc. Several memory leaks and crashing bugs are
avoided, e.g.: <https://github.com/ksh93/ksh/issues/95>.
For development builds, vmalloc stays enabled (along with its known
bugs) because this allows the use of the vmstat builtin, making it
much more efficient to test for memory leaks. For more info, see
the regression test script: src/cmd/ksh93/tests/leaks.sh
bin/package, src/cmd/INIT/package.sh:
- Add flags for build type. In $CCFLAGS, define _AST_ksh_release if
we're not on any git branch or on a git branch whose name starts
with a number. Otherwise, define _AST_git_commit as the first 8
characters of the current git commit hash.
src/lib/libast/features/vmalloc:
- If _AST_ksh_release is defined, disable vmalloc and force use of
the operating system's malloc. Discussion:
https://github.com/ksh93/ksh/issues/95
src/cmd/ksh93/include/version.h:
- Define new format for version string, adding a semantic version
number as well as (for non-release builds) the git commit hash.
src/cmd/ksh93/sh/init.c: e_version[]:
- Add a 'v' to the ${.sh.version} feature string if ksh was
compiled with vmalloc enabled. This allows scripts, such as
regression tests, to detect this.
src/cmd/ksh93/data/builtins.c: sh_optksh[]:
- Add a copyright line crediting the contributors to ksh 93u+m.
Resolves: https://github.com/ksh93/ksh/issues/95
When starting ksh +s, it gets stuck in an infinite loop continually
trying to parse its own binary as a shell script and rejecting it:
$ arch/linux.i386-64/bin/ksh +s
arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error]
arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error]
arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error]
arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error]
arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error]
[...]
$ echo 'echo "this is stdin"' | arch/linux.i386-64/bin/ksh +s
arch/linux.i386-64/bin/ksh: arch/linux.i386-64/bin/ksh: cannot execute [Exec format error]
(no loop, but still ksh trying to parse itself)
src/cmd/ksh93/sh/init.c: sh_init():
- When forcing on the '-s' option upon finding no command
arguments, also update sh.offoptions, a.k.a. shp->offoptions.
This avoids the inconsistent state causing this problem.
In main.c, there is:
if(sh_isoption(SH_SFLAG))
fdin = 0;
else
(code to open $0 as a file)
This was entering the else block because sh_isoption(SH_SFLAG)
was returning 0, and $0 is set to the ksh binary as it is
supposed to when no other script is provided. When I looked for
why sh_isoption was returning 0, I found main.c's
for(i=0; i<elementsof(shp->offoptions.v); i++)
shp->options.v[i] &= ~shp->offoptions.v[i];
Before this loop, shp->offoptions tracks which options were
explicitly disabled by the user on the command line. The effect
of this loop is to make "explicitly disabled" take precedence
over "implicitly enabled". My patch removes the registration of
the +s option.
Fixes: https://github.com/ksh93/ksh/issues/150
Co-authored-by: Martijn Dekker <martijn@inlv.org>
A recent change in Github's Ubuntu test runners exposed a problem
in the way the all_paths test function replicates 'whence -a'
functionality, causing spurious regression test failures.
The problem occurs when e.g. /bin is a symlink to /usr/bin, but
both are in $PATH. The all_paths function treats them as separate
but 'whence -a' detects a duplicate and will not output /usr/bin/*.
I have not succeeded in making all_paths match 'whence -a' exactly.
src/cmd/ksh93/tests/path.sh: function all_paths:
- Using the -ef test expression operator, remove entries from $PATH
that point to the same directory as another entry in $PATH (e.g.
when /bin is a symlink to /usr/bin but both are in $PATH).
Avoiding such dupes works around the problem.
src/cmd/ksh93/sh/name.c:
- The Empty macro (a constant pointer to the empty string) is already
defined in include/defs.h, so does not need to be repeated here.
GMT and UTC have identical time but are used in different contexts.
When the system time zone is set to GMT (e.g. in the UK at winter
time), the 'printf %T' test could fail as it correctly uses GMT
whereas the test expects UTC.
src/cmd/ksh93/tests/builtins.sh:
- Fix possible false negative in 'printf %T\\n now' test by
replacing GMT with UTC in both 'date' output and 'printf %T'
output, instead of only the former.
Issuing typeset floating point numerics having a precision of 0
failed as the precision/size was being overwritten with the string
length of the value, e.g. 'typeset -F0 x=5.67' would result in
'typeset -F 4 x=5.6700' as len('5.67') is 4.
src/cmd/ksh93/include/nval.h:
- Created a symbolic name of NV_FLTSIZEZERO to respresent a float
having a precision/size of 0. NV_FLTSIZEZERO needs to be a
negative value.
src/cmd/ksh93/bltins/typeset.c:
- In b_typeset(), added code to set tdata.argnum to NV_FLTSIZEZERO
for E, F, X options.
- In setall(), adjusted code to allow for tp->argnum to be negative.
src/cmd/ksh93/sh/name.c: nv_newattr():
- Adjusted option value only change code to handle NV_FLTSIZEZERO as
well as changed to directly setting np->nvsize instead of using
nv_setsize(np,size) as nv_setsize might contain conflicting and/or
redundant code.
- Added missing conditional check of '!(newatts&NV_INTEGER)' to
constrain the size==0 code block to justified strings as
NV_LJUST, NV_RJUST, or NV_ZFILL are only valid for strings if
NV_INTEGER is not set. This code block was mistakenly setting
the precision/size value to the length of the value of an
assignment for floats whereas it should only be performing
auto assignment length for justified strings.
'typeset -xu' and 'typeset -xl' would export the variable but fail
to change case in the value as the check between old and new
attributes did not provide the necesssary insight for lower or
upper case transcoding due to the lower or upper case attribute
being set within typeset.c prior to calling name.c nv_newattr
function.
Previous rhbz#1188377 patch added a conditional check for size==-1
which in effect caused the nv_newattr export code block return
optimization to never be executed as one cannot set any attributes
using the readonly builtin. By altering the size==-1 check to !trans
the export only optimization can run.
Also, the rhbz#1188377 patch altered new_attr function by setting
the new size to oldsize if run by the readonly builtin. The result
of setting size==oldsize allowed the succeeding if statement to
run more frequently and if size was a non-zero value resulted in
nv_setsize resetting the value to what it already was. Investigation
yielded that size was always 0 coming from the readonly builtin.
src/cmd/ksh93/bltins/typeset.c:
- Remove the setting of tdata.argnum to -1 as it is not needed due to
existing name.c nv_newattr() logic.
src/cmd/ksh93/sh/name.c: nv_newattr():
- Corrected the export only check optimization by using !trans instead
of using size==-1.
- Removed previous condition check to set size=oldsize if coming from
the readonly builtin. nv_newattr already had existing logic to
prevent changing the size via nv_setsize as size is always 0 when
coming from readonly builtin.
src/cmd/ksh93/tests/builtins.sh:
- Remove redundant extra bincat=$(whence -p cat).
- Move whence -v/-a tests to path.sh.
- Fix 'whence -q' test so errors are counted outside of a subshell.
src/cmd/ksh93/tests/path.sh:
- Add all_paths function that is basically a reimplementation of
'whence -a -p' in shell. Useful for testing 'whence'.
- Move whence -v/-a tests to here, changing them to use all_paths
where needed. Also fix the 'whence -a' function autoloading
regression test to do the same. This fixes the tests for systems
(such as Slackware) where commands such as 'ls' or 'chmod' have
more than one path in even the default $PATH.
The '-o nolog' option (which prevented function definitions from being
recorded in the history file) was removed a long time ago, leaving
only a stub for backwards compatibility to stop 'set' from erroring
out if the option is set. But some other vestiges remained.
src/cmd/ksh93/sh/path.c,
src/cmd/ksh93/sh/subshell.c,
src/cmd/ksh93/sh/xec.c:
- Remove a few pointless 'sh_onstate(SH_NOLOG)' statements. As of
93u+ or earlier, this is never checked for anywhere.
src/cmd/ksh93/sh.1:
- They forgot to remove the 'nolog' option documentation here.
Specify that it's obsolete and has no effect.
src/cmd/ksh93/data/builtins.c: sh_set[]:
- Be more concise.
Virtual/non-forking subshells that change the present working
directory (PWD) with 'cd' suffer from a serious race condition. The
PWD is changed within the same process. This means it may not be
possible to change back to the original PWD when exiting the
subshell, as some other process may destroy the PWD or modify its
permissions in the meantime. ksh did not handle this error
condition at all, so, after exiting a subshell that invoked 'cd',
it could silently end up running the script's following command(s)
in the wrong directory. Which might be 'rm -rf *'. So, ouch.
The proper and obvious fix is never to allow a virtual subshell to
change the PWD, as it can never be guaranteed you can return to a
previous directory. If the PWD is changed in a child process, there
is no need to restore it in the parent process, and this whole
problem is avoided. So subshells really should always fork on
encountering a 'cd' command.
But forking is slow. It is not uncommon for scripts to 'cd' in a
subshell that is run repeatedly in a loop.
There is also the issue of custom builtins that can be added to ksh
via shared libraries. In the standard shell language, 'cd' is the
only command that changes the PWD, so we could just make that
command fork the subshell it is run from. But there's no telling
what a custom builtin might do.
So this commit implements a compromise that will not affect
performance unless there is the pathological condition of a PWD
that has been rendered inaccessible in some way:
1. When entering a virtual subshell, if the parent shell's PWD
proves inaccessible upon saving it, the subshell will now fork into
a separate process, avoiding the unrestorable PWD problem.
2. If some attack renders the parent shell's PWD unrestorable
*after* ksh enters a virtual subshell, ksh will now error out when
exiting it. There is nothing else left to do then. Continuing would
mean running arbitrary commands in the wrong PWD.
src/cmd/ksh93/sh/subshell.c:
- Put all the code/variables only needed for fchdir() behind '#if
_lib_fchdir'. This makes it clearer what's what.
(I don't know if there is still any system out there without
fchdir(3); I haven't found any. The chdir(3) fallback version may
be removed later as there is no way to make it remotely secure.)
- Fix the attempt to use the O_PATH mode for open(2) as a fallback
for nonexistent O_SEARCH on Linux. Define _GNU_SOURCE on Linux,
or <fcntl.h> (which is included indirectly) won't define O_PATH.
- Fix use of O_SEARCH. The code was simply wrong, repeating an
open(".",O_RDONLY) instead. Since a nonexistent O_SEARCH is now
redefined as either O_PATH or O_RDONLY, we can simply
open(".",O_SEARCH) and be done with it.
- Fix fatal error handling. Introduce fatal error condition for
failure to fchdir(3) back to the parent's PWD; rename 'duped' to
'fatalerror' and use it for error numbers; save and restore errno
on fatal error so the message will report the cause. (We must
call errormsg() near the end of sh_subshell() to avoid crashes.)
- If open(".",O_SEARCH) was not able get a file descriptor to our
PWD on entry, then call sh_subfork() immediately before running
the subshell commands. (Forking earlier causes a crash.)
- When restoring the PWD, if fchdir(3) fails, do *not* fall back to
chdir(3). We already know the PWD is inaccessible, so if chdir(3)
"succeeds" then, it's very likely to be a substitute injected by
an attacker.
src/cmd/ksh93/bltins/cd_pwd.c:
- If we don't have fchdir(3), then sh_subshell() must fall back to
chdir(2) to restore the PWD. That is highly vulnerable, as a
well-timed rename would allow an attacker to usurp the PWD. We
can't do anything about that if some custom builtin changes the
PWD, but we can at least make 'cd' always fork a subshell, which
slows down ksh but removes the need for the parent shell ever to
restore the PWD. (There is certainly no popular system where this
is relevant and there might not be any such current system.)
This commit adds no regression test because a portable regression
test is not really doable. Different kernels, external /bin/pwd
utilities, etc. all have quite different behaviour under the
pathological condition of an inaccessible PWD, so both the
before-fix and the after-fix behaviour differs. See link below.
Resolves: https://github.com/ksh93/ksh/issues/141
Thanks to Stéphane Chazelas for the bug report.
src/cmd/ksh93/features/options:
- Fix unportable SHELLMAGIC test:
1. /bin/echo does not work on all systems, but /usr/bin/env is a
de-facto standard path (even NixOS gave in and has it).
2. Do not try to execute a temp file in /tmp as it might be
mounted noexec. Use our own $EXECROOT instead.
- rm unnecessary 'exec 9<&-' (it was never opened globally).
- Remove broken SHOPT_UCB test. It had a syntax error, but the
result is not used anywhere.
src/cmd/ksh93/bltins/typeset.c: b_typeset():
- For integer bases change argnum check to default values that
are < 2 or > 64 to 10 instead of allowing invalid base values
that ksh cannot process.
src/cmd/ksh93/bltins/typeset.c: setall():
- Remove argnum check for integer base of 1 as base cannot be 1.
- Relocate strlen(name) to inside of conditional check for
np->nvfun as this code does not need to run all.
- Remove no-op oldname code
src/cmd/ksh93/tests/attributes.sh:
- Add typeset -i base bounds checks to default base 10
src/cmd/ksh93/tests/builtins.sh:
- Fix a test so it doesn't fail if 'whence -a' finds multiple paths
for 'ls'.
src/cmd/ksh93/tests/coprocess.sh
- Update known failure comment with current information.
src/cmd/ksh93/sh/args.c: sh_argprocsub():
- Save and restore state more efficiently by just saving and
restoring all the state bits in one go using the
sh_{get,set}state() macros, which are defined in defs.h as:
#define sh_getstate() (sh.st.states)
#define sh_setstate(x) (sh.st.states = (x))
(and there is yet more evidence that it doesn't matter whether
we use a 'shp->' pointer or 'sh.' direct access).
src/cmd/ksh93/sh/main.c: exfile():
- Remove a no-op 'sh_offstate(SH_INTERACTIVE);'. It was in the
'else' clause of 'if(sh_isstate(SH_INTERACTIVE))' so if we get
there, it is known that this flag is already off.
- To properly disable job control, we also have to save and restore
the job.jobcontrol variable.
src/cmd/ksh93/sh/xec.c: sh_exec():
- Remove some no-op flaggery from this highly performance-sensitive
point in the code. Given the immediately preceding:
volatile int was_errexit = sh_isstate(SH_ERREXIT);
volatile int was_monitor = sh_isstate(SH_MONITOR);
the following:
sh_offstate(SH_ERREXIT);
if(was_errexit&flags)
sh_onstate(SH_ERREXIT);
can be reformulated as:
if(!(flags & sh_state(SH_ERREXIT)))
sh_offstate(SH_ERREXIT);
(IOW, if it was already on, don't turn it off and then on again)
...and the following:
if(was_monitor&flags)
sh_onstate(SH_MONITOR);
can be removed; it's a no-op because it wasn't preceded by an
sh_offstate() and if 'was_monitor' is true, this option is known
to be on. (I considered they may have forgotten an 'sh_offstate'
there like in the SH_ERREXIT case, but adding that causes several
regressions in a shtests run.)
src/cmd/ksh93/include/defs.h:
- Remove comment that is evidently long outdated; there is not (or
no longer) a Shscoped_t type defined anywhere, nor are these
struct fields replicated in any other type definition.
- Add comment to clarify what the 'states' int in 'struct
sh_scoped' is for.
By definition, subshells are never interactive, so they should
disable behaviour associated with interactive shells even if the
main shell is interactive.
Most visibly, running a background job from a subshell like
( some_command & )
now no longer prints a job ID that you cannot use in the main shell.
This behaviour change matches pdksh/mksh, bash, zsh, dash, et al.
Prior discussion:
https://www.mail-archive.com/austin-group-l@opengroup.org/msg06456.html
(plus the preceding thread)
src/cmd/ksh93/sh/subshell.c: sh_subshell():
- Before running the command(s) in the subshell using sh_exec(),
turn off the SH_INTERACTIVE shell state flag. (No need to add
code to restore it as this function already saves and restores
the entire shell state.)
src/cmd/ksh93/bltins/misc.c: b_bg():
- If there is no job control when using 'bg', 'fg' or 'disown',
always print the "no job control" error message and not only if
the shell is in the interactive state. This is also what
pdksh/mksh, bash and zsh do.
Mildly interesting: apparently there was once an idea to implement
shared-state command substitutions as a shell option like 'set -o
subshare'. They were implemented using a new ${ syntax; } instead,
but there is a vestigial SH_SUBSHARE option ID in shell.h plus a
check for it in subshell.c that would cause backtick-style command
substitutions (comsub==1) to share their state. That option isn't
defined in data/options.c so it's impossible for a user to set it.
src/cmd/ksh93/include/shell.h,
src/cmd/ksh93/sh/subshell.c:
- Remove SH_SUBSHELL option vestiges.
src/cmd/ksh93/include/defs.h:
- Correct my comment on 'comsub' flag; I was wrong about what the
values meant. 2 is for a shared-state comsub. (re: 4ce486a7)
Autoloading a function caused the calling script's $LINENO to be
off by the number of lines in the function definition file. In
addition, while running autoloaded functions, errors/warnings were
reported with wrong line numbers.
src/cmd/ksh93/sh/path.c:
- Save $LINENO (shp->inlineno) before autoloading a function, reset
it to 1 so that the correct line number offset is remembered for
the function definition, and restore it after.
src/cmd/ksh93/tests/variables.sh:
- Add regression test for $LINENO, directly and in error messages,
within and outside a non-autoloaded and an autoloaded function.
Fixes: https://github.com/ksh93/ksh/issues/116
ksh 93u+ introduced a regression in the combination of the
'set -o pipefail' and 'set -e'/'set -o errexit' options:
$ ksh93 -o errexit -o pipefail -c \
'(exit 3) | true; echo "still here despite $? status"'
still here despite 3 status
The bug is in how the the huge sh_exec() function in xec.c handles
the 'echeck' flag. Near the end of sh_exec(), this flag triggers a
sh_chktrap() call to check whether to trigger any traps, including
the ERR trap -- and that same function also handles the errexit
option, which is basically the same as 'trap "exit" ERR'.
We can learn more easily how sh_exec() works by inserting debug
warnings in all its 'switch(type&COMMSK)' cases, like:
case TCOM:
errormsg(SH_DICT,ERROR_warn(0),"[DEBUG] TCOM");
... and same for all the others. With that done, the output
of a very simple dummy pipeline looks as follows:
$ arch/*/bin/ksh -c 'true | true | true'
arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TFIL
arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TFORK
arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TFORK
arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TSETIO
arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TCOM
arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TCOM
arch/darwin.i386-64/bin/ksh: warning: [DEBUG] TCOM
So, it looks like sh_exec() handles this pipeline as follows:
TFIL
|_____TFORK
| |_____TCOM
|_____TFORK
| |_____TCOM
|_____TSETIO
|_____TCOM
Each time a pipeline like command1 | command2 | ... is executed,
sh_exec() is invoked with type TFIL; this then recursively invokes
sh_exec() to handle the individual elements. The last element of
the pipe triggers a sh_exec() run with type TSETIO; since it is run
in the current shell environment, it is effectively treated as a
command with an input redirection. All the previous elements are of
type TFORK instead, because they are executed asynchronously in
separate, forked subshell processes. Finally, the TFORK or TSETIO
code then recursively calls sh_exec() again with type TCOM to
actually execute the commands.
When reading the code, we find that the 'echeck' flag is set as
part of the TSETIO code. This makes sense of why only an error in
the last element of the pipe triggers the errexit/ERR trap action.
So that's the bug: the flag is set in the wrong place.
This can be fixed by setting that flag in the TFIL handling code
instead, as this is what calls everything else and collects all the
exit statuses. So the sh_chktrap() call is now executed after
handling the entire pipeline, at the TFIL recursion level.
This also allows getting rid of the special-casing in the buggy
TSETIO version. The SH_ERREXIT state is restored at the end of each
sh_exec() call, so since we're now doing this at a lower recursion
level, it will already have been restored.
src/cmd/ksh93/sh/xec.c: sh_exec():
- Fix the bug as per the above.
src/cmd/ksh93/tests/options.sh:
- Add tests for errexit and ERR trap combined with pipefail.
src/cmd/ksh93/tests/basic.sh:
- Tweak a couple of tests that reported a trap wasn't triggered
even if it was actually triggered more than once.
Fixes: https://github.com/ksh93/ksh/issues/121
Thanks to Stéphane Chazelas for the bug report.
'typeset -xu' and 'typeset -xl' would export the variable but fail
to change case in the value under certain conditions.
Original patch:
https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-xufix.patch
This applies the patch essentially without change and adds a
regression test based on the reproducer provided in the RH bug.
Unfortunately there is no description of how the patch works and
it's a little obscure to me. As far as I can figure out, the cause
of the problem was that nv_newattr() erroneously processed a
nonexistent size option-argument such as what can be given to
options like typeset -F, e.g. typeset -F3 for 3 digits after the
dot. A nonexistent size argument is represented by the value of -1.
Discussion/analysis: https://bugzilla.redhat.com/1506344
iousepipe() might write out of bounds, causing a crash, if
subpipe[2] is set to a value >= sh.gd.lim.open_max.
src/cmd/ksh93/sh/xec.c: iousepipe():
- Validate the FD using sh_iovalidfd() before the write.
Prior discussion:
https://bugzilla.redhat.com/1454804
On 2017-05-23 13:33:25 UTC, Paulo Andrade wrote:
> In previous ksh versions, when exiting the scope of a ksh
> (not posix) function, it would restore the trap table of
> the "calling context" and if the reason the function exited
> was a signal, it would call sh_fault() passing as argument
> the signal value.
> Newer ksh checks it, but calls kill(getpid(), signal_number)
> after restoring the trap table, but only calls for SIGINT and
> SIGQUIT.
[...]
> The old way appears to have been more appropriate, but there
> must be a reason to only pass SIGINT and SIGQUIT as it is an
> explicit patch.
The last paragraph is where I differ. This would not be the first
example of outright breakage that appeared to be added deliberately
and that 93u+m has fixed or removed, see e.g. 8477d2ce ('printf %H'
had code that deleted all multibyte characters), cefe087d, or
781f0a39. Sometimes it seems the developers added a little
experiment and then forgot all about it, so it became a misfeature.
In this instance, the correct pre-2012 ksh behaviour is still
explicitly documented in (k)sh.1: "A trap condition that is not
caught or ignored by the function causes the function to terminate
and the condition to be passed on to the caller". Meaning, if there
is no function-local trap, the signal defaults to the parent scope.
There is no language that limits this to SIGINT and SIGQUIT only.
It also makes no sense at all to do so -- signals such as SIGPIPE,
SIGTERM, or SIGSEGV need to be caught by default and to do
otherwise results in misbehaviour by default.
src/cmd/ksh93/sh/xec.c: sh_funscope():
- When resending a signal after restoring the global traps state,
remove the spurious check that limits this to SIGINT and SIGQUIT.
- Replace it with a check for nsig!=0, as that means there were
parent trap states to restore. Otherwise 'kill' may be called
with an invalid signal argument, causing a crash on macOS.
src/cmd/ksh93/tests/signal.sh:
- Update a test to check that a function-local SIGTERM trap is
triggered correctly when signalled from another process.
- Complete the tests for 3aee10d7; this bug needed fixing before
we could test that previous fix in a ksh function scope.
- Add a test for triggering global traps from ksh functions,
testing multiple POSIX-standard signals.
The ksh-20120801-trapcom.patch patch contains an off-by-one error,
which was also imported into 93u+m. When saving signals:
ceb77b136f/src/cmd/ksh93/sh/subshell.c (L572-L592)
572 if((nsig=shp->st.trapmax)>0 || shp->st.trapcom[0])
573 {
574 ++nsig;
575 savsig = malloc(nsig * sizeof(char*));
576 /*
577 * the data is, usually, modified in code like:
578 * tmp = buf[i]; buf[i] = strdup(tmp); free(tmp);
579 * so shp->st.trapcom needs a "deep copy" to properly save/restore pointers.
580 */
581 for (isig = 0; isig < nsig; ++isig)
582 {
583 if(shp->st.trapcom[isig] == Empty)
584 savsig[isig] = Empty;
585 else if(shp->st.trapcom[isig])
586 savsig[isig] = strdup(shp->st.trapcom[isig]);
587 else
588 savsig[isig] = NULL;
589 }
On line 574, the number of signals 'nsig' is increased by one. That
increase is permanent, so the 'for' loop on line 581 tries to save
one signal state too many.
The increase was a holdout from the ksh93 code from before the
patch. After the patch, it is not required; it is fine to malloc as
many records as there are trapcom elements to save. So it should
simply be removed. xec.c has the same code to save trap states for
ksh functions, and the same applies.
src/cmd/ksh93/sh/subshell.c: sh_subshell():
- Don't increase nsig.
src/cmd/ksh93/sh/xec.c: sh_funscope():
- Same.
src/cmd/ksh93/tests/signal.sh:
- Add test.
Original patch:
https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20140801-diskfull.patch
Prior discussion:
https://www.mail-archive.com/ast-users@lists.research.att.com/msg01037.htmlhttps://www.mail-archive.com/ast-users@lists.research.att.com/msg01038.htmlhttps://www.mail-archive.com/ast-users@lists.research.att.com/msg01042.htmlhttps://bugzilla.redhat.com/1212992
On Fri, 08 May 2015 14:37:45 -0700, Paulo Andrade wrote:
> I have a user with a ksh crashing problem, and that has
> some "Write error: No space left on device" messages
> in /var/log/messages.
>
> After some debugging, and creating a chroot on a file
> disk image, and a test user, and slowly filling the
> "on file" filesystem, e.g.
>
> dd if=/dev/zero of=/mnt/tmp/zerosN bs=1M count=1024
> dd if=/dev/zero of=/mnt/tmp/zerosN bs=1K count=2
>
> until leaving just around 12K, I managed to reproduce the
> problem, and be able to debug it with valgrind and vgdb;
> debugging on these conditions is tricky, as cannot tell
> valgrind to spawn gdb, because then gdb itself would fail
> to start.
>
> So, after following the code enough, I learned that at places
> it handles SH_JMPEXIT, there was almost non existing
> handling of SH_JMPERREXIT.
>
> ksh would evently cause a crash due to the struct
> subshell allocated on stack, in sh/subshell.c:sh_subshell
> kept set to the global subshell_data, after it siglongjmp
> back the stack due to, not fully handling the out of disk
> space errors. It would print a few messages, everytime
> a pipe was created, e.g.:
>
> /etc/profile: line 28: write to 3 failed [No space left on device]
>
> until eventually crashing due to corrupted memory; e.g. the
> references to stack data from sh_subsell in the global
> subshell_data. One strange thing to me in coredump analysis
> was that subshell_data prev field was pointing to itself when
> it eventually crashed, what later was understood and expected...
>
> The attached patch handles SH_JMPERREXIT in the code
> paths SH_JMPEXIT is handled, and the failed login, on
> full disk, ends in a pause() call:
>
> ---terminal 1---
> $ valgrind -q --leak-check=full --free-fill=0x5a --vgdb=full
> --vgdb-error=0 /bin/ksh -l
> ==17730== (action at startup) vgdb me ...
> ==17730==
> ==17730== TO DEBUG THIS PROCESS USING GDB: start GDB like this
> ==17730== /path/to/gdb /bin/ksh
> ==17730== and then give GDB the following command
> ==17730== target remote | /usr/lib64/valgrind/../../bin/vgdb --pid=17730
> ==17730== --pid is optional if only one valgrind process is running
> ==17730==
> ==17730== Syscall param mount(type) points to unaddressable byte(s)
> ==17730== at 0x563377A: mount (in /usr/lib64/libc-2.17.so)
> ==17730== by 0x493E58: fs3d_mount (fs3d.c:115)
> ==17730== by 0x493C8B: fs3d (fs3d.c:57)
> ==17730== by 0x423E41: sh_init (init.c:1302)
> ==17730== by 0x405CD3: sh_main (main.c:141)
> ==17730== by 0x405B84: main (pmain.c:45)
> ==17730== Address 0x0 is not stack'd, malloc'd or (recently) free'd
> ==17730==
> ==17730== (action on error) vgdb me ...
> ==17730== Continuing ...
> /etc/profile: line 28: write to 3 failed [No space left on device]
> ---8<---
>
> ---terminal 2---
> (gdb) c
> Continuing.
> ^C
> Program received signal SIGTRAP, Trace/breakpoint trap.
> 0x00000000055fa470 in __pause_nocancel () from /lib64/libc.so.6
> (gdb) bt
> #0 0x00000000055fa470 in __pause_nocancel () from /lib64/libc.so.6
> #1 0x000000000041e73d in sh_done (ptr=0x793360 <sh>, sig=255) at
> /home/pcpa/rhel/ksh/ksh-20120801/src/cmd/ksh93/sh/fault.c:665
> #2 0x0000000000407407 in exfile (shp=0x4542, iop=0xff, fno=0) at
> /home/pcpa/rhel/ksh/ksh-20120801/src/cmd/ksh93/sh/main.c:604
> #3 0x0000000000405c43 in sh_source (shp=0x793360 <sh>, iop=0x0,
> file=0x524804 <e_sysprofile> "/etc/profile")
> at /home/pcpa/rhel/ksh/ksh-20120801/src/cmd/ksh93/sh/main.c:109
> #4 0x00000000004060e4 in sh_main (ac=2, av=0xfff000498, userinit=0x0)
> at /home/pcpa/rhel/ksh/ksh-20120801/src/cmd/ksh93/sh/main.c:202
> #5 0x0000000000405b85 in main (argc=2, argv=0xfff000498) at
> /home/pcpa/rhel/ksh/ksh-20120801/src/cmd/ksh93/sh/pmain.c:45
> (gdb)
> ---8<---
Prior discussion:
https://bugzilla.redhat.com/1217236
Summary: doing
( some_simple_command & )
i.e., launching a background job from within a subshell, on a ksh
interactive login shell, caused misbehaviour (command not run).
For me on 93u+m, the misbehaviour was different -- an outright
crash in the job handling code following SIGCHLD, backtracing to:
0 ksh job_unpost + 49 (jobs.c:1655)
1 ksh job_reap + 1632 (jobs.c:468)
2 libsystem_platform.dylib _sigtramp + 29
3 ??? 0 + 4355528544
4 ksh ed_getchar + 102 (edit.c:1048)
5 ksh ed_emacsread + 659 (emacs.c:280)
[...]
Original patch:
https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-nohupfork.patch
Lines 1874-1886 in sh_exec() optimise the highly specific case of
'( simple_command & )' by avoiding a sh_subshell() call that sets
up a virtual subshell before forking:
https://github.com/ksh93/ksh/blob/bd283959/src/cmd/ksh93/sh/xec.c#L1874-L1886
1874 else if(((type=t->par.partre->tre.tretyp)&FAMP) && ((type&COMMSK)==TFORK))
1875 {
1876 pid_t pid;
1877 sfsync(NIL(Sfio_t*));
1878 while((pid=fork())< 0)
1879 _sh_fork(shp,pid,0,0);
1880 if(pid==0)
1881 {
1882 sh_exec(t->par.partre,flags);
1883 shp->st.trapcom[0]=0;
1884 sh_done(shp,0);
1885 }
1886 }
1887 else
1888 sh_subshell(shp,t->par.partre,flags,0);
The original patch inserts the following before the sh_done call on
line 1884:
sh_offoption(SH_INTERACTIVE);
sh_done() checks for SH_INTERACTIVE and only runs job handling code
if that option is on.
Also, I had missed the need for an update of shgd->current_pid
here. Since 843b546c replaced lots of getpid() calls by reading
that variable, this could cause ksh to SIGCHLD the wrong process.
But even after adding the shgd->current_pid update to the RH patch,
I was still able to make it crash.
src/cmd/ksh93/sh/xec.c: sh_exec(): case TPAR:
- Disable this optimisation outright for interactive or job
control-enabled shells. I really don't trust it at all, but there
have been no problem reports for scripts without job control, so
keep it until such reports surface.
- Update shgd->current_pid so the child doesn't end up signalling
the wrong process (re: 843b546c).
___________________________________________________________________
P.S.:
It was noted in the Red Hat discussion that ( ... & ) does a double
fork, i.e., a virtual/non-forked subshell always forks before
forking the background job. This extra fork is done by the
following two lines in under 'case TFORK:' in sh_exec() in xec.c:
https://github.com/ksh93/ksh/blob/bd283959/src/cmd/ksh93/sh/xec.c#L1534-L1535
1534 if((type&(FAMP|TFORK))==(FAMP|TFORK))
1535 sh_subfork();
This is executed if we're in a virtual/non-forked subshell, i.e.
shp->subshell is true (note it is false for forked subshells). So
it forks the virtual subshell (the first fork) before running the
background job (the second fork). A background job is recognised by
'type' having both the FAMP (AMP == ampersand == &) and TFORK bits
set and no others.
So the obvious way to remove the double fork is to comment out
these two lines. Indeed, testing that confirms it gone and ksh
appears to work fine at first glance. Does it really? Nearly!
There are just four regression failures in a 'bin/shtests -p' run:
options.sh[394]: & job delayed by --pipefail, expected '1212 or 1221', got '1122'
signal.sh[280]: subshell ignoring signal does not send signal to parent (expected 'SIGUSR1', got 'done')
signal.sh[289]: subshell catching signal does not send signal to parent (expected 'SIGUSR1', got 'done')
subshell.sh[467]: sleep& in subshell hanging
So, those two lines cannot be removed now and we have to keep the
double fork. But removing them doesn't appear to break very much,
so it may be possible to add some tests so we only do an extra fork
when really necessary. That is a project for another day.
For one Red Hat customer, the following reproducer consistently
crashed, tough I was not able to reproduce it and neither was RH.
However, the crash analysis is sound (see below).
function dlog
{
fc -ln -0
}
trap dlog DEBUG
>/tmp/blah
Original patch: https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20140801-arraylen.patch
The Red Hat bug thread is closed to the public as it also contains
some correspondence with their customer. But it has an excellent
crash analysis from Thomas Gardner which I'm including here for the
record (the line numbers are for their ksh at the time, not 93u+m).
===begin analysis===
> The creation of an empty file instead of a command that executes
> anything causes the coredump.
[...]
> Here is my analysis on the core that was provided by the customer:
>
> (gdb) bt
> #0 sh_fmtq (string=0x1 <Address 0x1 out of bounds>)
> at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/string.c:340
> #1 0x0000000000457e96 in out_string (cp=<value optimized out>, c=32,
> quoted=<value optimized out>, iop=<value optimized out>)
> at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/xec.c:444
> #2 0x000000000045804c in sh_debug (shp=0x76d180, trap=0x7f2f13a821e0 "dlog",
> name=<value optimized out>, subscript=<value optimized out>,
> argv=0x76e070, flags=<value optimized out>)
> at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/xec.c:548
> #3 0x000000000045a867 in sh_exec (t=0x7f2f13aafad0, flags=4)
> at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/xec.c:1265
> [...need go no further...]
>
> In frame 2, we can see it cycling through your classic
> (char **)argv array like:
>
> 543 while(cp = *argv++)
> 544 {
> 545 if((flags&ARG_EXP) && argv[1]==0)
> 546 out_pattern(iop, cp,' ');
> 547 else
> 548 out_string(iop, cp,' ',n?0: (flags&(ARG_RAW|ARG_NOGLOB))||*argv);
> 549 }
> 550 if(flags&ARG_ASSIGN)
> 551 sfputc(iop,')');
> 552 else if(iop==stkstd)
>
> (we seg-fault after going down the out_string function in line
> 548 up there). The string pointer that points to = 0x1 up in
> frame #0 (sh_fmtq) traces back to the "cp" variable in line 548
> up there. The "argv" variable being referenced up there just gets
> passed in as the fifth argument to this function.
>
> In frame #3 (sh_exec, line 1265), the line that makes the call
> that takes us to frame 2 is:
>
> 1265 int n = sh_debug(shp,trap,(char*)0,(char*)0, com, ARG_R AW);
>
> so "com" (the fifth argument) is what's going wrong as it
> descends down through these calls. Looking at where it comes
> from, well, it's assigned here:
>
> 1241 if(argn==0)
> 1242 {
> 1243 /* fake 'true' built-in */
> 1244 np = SYSTRUE;
> 1245 *argv = nv_name(np);
> 1246 com = argv;
> 1247 }
>
> because as we can see:
>
> (gdb) f 3
> #3 0x000000000045a867 in sh_exec (t=0x7f2f13aafad0, flags=4)
> at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/xec.c:1265
> 1265 int n = sh_debug(shp,trap,(char*)0,(char*)0, com, ARG_RAW);
> (gdb) p argn
> $2 = 0
> (gdb)
>
> argn is == 0 here. The tip-off here is that nv_name clearly
> returns a simple pointer to an array of characters, not an array
> of pointers to arrays of characters as is evidenced by the fact
> that the assignment is "*argv = nv_name(np);" not "argv =
> nv_name(np);". Looking at the function nv_name proves that it
> does indeed return a single pointer to an array of characters,
> not a pointer to an array of pointers to arrays of characters.
> Now, com is defined as a 'char **':
>
> 1002 char *cp=0, **com=0, *comn;
>
> (as it is expected to be in the calls that follow) also, that
> argv is also defined as the effective equivalent a 'char **':
>
> 1237 static char *argv[1];
>
> Yup, argv is actually an array of pointers (char ** equivalent),
> but that array is restricted to having exactly one element.
> Recalling the assignment in the previously quoted line:
>
> 1245 *argv = nv_name(np);
>
> we see that the one and only element in that argv array is
> getting assigned a pointer to an array of characters here.
> Nothing necessarily wrong with that, but remember the loop we
> looked at earlier in frame #2 (sh_debug). It went like:
>
> 543 while(cp = *argv++)
> 544 {
> 545 if((flags&ARG_EXP) && argv[1]==0)
> 546 out_pattern(iop, cp,' ');
> 547 else
> 548 out_string(iop, cp,' ',n?0: (flags&(ARG_RAW|ARG_NOGLOB))||*argv);
> 549 }
>
> which is clearly expecting argv in this context (com in frame 3,
> which really points to that static local single element array
> that is also pointed to by argv in frame 2) to be an array of
> pointers of indefinite size, each element being a pointer, but
> whose last element will be a null pointer. Well, in frame 3 it is
> clearly an array with only a single element, and that one element
> is NOT pointing to null. Watch this:
>
> (gdb) f 3
> #3 0x000000000045a867 in sh_exec (t=0x7f2f13aafad0, flags=4)
> at /usr/src/debug/ksh-20120801/src/cmd/ksh93/sh/xec.c:1265
> 1265 int n = sh_debug(shp,trap,(char*)0,(char*)0, com, ARG_RAW);
> (gdb) p com
> $8 = (char **) 0x76e060
> (gdb) p &argv
> $9 = (char *(*)[1]) 0x76e060
> (gdb) p com[0]
> $11 = 0x5009c6 "true"
> (gdb) p com[1]
> $10 = 0x1 <Address 0x1 out of bounds>
> (gdb) p argv[0]
> $12 = 0x5009c6 "true"
> (gdb) p argv[1]
> $13 = 0x1 <Address 0x1 out of bounds>
> (gdb)
>
> So, as expected, com and &argv point to the same place, the first
> element points to the constant string "true", but since the array
> is defined as having only one element, when you refer to a second
> element in that array, you get well, whatever random crap happens
> to be in that memory location. When we try to reproduce this
> problem, apparently we're getting 0 there (or we're not quite
> following this same code path, which is also possible), but the
> customer happens to have a "1" in that memory location.
===end analysis===
src/cmd/ksh93/sh/xec.c: sh_exec():
- When processing TCOM (simple command) with an empty/null command,
increase the size of the static dummy argv[1] array to argv[2],
ensuring a terminating NULL element so that 'while(cp = *argv++)'
loops don't crash. (Note that static objects are automatically
initialised to zero in C.)
src/cmd/ksh93/tests/io.sh:
- Adapt the reproducer, testing a null-command redirection 1000x.
Of course I was wrong to say the bug had nothing to do with
functions; traps in ksh functions are local, are handled the same
way as traps that are local to virtual subshells, and had the same
crashing bug. So this adds a test for that as well.
Contrary to the RH bug report, this is yet another bug with
virtual/non-forked subshells and has nothing to do with functions.
If a signal is ignored (empty trap) in the main shell while any
trap (empty or not) is set on the same signal in a subshell, a
crash eventually occurred upon restoring state when leaving the
subshell.
Original patch:
https://src.fedoraproject.org/rpms/ksh/blob/642af4d6/f/ksh-20120801-trapcom.patch
Prior discussion:
https://bugzilla.redhat.com/1117404
Paulo Andrade wrote there:
> The problem is that the sh_subshell function was saving pointers
> that could change, and when restoring, bad things would happen.
[...]
> The only comment I added:
> /* contents of shp->st.trapcom may change */
> may be a bit misleading, the "bad" save/restore already knows it,
> probably I should have added a better description telling that the
> data is, usually, modified in code like:
>
> tmp = buf[i]; buf[i] = strdup(tmp); free(tmp);
>
> so the shp->st.trapcom needs a "deep copy", as done in the
> patch, to properly save/restore pointers.
src/cmd/ksh93/sh/subshell.c,
src/cmd/ksh93/sh/xec.c:
- sh_subshell(), sh_funscope(): Make *savsig/*savstak into a
**savsig array. Use strdup(3) to save the data and get known
pointers that will not change. Free these upon restore.
- Change the comment from the patch as Paulo wished he had done.
src/cmd/ksh93/tests/subshell.sh:
- Test 2500 times. This should trigger the crash most of the time.
