If I haven't missed anything, this should make the non-interactive
aspects of job control in scripts work as expected, except for the
"<command unknown>" issue in the output of 'bg', 'fg' and 'jobs'
(which is not such a high priority as those commands are really
designed for interactive use).
Plus, I believe I now finally understand what these three are for:
* The job.jobcontrol variable is set to nonzero by job_init() in
jobs.c if, and only if, the shell is interactive *and* managed to
get control of the terminal. Therefore, any changing of terminal
settings (tcsetpgrp(3), tty_set()) should only be done if
job.jobcontrol is nonzero. This commit changes several checks for
sh_isoption(SH_INTERACTIVE) to checks for job.jobcontrol for
better consistency with this.
* The state flag, sh_isstate(SH_MONITOR), determines whether the
bits of job control that are relevant for both scripts and
interactive shells are active, which is mostly making sure that a
background job gets its own process group (setpgid(3)).
* The shell option, sh_isoption(SH_MONITOR), is just that. When the
user turns it on or off, the state flag is synched with it. It
should usually not be directly checked for, as the state may be
temporarily turned off without turning off the option.
Prior discussion:
https://www.mail-archive.com/austin-group-l@opengroup.org/msg06456.html
src/cmd/ksh93/bltins/typeset.c, src/cmd/ksh93/sh/args.c:
- Move synching the SH_MONITOR state flag with the SH_MONITOR
shell option from b_set() (the 'set' builtin) to sh_applyopts()
which is indirectly called from b_set() and is also used when
parsing the shell invocation command line. This ensures -m is
properly enabled in both scenarios.
src/cmd/ksh93/sh/jobs.c:
- job_init(): Do not refuse to initialise job control on
non-interactive shells. Instead, skip everything that should only
be done on interactive shells (i.e., everything to do with the
terminal). This function is now even more of a mess than it was
before, so refactoring may be desirabe at some point.
- job_close(), job_set(), job_reset(), job_wait(): Do not reset the
terminal process group (tcsetpgrp()) if job.jobcontrol isn't on.
src/cmd/ksh93/sh/xec.c:
- sh_exec(): TFORK: For SIGINT handling, check the SH_MONITOR
state flag, not the shell option.
- sh_exec(): TFORK: Do not turn off the SH_MONITOR state flag in
forked children. The non-interactive part of job control should
stay active. Instead, turn off the SH_INTERACTIVE state flag so
we don't get interactive shell behaviour (i.e. job control noise
on the terminal) in forked subshells.
- _sh_fork(), sh_ntfork(): Do not reset the terminal process group
(tcsetpgrp()) if job.jobcontrol isn't on. Do not turn off the
SH_MONITOR state flag in forked children.
src/cmd/ksh93/sh/subshell.c: sh_subfork():
- Do not turn off the monitor option and state in forked subshells.
The non-interactive part of job control should stay active.
src/cmd/ksh93/bltins/misc.c: b_bg():
- Check isstate(SH_MONITOR) instead of sh_isoption(SH_MONITOR) &&
job.jobcontrol before throwing a 'no job control' error.
This fixes a minor bug: fg, bg and disown could quietly fail.
src/cmd/ksh93/tests/jobs.sh:
- Add tests for 'fg' with job control IDs (%%, %1) in scripts.
- Add test checking that a background job launched from a subsell
with job control enabled correctly becomes the leader of its own
process group.
Makes progress on: https://github.com/ksh93/ksh/issues/119
Another longstanding whopper of a bug in basic ksh93 functionality:
run a ${ shared-state; } command substitution twice and job control
promptly loses track of all your running jobs. New jobs are tracked
again until you run another two shared-state command substitutions.
This is in at least 93t+, 93u-, 93u+, 93v- and ksh2020.
$ sleep 300 &
[1] 56883
$ jobs # OK
[1] + Running sleep 300 &
$ v=${ echo hi1; }
$ jobs # OK
[1] + Running sleep 300 &
$ v=${ echo hi2; }
$ jobs # Nothing!
$ fg
ksh: fg: no such job
src/cmd/ksh93/sh/subshell.c: sh_subshell():
- The current environment number shp->curenv (a.k.a. sh.curenv) was
not being restored if the virtual subshell we're leaving is of
the shared-state command substitution variety as it was wrongly
considered to be part of the environment that didn't need
restoring. This caused it to be out of sync with shp->jobenv
(a.k.a. sh.jobenv) which did get restored from savedcurenv.
Restore both from savedcurenv at the same time for any subshell.
(How these numbers are used exactly remains to be discovered.)
src/cmd/ksh93/tests/jobs.sh:
- Added, with a test for this bug to start it off. There is no
other test script where job control fits, and a lot more related
fixes are anticipated: https://github.com/ksh93/ksh/issues/119
src/cmd/ksh93/bltins/enum.c:
- enum_type[]: Fix typos; minor edit for style.
- enum_type[], enuminfo(): Make the list of supported values
comma-separated, instead of using a comma at the start of each.
src/cmd/ksh93/sh/nvtype.c:
- sh_opttype[]: Fix typos.
It was easier than expected to fix this one. The many regression
test failures caused by disabling it were all due to one bug:
'typeset -p' output broke when building without this option.
src/cmd/ksh93/sh/nvtree.c: nv_attribute():
- In this function to print the attributes of a name-value pair,
move four lines of code out of #if SHOPT_FIXEDARRAY...#endif that
were inadvertently moved into the #if block in ksh93 2012-05-18.
See the changes to nvtree.c in this multishell repo commit:
https://github.com/multishell/ksh93/commit/aabab56a
src/cmd/ksh93/data/builtins.c:
- Update/rewrite 'typeset -a' documentation.
- Make it adapt to SHOPT_FIXEDARRAY.
- Fix a few typos.
src/cmd/ksh93/tests/arrays2.sh:
- Only one regression test needs a SHOPT_FIXEDARRAY check.
.github/workflows/ci.yml:
- Disable SHOPT_FIXEDARRAY when regression-testing without SHOPTs.
- Enable xtrace, add ':' commands for traced comments. This should
make the CI runner output logs a little more readable.
Many compile-time options were broken so that they could not be
turned off without causing compile errors and/or regression test
failures. This commit now allows the following to be disabled:
SHOPT_2DMATCH # two dimensional ${.sh.match} for ${var//pat/str}
SHOPT_BGX # one SIGCHLD trap per completed job
SHOPT_BRACEPAT # C-shell {...,...} expansions (, required)
SHOPT_ESH # emacs/gmacs edit mode
SHOPT_HISTEXPAND # csh-style history file expansions
SHOPT_MULTIBYTE # multibyte character handling
SHOPT_NAMESPACE # allow namespaces
SHOPT_STATS # add .sh.stats variable
SHOPT_VSH # vi edit mode
The following still break ksh when disabled:
SHOPT_FIXEDARRAY # fixed dimension indexed array
SHOPT_RAWONLY # make viraw the only vi mode
SHOPT_TYPEDEF # enable typeset type definitions
Compiling without SHOPT_RAWONLY just gives four regression test
failures in pty.sh, but turning off SHOPT_FIXEDARRAY and
SHOPT_TYPEDEF causes compilation to fail. I've managed to tweak the
code to make it compile without those two options, but then dozens
of regression test failures occur, often in things nothing directly
to do with those options. It looks like the separation between the
code for these options and the rest was never properly maintained.
Making it possible to disable SHOPT_FIXEDARRAY and SHOPT_TYPEDEF
may involve major refactoring and testing and may not be worth it.
This commit has far too many tweaks to list. Notables fixes are:
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/data/options.c:
- Do not compile in the shell options and documentation for
disabled features (braceexpand, emacs/gmacs, vi/viraw), so the
shell is not left with no-op options and inaccurate self-doc.
src/cmd/ksh93/data/lexstates.c:
- Comment the state tables to associte them with their IDs.
- In the ST_MACRO table (sh_lexstate9[]), do not make the S_BRACE
state for position 123 (ASCII for '{') conditional upon
SHOPT_BRACEPAT (brace expansion), otherwise disabling this causes
glob patterns of the form {3}(x) (matching 3 x'es) to stop
working as well -- and that is ksh globbing, not brace expansion.
src/cmd/ksh93/edit/edit.c: ed_read():
- Fixed a bug: SIGWINCH was not handled by the gmacs edit mode.
src/cmd/ksh93/sh/name.c: nv_putval():
- The -L/-R left/right adjustment options to typeset do not count
zero-width characters. This is the behaviour with SHOPT_MULTIBYTE
enabled, regardless of locale. Of course, what a zero-width
character is depends on the locale, but control characters are
always considered zero-width. So, to avoid a regression, add some
fallback code for non-SHOPT_MULTIBYTE builds that skips ASCII
control characters (as per iscntrl(3)) so they are still
considered to have zero width.
src/cmd/ksh93/tests/shtests:
- Export the SHOPT_* macros from SHOPT.sh to the tests as
environment variables, so the tests can check for them and decide
whether or how to run tests based on the compile-time options
that the tested binary was presumably compiled with.
- Do not run the C.UTF-8 tests if SHOPT_MULTIBYTE is not enabled.
src/cmd/ksh93/tests/*.sh:
- Add a bunch of checks for SHOPT_* env vars. Since most should
have a value 0 (off) or 1 (on), the form ((SHOPT_FOO)) is a
convenient way to use them as arithmetic booleans.
.github/workflows/ci.yml:
- Make GitHub do more testing: run two locale tests (Dutch and
Japanese UTF-8 locales), then disable all the SHOPTs that we can
currently disable, recompile ksh, and run the tests again.
src/lib/libast/misc/optget.c:
- Add screen* (which includes tmux) and dtterm* (CDE terminal) to
the glob pattern deciding whether to use ANSI boldface sequences.
- Don't bother parsing the env var if stderr is not on a terminal.
src/cmd/ksh93/sh.1:
- Extend self-documentation documentation; document how optget(3)
uses the ERROR_OPTIONS env var to control boldface output.
- Tweaks and minor edits.
The >;word and <>;word redirection operators cannot be used with
the 'exec' builtin, but the 'redirect' builtin (which used to be
an alias of 'command exec') permitted them. However, they do not
have the documented effect of the added ';'. So this commit blocks
those operators for 'redirect' as they are blocked for 'exec'.
It also tweaks redirect's error message if a non-redirection
argument is encountered.
src/cmd/ksh93/sh/parse.c: simple():
- Set the lexp->inexec flag for SYSREDIR (redirect) as well as
SYSEXEC (exec). This flag is checked for in sh_lex() (lex.c) to
throw a syntax error if one of these two operators is used.
src/cmd/ksh93/sh.1, src/cmd/ksh93/data/builtins.c:
- Documentation tweaks.
src/cmd/ksh93/sh/xec.c, src/cmd/ksh93/bltins/misc.c:
- When 'redirect' gives an 'incorrect syntax' (e_badsyntax) error
message, include the first word that was found not to be a valid
redirection. This is simply the first argument, as redirections
are removed from the arguments list.
src/cmd/ksh93/tests/io.sh:
- Update test to reflect new error message format.
Now that the Make Abstract Machine files are maintained manually
and not generated automatically, unused variables are an annoying
distraction -- and there are many.
But the language/format is very simple and very parseable using
shell, awk, etc. -- so this was easy to automate. All variables are
declared with 'setv' and they are used if an expansion of the form
${varname} exists (the braces are mandatory in Mamfiles).
bin/Mamfile_rm_unused_vars:
- Added for reference and future use.
src/*/*/Mamfile:
- Remove all unused 'setv' variable declarations.
Permanent redirections of that form broke in subshells when used
with the 'redirect' command, because I had overlooked one instance
where the new 'redirect' builtin needs to match the behaviour of
the 'exec' builtin.
src/cmd/ksh93/tests/io.sh: sh_exec():
- Do not restore file descriptors in (virtual) subshells for
'redirect' just as this isn't done for 'exec'.
src/cmd/ksh93/tests/io.sh:
- Add regression test for this bug.
- Complete the test for f9427909 which I committed prematurely.
Fixes: https://github.com/ksh93/ksh/issues/167
This is some nonsense: redirections that store a file descriptor
greater than 9 in a variable, like {var}<&2 and the like, stopped
working if brace expansion was turned off. '{var}' is not a brace
expansion as it doesn't contain ',' or '..'; something like 'echo
{var}' is always output unexpanded. And redirections and brace
expansion are two completely unrelated things. It wasn't documented
that these redirections require the -B/braceexpand option, either.
src/cmd/ksh93/sh/lex.c: sh_lex():
- Remove incorrect check for braceexpand option before processing
redirections of this form.
src/cmd/ksh93/COMPATIBILITY:
- Insert a brief item mentioning this.
src/cmd/ksh93/sh.1:
- Correction: these redirections do not yield a file descriptor >
10, but > 9, a.k.a. >= 10.
- Add a brief example showing how these redirections can be used.
src/cmd/ksh93/tests/io.sh:
- Add a quick regression test.
src/cmd/ksh93/sh/args.c: sh_argprocsub():
- Fix compiler warnings with SHOPT_DEVFD on by including "io.h".
- Without SHOPT_DEVFD, the FIFO code didn't consider that libast's
pathtemp(3) may also fail and return null. Add a check for this.
It was trivial to crash ksh by making an autoloaded function
definition file autoload itself, causing a stack overflow due to
infinite recursion. This commit adds loop detection that stops a
function that is being autoloaded from autoloading itself either
directly or indirectly, without removing the ability of autoloaded
function definition files to autoload other functions.
src/cmd/ksh93/sh/path.c: funload():
- Detect loops by checking if the path of a function to be
autoloaded was already added to a new internal static tree,
and if not, adding it while the function is being loaded.
src/cmd/ksh93/tests/path.sh:
- Add regression test.
- Tweak a couple of others to be freeze- and crash-proof.
NEWS:
- Add this fix + a forgotten entry for the previous fix (6f3b23e6).
Fixes: https://github.com/ksh93/ksh/issues/136
Reproducer from @Saikiran-m:
| ~# sh -c `perl -e 'print "a"x100000'`
| genunix: NOTICE: core_log: sh[1221] core dumped: /var/cores/core.sh.0.1602153496
| Memory fault(coredump)
The crash was in trying to decide whether the name was suitable for
autoloading as a function on $FPATH. This calls strmatch() to check
the name against a regex for valid function name. But the libast
regex code is not designed optimally and uses too much recursion,
limiting the length of the strings it's able to cope with.
src/cmd/ksh93/sh/path.c: path_search():
- Before calling strmatch(), check that the name is shorter than
256 bytes. The maximum length of file names on Linux and macOS is
255 bytes, so an autoload function can't have a name longer than
that anyway.
src/cmd/ksh93/tests/path.sh:
- Add test for this bug.
- Tweak 'command -x' test to not leave a hanging process on Ctrl+C.
Fixes: https://github.com/ksh93/ksh/issues/144
Well, that commit was based on a silly oversight: of course it's
necessary to pass ${KSH_RELFLAGS} to the feature tests too as they
use this flag to determine whether to enable or disable vmalloc.
On further analysis I think the annoying warnings can be solved in
a different way. Quotes (single or double) in 'exec -' commands
don't seem to be special to mamake at all; it looks like they are
passed on to the shell as is. So Mamfile variables are expanded and
the expansions backslash-escaped the same way regardless of quotes.
Which means we can make the shell remove the unwanted level of
backslashes by using double instead of single quotes.
src/*/*/Mamfile:
- On iffe commands, restore ${KSH_RELFLAGS}, using double quotes to
group the compiler command as one argument to iffe.
This time it was failing on a 64-bit Debian Linux system with very
few and short environment variables. Sigh.
src/cmd/ksh93/sh/path.c:
- Combine the strategy from 63979488 with that of 8f5235a5.
That fix turned out to be insufficient as NixOS has huge
environment variable lists because (due to each software package
being installed in its own directory tree) it has to keep dozens
of directories in variables like XDG_CONFIG_DIRS and others.
The 'command -x' regression test was failing on NixOS.
src/cmd/ksh93/sh/path.c:
- Different strategy. Leave twice the size of the existing
environment free.
Hopefully this will deal with ksh crashing in macOS Terminal.app
once and for all. Trigger: press Command-F to open the find bar,
then press Esc to close it, then press Esc again. Result: crash
somewhere random in the job control code.
Turns out macOS Terminal.app apparently (and wrongly) sends <Esc>
followed by <Ctrl+L> to the terminal, which ksh takes as a sequence
for clearing the screen. The related crash ultimately traced back
to the code for that in emacs.c. The other crash was in the code
for double-ESC file name completion.
This commit also fixes a non-robust invocation of the 'tput'
command by using the direct path found in $(getconf PATH).
src/cmd/ksh93/features/cmds:
- Remove unused tests for the presence of commands
(newgrp,test,id,wc,cut,logname,pfexec).
- Replace 'cmd tput' test by 'pth tput' which will find its path
in $(getconf PATH) and store that path as the macro value.
- Add two tests to determine if 'tput' supports terminfo and/or
termcap codes. (FreeBSD still requires old termcap codes.)
src/cmd/ksh93/edit/emacs.c: escape():
- Fix a crash in the code for double-ESC completion. Check if the
cursor is on a non-zero position; this caused a bus error
(invalid address access) in the subsequent ed_expand call.
- For <Esc><Ctrl+L> (clear screen), fix the strange crash in macOS
Terminal by not using sh_trap() to invoke "tput clear", which
causes ksh itself to invoke that command. ksh apparently doesn't
cope with doing this while SIGWINCH (window size change signal)
is sent by Terminal. The fix is to just use the C standard
system(3) function to invoke tput. This invokes tput via /bin/sh,
but what the hey. (Note that ksh also ran any function or alias
called 'tput' instead of the real command, and that is now also
fixed.)
- Use the new _pth_tput test result to invoke tput with the
hardcoded default system path, increasing robustness.
src/cmd/ksh93/edit/edit.c: ed_setup():
- Use the new _pth_tput test result to invoke tput with the
hardcoded default system path, increasing robustness.
- When getting the escape code for "cursor up", use the new
_tput_terminfo and _tput_termcap test results to determine which
kind of command code to send. This fixes it on FreeBSD.
Something similar was previously done in 07cc71b8 from a Debian
patch, and eventually reverted; it redefined the ast atomic
functions asoincint() and asodecint() to be gcc-specific. This
imports the upstream version from the ksh 93v- beta instead.
This commit is based on an OpenSUSE patch:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-joblock.dif
src/cmd/ksh93/include/jobs.h:
- Replace job locking mechanism with the 93v- version which uses
the atomic libast functions asoincint(), asogetint() and
asodecint(). See: src/lib/libast/man/aso.3
src/cmd/ksh93/sh/jobs.c: job_subsave():
- Revert gcc optimiser bug workaround from c258a04f.
It should now be unnecessary.
I got one intermittent regression test failure due to 'argument
list too long' on a Debian x86_64 system.
src/cmd/ksh93/sh/path.c: path_xargs():
- Leave extra argument space for systems that need extra bytes:
1KiB per extra byte, with a minimum of 2KiB (the old value).
This fixes the function that sets ${.sh.match}. Patch from OpenSUSE:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-limit-name-len.dif
src/cmd/ksh93/sh/init.c: sh_setmatch():
- Fix node size calculation, possibly preventing data corruption.
src/cmd/ksh93/include/ulimit.h: Limit_t:
- Defining the 'name' struct member as 'char name[16]' makes
no sense as the name is being initialised statically in
data/limits.c; just make it a 'char *name' pointer.
This fixes the following regressions marked TODO in attributes.sh:
$ typeset -L 13 bar; readonly bar; typeset -p bar
typeset -r -L 0 foo # exp.: typeset -r -L 13 foo
$ typeset -R 13 bar; readonly bar; typeset -p bar
typeset -r -R 0 bar # exp.: typeset -r -R 13 bar
$ typeset -Z 13 baz; readonly baz; typeset -p baz
typeset -r -Z 0 -R 0 baz # exp.: typeset -r Z 13 -R 13 baz
I've discovered that these were briefly fixed between fdb9781e (Red
Hat patch for typeset -xu/-xl) and 95fe07d8 (reversal of patch,
different -xu/-xl fix, but reintroduced these regressions).
src/cmd/ksh93/sh/name.c: nv_newattr():
- Replace check from 95fe07d8 with a new one that combines its
approach with that of fdb9781e: do not change size (and hence
return early) if NV_RDONLY and/or NV_EXPORT are the only
attributes that are changing.
src/cmd/ksh93/tests/attributes.sh:
- Enable the TODO regression tests.
This commit corrects how shortint was being applied to various
possible typeset variables in error. The short integer option
modifier 'typeset -s' should only be able to be applied if the
the variable is also an integer. Several issues were resolved
with this fix:
- 'typeset -s': created a short integer having an invalid base
of zero. 'typeset -s foo' created 'typeset -s -i 0 foo=0' and
now will result in an empty string.
- 'typeset -sL': previously resulted in a segmentation fault.
The following are the various incorrect 'typeset' instances
that have been fixed:
$ 'export foo; typeset -s foo; readonly foo; typeset -p foo'
(before) typeset -x -r -s -i 0 foo=0
( after) typeset -x -r foo
$ 'typeset -sL foo=1*2; typeset -p foo'
(before) Segmentation fault (core dumped)
( after) typeset -L 3 foo='1*2'
$ 'typeset -sR foo=1*2; typeset -p foo'
(before) typeset -s -i foo=2
( after) typeset -R 3 foo='1*2'
$ 'typeset -sZ foo=1*2; typeset -p foo'
(before) typeset -F 0 foo=2
( after) typeset -Z 3 -R 3 foo='1*2'
src/cmd/ksh93/bltins/typeset.c: b_typeset():
- Add conditional check within the 's' option to only
apply NV_SHORT as well as remove any NV_LONG flag
if NV_INTEGER flag was set.
- Relocate shortint conditional logic to the 'i' option.
src/cmd/ksh93/tests/attributes.sh:
- Adjust regression tests for '-s' and add '-si' check.
This fixes part of https://github.com/ksh93/ksh/issues/87:
Scalar arrays (-a) and associative arrays (-A) of a type created by
'enum' did not consistently block values not specified by the enum
type, yielding corrupted results.
An expansion of type "${array[@]}" yielded random numbers instead
of values for associative arrays of a type created by 'enum'.
This does not yet fix another problem: ${array[@]} does not yield
all values for associative enum arrays.
src/cmd/ksh93/bltins/enum.c: put_enum():
- Always throw an error if the value is not in the list of possible
values for an enum type. Remove incorrect check for the NV_NOFREE
flag. Whatever that was meant to accomplish, I've no idea.
src/cmd/ksh93/sh/array.c: nv_arraysettype():
- Instead of sh_eval()ing a shell assignment, use nv_putval()
directly. Also use the stack (see src/lib/libast/man/stk.3)
instead of malloc to save the value; it's faster and will be
auto-freed at some point. This shortens the function and makes it
faster by not entering into a whole new shell context -- which
also fixes another problem: the error message from put_enum()
didn't cause the shell to exit for indexed enum arrays.
src/cmd/ksh93/sh/name.c: nv_setlist():
- Apply a patch from David Korn that correctly sets the data type
for associative arrays, fixing the ${array[@]} expansion yielding
random numbers. Thanks to @JohnoKing for the pointer.
https://github.com/ksh93/ksh/issues/87#issuecomment-662613887https://www.mail-archive.com/ast-developers@lists.research.att.com/msg00697.html
src/cmd/ksh93/tests/enum.sh:
- Add tests checking that invalid values are correctly blocked for
indexed and associative arrays of an enum type.
Makes progress on: https://github.com/ksh93/ksh/issues/87
Turns out the assumption I was operating on, that Linux and macOS
align arguments on 32 or 64 bit boundaries, is incorrect -- they
just need some extra bytes per argument. So we can use a bit more
of the arguments buffer on these systems than I thought.
src/cmd/ksh93/features/externs:
- Change the feature test to simply detect the # of extra bytes per
argument needed. On *BSD and commercial Unices, ARG_EXTRA_BYTES
shows as zero; on Linux and macOS (64-bit), this yields 8. On
Linux (32-bit), this yields 4.
src/cmd/ksh93/sh/path.c: path_xargs():
- Do not try to calculate alignment, just add ARG_EXTRA_BYTES to
each argument.
- Also add this when substracting the length of environment
variables and leading and trailing static command arguments.
src/cmd/ksh93/tests/path.sh:
- Test command -v/-V with -x.
- Add a robust regression test for command -x.
src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/sh.1:
- Tweak docs. Glob patterns also expand to multiple words.
iffe feature test that add a -D_LARGEFILE64_SOURCE compiler flag to
detect the presence of 64-bit types like off64_t are very
incorrect; they always find the type even if the rest of the source
is not compiled with that flag, causing an inconsistent compilation
environment. This was the cause of mysterious failures to compile
some feature tests on Linux i386 -- it tried to use an off64_t type
that was wrongly detected.
A flag like -D_LARGEFILE64_SOURCE needs to be added to the compiler
flags consistently so it is used for compiling all files and tests.
src/lib/libast/features/dirent,
src/lib/libast/features/fs,
src/lib/libast/features/lib,
src/lib/libast/features/mmap,
src/cmd/ksh93/features/rlimits:
- Remove the -D_LARGEFILE64_SOURCE flag from all the tests that
used it.
- Fix some preprocessor directives for compiling without
_LARGEFILE64_SOURCE. We cannot rely on the result of the _lib_*64
tests because those functions are still found in glibc even if
_LARGEFILE64_SOURCE is not defined; we have to check for the
existence of the type definitions before using them.
src/cmd/INIT/cc.linux.i386,
src/cmd/INIT/cc.linux.i386-icc:
- Add/update compiler wrappers to hardcode -D_LARGEFILE64_SOURCE
in the flags for the default compiler. If it is overriden with
$CC, then it needs to be added manually if desired.
This test depends on the correctness of the locale data provided
by the OS, and some installations are broken. Failures of this test
most likely do not represent a bug in ksh or libast.
This takes another small step towards disentangling the build
system from the old AT&T environment. The USAGE_LICENSE macros with
author and copyright information, which was formerly generated
dynamically for each file from a database, are eliminated and the
copyright/author information is instead inserted into the AST
getopt usage strings directly.
Repetitive license/copyright information is also removed from the
getopt strings in the builtin commands (src/lib/libcmd/*.c and
src/cmd/ksh93/data/builtins.c). There's no need to include 55
identical license/copyright strings in the ksh binary; one (in the
main ksh getopt string, shown by ksh --man) ought to be enough!
This makes the ksh binary about 10k smaller.
It does mean that something like 'enum --author', 'typeset
--license' or 'shift --copyright' will now not show those notices
for those builtins, but I doubt anyone will care.
This commit fixes 'command -x' to adapt to OS limitations with
regards to data alignment in the arguments list. A feature test is
added that detects if the OS aligns the argument on 32-bit or
64-bit boundaries or not at all, allowing 'command -x' to avoid
E2BIG errors while maximising efficiency.
Also, as of now, 'command -x' is a way to bypass built-ins and
run/query an external command. Built-ins do not limit the length of
their argument list, so '-x' never made sense to use for them. And
because '-x' hangs on Linux and macOS on every ksh93 release
version to date (see acf84e96), few use it, so there is little
reason not to make this change.
Finally, this fixes a longstanding bug that caused the minimum exit
status of 'command -x' to be 1 if a command with many arguments was
divided into several command invocations. This is done by replacing
broken flaggery with a new SH_XARG state flag bit.
src/cmd/ksh93/features/externs:
- Add new C feature test detecting byte alignment in args list.
The test writes a #define ARG_ALIGN_BYTES with the amount of
bytes the OS aligns arguments to, or zero for no alignment.
src/cmd/ksh93/include/defs.h:
- Add new SH_XARG state bit indicating 'command -x' is active.
src/cmd/ksh93/sh/path.c: path_xargs():
- Leave extra 2k in the args buffer instead of 1k, just to be sure;
some commands add large environment variables these days.
- Fix a bug in subtracting the length of existing arguments and
environment variables. 'size -= strlen(cp)-1;' subtracts one less
than the size of cp, which makes no sense; what is necessary is
to substract the length plus one to account for the terminating
zero byte, i.e.: 'size -= strlen(cp)+1'.
- Use the ARG_ALIGN_BYTES feature test result to match the OS's
data alignment requirements.
- path_spawn(): E2BIG: Change to checking SH_XARG state bit.
src/cmd/ksh93/bltins/whence.c: b_command():
- Allow combining -x with -p, -v and -V with the expected results
by setting P_FLAG to act like 'whence -p'. E.g., as of now,
command -xv printf
is equivalent to
whence -p printf
but note that 'whence' has no equivalent of 'command -pvx printf'
which searches $(getconf PATH) for a command.
- When -x will run a command, now set the new SH_XARG state flag.
src/cmd/ksh93/sh/xec.c: sh_exec():
- Change to using the new SH_XARG state bit.
- Skip the check for built-ins if SH_XARG is active, so that
'command -x' now always runs an external command.
src/lib/libcmd/date.c, src/lib/libcmd/uname.c:
- These path-bound builtins sometimes need to run the external
system command by the same name, but they did that by hardcoding
an unportable direct path. Now that 'command -x' runs an external
command, change this to using 'command -px' to guarantee using
the known-good external system utility in the default PATH.
- In date.c, fix the format string passed to 'command -px date'
when setting the date; it was only compatible with BSD systems.
Use the POSIX variant on non-BSD systems.
Three OpenSUSE patches from:
https://build.opensuse.org/package/show/shells/ksh
As usual, the relevant bug is not currently public:
https://bugzilla.opensuse.org/show_bug.cgi?id=844071
src/cmd/ksh93/sh/xec.c: sh_debug()/sh_exec():
- Fix stk restoration. [bnc#844071]
src/lib/libast/misc/stk.c:
- Fix stk aliasing code. [bnc#844071]
(ksh93-stkalias.dif)
- Make a unknown location fatal in stkset() so that we get a core
dump right away instead of later in an unrelated part of code.
(ksh93-stkset-abort.dif)
src/lib/libast/man/stk.3,
src/lib/libast/man/stak.3:
- Update manual with new stkset() behaviour. (93u+m addition)
(Note that stak is implemented as macros that translate to stk)
This backports most of the Cdt (container data types) mechanism
from the ksh 93v- beta, based on ground work done by OpenSUSE:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-dttree-crash.dif
plus adaptations to match ksh 93u+m and an updated manual page
(src/lib/libast/man/cdt.3) added directly from the 93v- sources.
| Thu Dec 20 12:48:02 UTC 2012 - werner@suse.de
|
| - Add ksh93-dttree-crash.dif - Allow empty strings in (dt)trees
| (bnc#795324)
|
| Fri Oct 25 14:07:57 UTC 2013 - werner@suse.de
|
| - Rework patch ksh93-dttree-crash.dif
As usual, precious little information is available because the
OpenSUSE bug report is currently closed to the public:
https://bugzilla.opensuse.org/show_bug.cgi?id=795324
However, a cursory inspection suggests that this code contains
improvements to do with concurrent processing and related
robustness. The new cdt.3 manual page adds a lot about that.
This has been in production use on OpenSUSE for a long time,
so hopefully this will make ksh a little more stable again.
Only one way to find out: let's commit and test this...
BTW, to get a nice manual, use groff and ghostscript's ps2pdf:
$ groff -tman src/lib/libast/man/cdt.3 | ps2pdf - cdt.3.pdf
This fixes annoying warnings from feature tests that show up when
building with IFFEFLAGS=-d1 (show compiler output from iffe), e.g.:
| In file included from <built-in>:367:
| <command line>:3:26: warning: missing terminating '"' character [-Winvalid-pp-token]
| #define _AST_git_commit \"a5c53a59\"
| ^
| 1 warning generated.
This means the double quotes were incorrectly escaped, which is
probably a bug in mamake -- but they're done correctly for the .c
files that actually need these flags. I may or may not trace the
mamake bug sometime.
src/*/*/Mamfile:
- Remove ${KSH_SHOPTFLAGS} en ${KSH_RELFLAGS} from the iffe
invocations; they are not relevant for feature tests, only when
actually compiling .c files (the $CC commands).
This makes ksh build at least on AIX 7.1 on RISC (PowerPC).
There are 4 regression test failures:
leaks.sh[159]: memory leak on PATH reset before PATH search
(leaked approx 220 KiB after 16384 iterations)
pty.sh[351]: POSIX sh 104(C): line 364: expected
"^done\r?\n$", got EOF
signal.sh[280]: subshell ignoring signal does not send
signal to parent (expected 'SIGUSR1', got 'done')
signal.sh[282]: parent does not wait for child to complete
before handling signal
src/cmd/INIT/iffe.sh:
- Unset LIBPATH on AIX. The features/pty output{ ... }end will fail
to link to libiconv otherwise, causing a build failure. See:
https://www.ibm.com/support/pages/member-libiconvso2-not-found-archive
src/cmd/builtin/pty.c:
- CMIN is not defined on AIX, so set it to 1 if it's not defined.
src/cmd/ksh93/README:
- Update list of tested OSs.
It was unreasonably hard to debug problems with iffe tests that
fail to compile where they should (particularly output{ ... }end
blocks that write esserntial headers).
In e72543a9 the problem was already somewhat mitigated by making
some of the failing output{ ... }end blocks emit #error directives
so that invalid/incomplete headers would cause an error at a
sensible point, and not a much harder to track error later.
This commit further mitigates the problem by making the Mamfiles
respect an IFFEFLAGS environmenet variable that is prefixed to
every iffe command's arguments. The typical use would be to export
IFFEFLAGS=-d1 to enable debug level 1: show compiler output for all
iffe tests. This now makes it reasonably feasible to detect
problems in the feature tests themselves.
src/**/Mamfile:
- Import IFFEFLAGS environment variable using setv.
- Prefix ${IFFEFLAGS} to every iffe command.
src/**/features/*:
- Amend the new fail error messages to recommend exporting
IFFEFLAGS=-d1 to show the cause of the failure.
README.md, TODO:
- Updates.
It is not correct to save sh.ifstable (a.k.a. shp->ifstable) before
calling a function and then restore it after; this can cause field
splitting to malfunction. See 70368c57.
The change to init.c in the Red Hat patch applied in 18b3f4aa
(shp->ifstable[0] = S_EOF) appears to be sufficient.
src/cmd/ksh93/bltins/alarm.c:
- Revert save/restore of sh.ifstable.
src/cmd/ksh93/tests/builtins.sh:
- Tweak the regression test to work correctly on a slower machine,
i.e. a Raspberry Pi running FreeBSD 12.2 arm64 (thanks to hyenias
for providing testing access).
The only proper documentation of the MAM language is in Glenn
Fowler's paper, which is unfortunately copyrighted so we can't
include it. But we can at least provide a link to it.
src/**/Mamfile:
- Add header comment.
src/cmd/INIT/mamake.c:
- Re-enable clang warnings on unused values (there aren't any).
This commit resolves the following incorrect variable assignments:
$ unset a; typeset -uF a=2; typeset -p a
typeset -X a=0x1.0000000000p+1
$ unset a; typeset -Fu a=2; typeset -p a
typeset -X a=0x1.0000000000p+1
$ unset a; typeset -ulF a=2; typeset -p a
typeset -l -X a=0x1.0000000000p+1
$ unset a; typeset -Ful a=2; typeset -p a
typeset -l -X a=0x1.0000000000p+1
$ unset a; typeset -Eu a=2; typeset -p a
typeset -E -X a=2
$ unset a; typeset -Eul a=2; typeset -p a
typeset -l -E -X a=2
src/cmd/ksh93/bltins/typeset.c:
- If the unsigned option (-u) was provided in conjunction with a
floating point (-F) then due to a flag collision with NV_UNSIGN
and NV_HEXFLOAT both having the value of NV_LTOU caused the
floating point to become a hexadecimal floating point (-X) in
error. Also, if a -E option flag was followed with a -u option
then the resulting variable would be both a scientific notation
and a hexadecimal floating point at the same time.
src/cmd/ksh93/tests/attributes.sh:
- Add regression tests.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
An unquoted variable expansion evaluated in a DEBUG trap action
caused IFS field splitting to be deactivated in code executed after
the trap action. Thanks to Koichi Nakashima for the reproducer:
| v=''
| trap ': $v' DEBUG
| A="a b c"
| set -- $A
| printf '%s\n' "$@"
|
| Expected
|
| a
| b
| c
|
| Actual
|
| a b c
src/cmd/ksh93/sh/fault.c: sh_trap():
- Remove incorrect save/restore of sh.ifstable, the internal state
table for field splitting. This reverts three lines added in ksh
93t+ 2009-11-30. Analysis: As an expansion is split into fields
(macro.c, lines 2367-2471), sh.ifstable is modified. If that
happens within a DEBUG trap, any modifications in ifstable are
undone by the restoring memccpy, leaving an inconsistent state.
src/cmd/ksh93/COMPATIBILITY:
- Document the DEBUG trap fixes, particularly the incorrect
inheritance by subshells and functions that some scripts may now
rely on because this bug is so longstanding. (re: 2a835a2d)
src/cmd/ksh93/tests/basic.sh:
- Add relevant tests.
Resolves: https://github.com/ksh93/ksh/issues/155
TODO: add a -T (-o functrace) option as in bash, which should allow
subshells and ksh-style functions to inherit DEBUG traps.
P.S.: The very handy multishell repo allows us to use 'git blame'
to trace the origin of the recently fixed DEBUG trap bugs.
The off-by-one error causing various bugs, reverted in 2a835a2d,
was introduced in ksh 93t 2008-07-25:
https://github.com/multishell/ksh93/commit/8e947ccf
(fault.c, line 321)
The incorrect check causing the exit status bug, reverted in
d00b4b39, was introduced in ksh 93t 2008-11-04:
https://github.com/multishell/ksh93/commit/b1ade268
(fault.c, line 459)
The ifstable save/restore causing the field splitting bug, reverted
in this commit, was introduced in ksh 93t+ 2009-11-30:
https://github.com/multishell/ksh93/commit/53d9f009
(fault.c, lines 440, 444, 482)
So all the bugs reported in #155 were fixed by simply reverting
these specific changes. I think that they are some experiments that
the developers simply forgot to remove. I've suspected such a thing
multiple times before. ksh93 was developed by researchers who were
genius innovators, but incredibly sloppy maintainers.
Turns out the previous commit also fixed the bug that disables the
DEBUG trap if a redirection is used in a DEBUG trap action -- in
other words, that's the same bug.
src/cmd/ksh93/tests/basic.sh:
- Add test from the reproducer in the bug report.
Makes progress on: https://github.com/ksh93/ksh/issues/155
This trap failed to be restored correctly when being trapped in
a subshell, causing corruption or a crash when restoring the
parent shell environment's trap upon leaving the subshell.
Thanks to Koichi Nakashima for the report and reproducer.
src/cmd/ksh93/sh/fault.c: sh_sigreset():
- Fix an off-by-one error in the loop that restores the
pseudosignal traps.
src/cmd/ksh93/tests/basic.sh:
- Test overwriting the main shell trap in a subshell for all
pseudosignals.
Makes progress on: https://github.com/ksh93/ksh/issues/155
The -P option only ever worked on Solaris so it's questionable it
should have been in the general-purpose manual to begin with. And
now it doesn't even work on Solaris as it disable SHOPT_PFSH with a
patch (that functionality is now provided by a wrapper that works
with all shells). So it's long past time to stop documenting it.
For the same reason, this also removes the info about invoking ksh
as pfksh, etc. -- this is still possible on Solaris with the new
method, but the functionality is no longer actually provided by
ksh. If the Solaris maintainers want it back in the man page, that
should be done by adding a patch to their build system.
UnixWare's ps prefers to read psinfo (from the proc structure in
kernel memory) within /proc as an anti-Trojan horse measure.
Updates to argv[0] are still reflected within /proc/$pid/cmdline,
which is useful for diagnostic purposes.
src/cmd/ksh93/sh/main.c:
- Remove __USLC__ from the list of platforms excluded from the
fixargs method.
src/cmd/ksh93/tests/basic.sh:
- Read /proc/$pid/cmdline instead of ps on UnixWare.
SHOPT_KIA enables the -R option that generates a cross-reference
database from a script. However, no tool to analyse this database
is shipped or seems to be available anywhere (in spite of multiple
people looking for one), and the format is very opaque. No usage
examples are known or findable on the internet. This seems like it
should not be compiled in by default, although we'll keep the code
in case some way to use it is found.
src/cmd/ksh93/SHOPT.sh:
- Disable SHOPT_KIA by default by removing the default 1 value.
src/cmd/ksh93/sh/args.c, src/cmd/ksh93/sh/parse.c:
- Fix a couple of preprocessor logic bugs that made it impossible
to compile ksh without SHOPT_KIA.
src/cmd/ksh93/data/builtins.c:
- Fix typo in -R doc in ksh --man (in case SHOPT_KIA is enabled).
src/cmd/ksh93/sh.1:
- Since sh.1 is not generated dynamically, remove the -R doc.
The build system is adapted to make SHOPT_* compile-time options
editable without nmake. We can now easily change ksh's compile-time
options by editing src/cmd/ksh93/SHOPT.sh. The bin/package script
is adapted to turn these into compile flags. This resolves the most
important drawback of not using nmake.
Also, mamake now has support for indented Mam (Make Abstract
Machine) code. Only one type of block (make...done) is supported in
Mamfiles, so they are easy to indent automatically. A script to
(re)do this is included.
Since nmake is not going to be restored (it has too many problems
that no one is interested in fixing), this at least makes mamake
significantly easier to work with.
The Makefiles are deleted. They may still be handy for reference to
understand the Mamfiles, but they haven't actually matched the
Mamfiles for a while -- and you can still look in the git history.
Deleting them requires some adaptations to bin/package and mamake.c
because, even though they do not use those files, they still looked
for them to decide whether to build code in a directory.
Finally, this commit incorporates some #pragmas for clang to
suppress annoying warnings about the coding style used in this
historic code base. (gcc does not complain so much.)
src/cmd/ksh93/SHOPT.sh:
- Added.
bin/package, src/cmd/INIT/package.sh:
- cd into our own directory in case we were run from another dir.
- $makefiles: only look for Mamfiles.
- Add ksh compile-options via KSH_SHOPTFLAGS. Include SHOPT.sh.
- make_recurse(): Do not write a missing Makefile.
- finalize environment: Look for Mamfiles instead of Makefiles.
src/cmd/INIT/mamake.c:
- Tell clang to suppress annoying warnings about coding style.
- Update version string and self-documentation.
- input(): Add support for indented Mam code by skipping initial
whitespace on each input line.
- files[]: Instead of looking for various of Makefiles to decide
where to build, only look for Mamfiles.
src/Makefile, src/cmd/INIT/Makefile, src/cmd/Makefile,
src/cmd/builtin/Makefile, src/cmd/ksh93/Makefile, src/lib/Makefile,
src/lib/libast/Makefile, src/lib/libcmd/Makefile,
src/lib/libdll/Makefile, src/lib/libsum/Makefile:
- Removed.
src/Mamfile, src/cmd/INIT/Mamfile, src/cmd/Mamfile,
src/cmd/builtin/Mamfile, src/cmd/ksh93/Mamfile, src/lib/Mamfile,
src/lib/libast/Mamfile, src/lib/libcmd/Mamfile,
src/lib/libdll/Mamfile, src/lib/libsum/Mamfile:
- Indent the code with tabs.
- In ksh93/Mamfile, add ${KSH_SHOPT_FLAGS} to every $CC command.
- In ksh93/Mamfile, add "prev SHOPT.sh" for every *.o file
so they are rebuilt whenever SHOPT.sh changes.
bin/Mamfile_indent:
- Added, in case someone wants to re-indent a Mamfile.
src/cmd/INIT/proto.c, src/cmd/INIT/ratz.c, src/cmd/INIT/release.c,
src/lib/libast/features/common, src/lib/libast/include/ast.h:
- Tell clang to suppress annoying warnings about coding style that
it disapproves of (mainly concerning the use of parentheses).
src/cmd/INIT/cc.darwin, src/cmd/INIT/cc.freebsd,
src/cmd/INIT/cc.openbsd:
- Remove now-redundant clang warning suppression flags.
Resolves: https://github.com/ksh93/ksh/issues/60
What is this for? See cefe087d
src/cmd/ksh93/Mamfile:
- Make iffe generate a test for the presence of setproctitle(3).
src/cmd/ksh93/sh/main.c:
- Include setproctitle test result.
- Re-enable fixargs() for FreeBSD and DragonFly BSD.
Disable it for UnixWare.
- fixargs(): Add _lib_setproctitle version. Keep it simple with a
128-character buffer array -- should be plenty for 'ps' output.
- fixargs(): Fix an off-by-one in zeroing the rest of the buffer.
src/cmd/ksh93/tests/basic.sh:
- Update the relevant regression test to run on FreeBSD/DragonFly
and tolerate the "ksh: " prefix added by setproctitle(3).
src/cmd/ksh93/include/version.h:
- Centrally define the 93u+m copyright (SH_RELEASE_CPYR) for adding
to the original AT&T copyright in 'ksh --man' and 'shcomp --man'.
- Centrally define the binary header version number for bytecode
generated by shcomp: SHCOMP_HDR_VERSION.
- Bump SHCOMP_HDR_VERSION from 3 to 4. Converting all the preset
aliases to builtin commands has caused new bytecode to be
incompatible with old ksh. (However, old bytecode runs fine on
93u+m, because shcomp pre-expands the preset aliases.)
src/cmd/ksh93/sh/shcomp.c:
- Instead of keeping its own version date (not changed since 2003),
use the same version string as ksh itself (SH_RELEASE).
- Use SH_RELEASE_CPYR for the extra 93u+m copyright string.
- Use SHCOMP_HDR_VERSION for the bytecode header.
src/cmd/ksh93/sh/parse.c: sh_parse():
- Use SHCOMP_HDR_VERSION for the bytecode version check.
src/cmd/ksh93/data/builtins.c: opt_ksh[]:
- Use SH_RELEASE_CPYR for the extra 93u+m copyright string.
src/cmd/ksh93/COMPATIBILITY:
- Mention that 93u+m shcomp bytecode won't run on older ksh.
- Document changes in printf %T (re: 9526b3fa).
src/cmd/ksh93/README:
- Mention that we run on UnixWare (with major regressions).
https://github.com/ksh93/ksh/pull/159#issuecomment-764667929
This adds informative error messages if incompatible options are
given. It also documents the exclusive -m, -n and -T options on
separate usage lines, as was already done with -f. The usage
message for incompatible options now looks something like this:
| $ ksh -c 'typeset -L10 -F -f -i foo'
| ksh: typeset: -i/-F/-E/-X cannot be used with -L/-R/-Z
| ksh: typeset: -f cannot be used with other options
| Usage: typeset [-bflmnprstuxACHS] [-a[type]] [-i[base]] [-E[n]]
| [-F[n]] [-L[n]] [-M[mapping]] [-R[n]] [-X[n]]
| [-h string] [-T[tname]] [-Z[n]] [name[=value]...]
| Or: typeset -f [name...]
| Or: typeset -m [name=name...]
| Or: typeset -n [name=name...]
| Or: typeset -T [tname[=(type definition)]...]
| Help: typeset [ --help | --man ] 2>&1
(see also the previous commit, e21a053e)
Unfortunately the first "Usage" line has some redundancies with the
"Or:" lines showing separate usages. It doesn't seem to be possible
to avoid this; it's a flaw in how libast generates everything
(usage, help, manual) from one huge getopt(3) string. I still think
the three added "Or:" lines are an improvement as it wasn't
previously shown that these options need to be used on their own.
src/cmd/ksh93/bltins/typeset.c: b_typeset():
- Instead of only showing a generic usage message, add an
informative error message if incompatible options were given.
- Conflicting options detection was failing because NV_LJUST and
NV_EXPNOTE have the same bitmask value. Use a new 'isadjust'
flag for -L/-R/-Z to remember if one of these was set.
- Detect conflict between -L/-R/-Z and a float option, not just -i.
src/cmd/ksh93/include/name.h, src/cmd/ksh93/data/msg.c:
- Add the two new error messages for incompatible options.
src/cmd/ksh93/data/builtins.c: sh_opttypeset[]:
- Add a space after 'float' in in "[+float?\btypeset -lE\b]" as
this makes 'float' appear on its own line, improving formatting.
- Show -m, -n, -T on separate usage lines like -f, as none of these
can be combined with other options.
- Remove "cannot be combined with other options" from -m and -n
descriptions, as that should now be clear from the separate usage
lines -- and even if not, the error message is now informative.
src/cmd/ksh93/sh.1, src/cmd/ksh93/COMPATIBILITY:
- Update.
src/cmd/ksh93/tests/types.sh:
- Remove obsolete test: 'typeset -RF' is no longer accepted.
(It crashed in 93u+, so this is not an incompatibility...)
Resolves: https://github.com/ksh93/ksh/issues/48
For example, this changes 'typeset -Q' (a bad option) from:
| ksh: typeset: -Q: unknown option
| Usage: typeset [-bflmnprstuxACHS] [-a[type]] [-i[base]] [-E[n]]
| [-F[n]] [-L[n]] [-M[mapping]] [-R[n]] [-X[n]]
| [-h string] [-T[tname]] [-Z[n]] [name[=value]...]
| Or: typeset [ options ] -f [name...]
to:
| ksh: typeset: -Q: unknown option
| Usage: typeset [-bflmnprstuxACHS] [-a[type]] [-i[base]] [-E[n]]
| [-F[n]] [-L[n]] [-M[mapping]] [-R[n]] [-X[n]]
| [-h string] [-T[tname]] [-Z[n]] [name[=value]...]
| Or: typeset -f [name...]
| Help: typeset [ --help | --man ] 2>&1
src/lib/libast/misc/optget.c: args():
- Revert the changes done in 6916a873 and ae92cd89. The --help and
--man labels weren't added consistently (they did not show up in
the example above) whereas they did show up unnecessarily in the
manual page itself.
- In the usage section and usage messges, only show an [ options ]
label on the first usage line; don't redundantly repeat on second
and further ("Or:") lines.
- In usage and --help (but not --man), add a new "Help:" line
telling the user about the --help and --man options. This
replaces the reverted changes. Show the 2>&1 redirection as a
reminder that you need to do this to pipe it into a pager, as
everything is written to standard error!
- Add some comments clarifying what I think this code does...
src/cmd/ksh93/tests/builtins.sh:
- Update to match changes in getopts usage output.
src/cmd/ksh93/bltins/typeset.c:
- The new_argv[] array was one item too short (should be argc+2).
- Use AST stakalloc(3) to allocate it instead of a dynamic array;
this restores compatibility with ISO C90.
src/lib/libast/features/standards, src/cmd/INIT/cc.unixware.i386:
- Add support for UnixWare.
- Do not define any standards macros on this system, as on FreeBSD
and DragonFly BSD.
This fixes the following:
trap ':' DEBUG
r=$(exit 123)
echo $? # Expected 123, but actually 0.
Thanks to Koichi Nakashima for the report and reproducer.
src/cmd/ksh93/sh/fault.c: sh_trap():
- Restore the saved current exit status (exitval) for all traps.
Do not except the DEBUG trap from doing that. I've no idea why
this exception was made, but it's not correct.
src/cmd/ksh93/tests/basic.sh:
- Add tests.
Makes progress on: https://github.com/ksh93/ksh/issues/155
Commit d1483150 did not fully fix#153.
Test case from Harald van Dijk that was still failing:
$ mkdir test
$ cd test
$ rmdir $PWD
$ mkdir $PWD
$ ksh -c "(cd /); pwd"
/
Forking a virtual subshell in that case is needed to avoid ending
up in a directory that replaced the PWD, because it will not be
possible for a process to change back to the original directory.
src/cmd/ksh93/bltins/cd_pwd.c:
- When deciding whether to fork, instead of attempting to opendir
the PWD, compare the inodes $PWD and "." to determine if $PWD
still actually refers to the current directory. This uses the
test_inode() function which is also used by 'test foo -ef bar'.
src/cmd/ksh93/tests/subshell.sh:
- Add test based on the above.
Progresses: https://github.com/ksh93/ksh/issues/153
src/cmd/ksh93/tests/variables.sh:
- Fork the subshell with the test that includes unsetting LINENO
and changing its type. Otherwise, some side effect of that leaks
out of the subshell, messing up $LINENO. This is a bug, but it's
low priority -- we may get to it someday. Marked with a TODO.
- Do the LC_* tests in their own subshell. Skip them if changing
LANG to an invalid value does not produce a diagnostic message.
This occurs on OpenBSD and Alpine Linux (with musl libc). It
looks like their C libraries do not verify the locale, so
failures here are not a ksh problem; skip the tests in that case.
AIX on ibm.risc comes with a broken version of ksh88 as /bin/sh
where the following causes breakage in the parser (spurious syntax
errors):
(set -o posix) 2>/dev/null && set -o posix
However, prefixing it with 'command' (while keeping the subshell)
circumvents the problem. So, why not.
(command set -o posix) 2>/dev/null && set -o posix
This commit also further mitigates the problems with restoring an
inaccessible or nonexistent PWD on exiting a virtual subshell.
Harald van Dijk writes:
> On a build of ksh with -fsanitize=undefined to help diagnose
> problems:
>
> $ mkdir deleted
> $ cd deleted
> $ rmdir ../deleted
> $ ksh -c '(cd /; (cd /)); :'
> /home/harald/ksh/src/cmd/ksh93/sh/subshell.c:561:22: runtime
> error: null pointer passed as argument 1, which is declared to
> never be null
> Segmentation fault (core dumped)
>
> Note that it segfaults the same with default compilation flags,
> but it does not print out the useful extra message. The code
> assumes that pwd is non-null and passes it to strcmp without
> checking, but it will be null if the current directory cannot be
> determined, for instance because it has been deleted.
src/cmd/ksh93/sh/subshell.c: sh_subshell():
- Avoid the null pointer dereference reported above.
src/cmd/ksh93/bltins/cd_pwd.c: b_cd():
- Fork a virtual subshell even on systems with fchdir(2) if the
present working directory tests as inaccessible on invoking 'cd';
it may no longer exist and fchdir would fail to get a handle.
(For the test we have to opendir(3) the full path to the PWD and
not ".", as the latter may succeed even if the PWD is gone.)
src/cmd/ksh93/data/builtins.c:
- Update 'cd' version string.
Fixes: https://github.com/ksh93/ksh/issues/153
Related: https://github.com/ksh93/ksh/issues/141
src/cmd/ksh93/sh/main.c: sh_main():
- Reading the code makes it obvious that the shp->comdiv-- decrease
in the 'else' block is never reached unless that pointer is still
null, in which case it makes no sense to decrease it. Must be
some kind of missed leftover from old code. Remove the decrease.
This change is backported from the abandoned ksh 93v- beta.
src/cmd/ksh93/sh/subshell.c: sh_subsavefd():
- Do not subtract 1 from fd, as this would cause a negative shift
operand for stdin (fd==0).
Instead, we now link to the libm system math library where needed
by adding -lm to the relevant compile commands in the Mamfiles.
This is not needed on every system but never does any harm.
(This adds more custom edits to the Mamfiles, which were originally
generated from the nmake Makefiles. This takes us further from
restoring nmake, but that already wasn't going to happen anyway,
due to its many problems... the path forward will be to translate
the Mamfiles to some other, current make system such as GNU make.)
bin/package, src/cmd/INIT/package.sh:
- Remove LDFLAGS=-lm hack for DragonFly BSD, NetBSD and Solaris.
src/cmd/builtin/Mamfile,
src/cmd/ksh93/Mamfile,
src/lib/libdll/Mamfile:
- Add -lm where linking failed on any of the three mentioned OSs.
src/lib/libdll/features/dll:
- In the output test program, add missing #include <math.h>, fixing
unknown identifier errors on NetBSD (ldexp, ldexpl).
src/cmd/builtin/features/pty:
- Add missing #include <stdio.h> to make printf work on all systems
(this is just a feature test, no need to bother with sfio here).
src/lib/libast/features/stdio:
- Undef __FILE_T to avoid interference from system headers on QNX.
(There are still other problems preventing a build on QNX 6.5.0.
The shipped version of gcc seems to be broken.)
This now makes ksh build on DragonFly BSD.
bin/package, src/cmd/INIT/package.sh:
- DragonFly also needs the -lm hack for LDFLAGS.
src/cmd/ksh93/sh/main.c, src/cmd/ksh93/tests/basic.sh:
- fixargs() doesn't work on DragonFly either
(re: 9b7c392a, 159fb9ee, cefe087d).
The following are backported from:
https://github.com/att/ast/issues/26#issuecomment-313927854https://github.com/att/ast/pull/19
src/lib/libast/comp/setlocale.c:
- Add missing #include <errno.h> since errno is used.
src/lib/libast/features/standards:
- Do not set any standards macros (_POSIX_SOURCE etc) on FreeBSD or
DragonflyBSD; they disable too much functionality on those.
src/lib/libast/features/wchar:
- Set _STDFILE_DECLARED on DragonFly, too.
src/lib/libast/include/sfio.h, src/lib/libast/include/sfio_t.h,
src/lib/libast/sfio/_sfopen.c, src/lib/libast/sfio/sfclrlock.c,
src/lib/libast/sfio/sfhdr.h, src/lib/libast/sfio/sfnew.c,
src/lib/libast/sfio/sfset.c:
- Rename SF_* macros to SFIO_* to avoid a conflict with system
headers.
src/lib/libast/string/strexpr.c:
- Rename error() to err() to avoid a conflict.
~- and ~+ are ksh93-specific tilde expansions that expand to
$OLDPWD and $PWD, respectively. On some systems, $OLDPWD is not set
on entry to the test script, because it is not exported to the
environment. This made it unset before any 'cd' was executed,
which (correctly) disabled ~- expansion.
src/cmd/ksh93/tests/basic.sh:
- Before testing 'cd ~-', make sure $OLDPWD is set by cd'ing to
/dev first (a directory guaranteed by POSIX).
Solaris /bin/ksh disables the SHOPT_PFSH compile option ("solaris
exec_attr(4) profile execution") with a patch. Since this option
applies to Solaris and variants only, let's upstream that change.
(Solaris now provides pfksh93 as a wrapper around ksh93, and does
the same for other shells, so profiling functionality is no longer
ksh-specific.)
If you want to re-enable it, add -DSHOPT_PFSH to your $CCFLAGS.
Original patch:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/150-CR7168611.patch
src/cmd/ksh93/Makefile:
- Add note that edits in Makefile are ineffective as we do not ship
nmake.
- Disable SHOPT_PFSH, cosmetically.
src/cmd/ksh93/Mamfile:
- Remove -DSHOPT_PFSH from all compiler commands.
bin/package, src/cmd/INIT/package.sh:
- CCFLAGS overwrites the autodetected optimisation flags (e.g. -Os)
if set. Unfortunately, that also happened when we added something
to CCFLAGS for a release build or to add an extra flag needed by
Solaris. The fix is to use a new flags variable (KSH_RELFLAGS)
instead. This needs to be done in a different place as it needs
to be added to the mamake command as an assignment argument.
- Remove the Solaris CCFLAGS hack; see features/common below.
src/*/*/Mamfile:
- Add ${KSH_RELFLAGS} to all the compiler commands.
src/lib/libast/features/common:
- Enable POSIX standard on Solaris (i.e.: if __sun is defined) by
defining _XPG6 directly in the feature test that generates
ast_std.h, which is indirectly included by everything. This
removes the need to pass -D_XPG6 via CCFLAGS. (Doing so
automatically with gcc was not otherwise possible.)
src/cmd/INIT/cc.sol11.*:
- No longer pass -D_XPG6, as per above.
src/cmd/ksh93/sh/main.c: fixargs():
- Erase the entire length of the command arguments buffer (the
space from argv[0] until environ[0]) so that remnants of longer
command arguments aren't left in 'ps' output when executing a
hashbang-ess script with a shorter command line.
- Disable fixargs() on FreeBSD. It has never had any effect on that
system; apparently it either requires another method to rewrite
arguments for 'ps' output purposes (which?) or it's not possible.
src/cmd/ksh93/tests/basic.sh:
- Skip the test if running on FreeBSD.
This applies a patch from Solaris:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/160-CR7175995.patch
There is no public information on why it's needed, but it seems
sensible on the face of it. Using a file called '.profile' in the
PWD on login, without a directory path, is redundant at best, since
"$HOME/.profile" (e_profile, see data/msg.c) is already used. And
if the PWD is not $HOME at login time, it seems to me there are
serious problems and the last thing you want is to read some
random and probably dodgy '.profile' from the PWD.
src/cmd/ksh93/sh/init.c: sh_init(): login_files[]:
- Remove redundant/problematic ".profile" entry.
This change was pulled in from:
https://raw.githubusercontent.com/oracle/solaris-userland/master/components/ksh93/patches/185-Bug17714341.patch
No public information about the reasons for this change is
available, but it seems reasonable to trust that the Solaris people
found a legitimate need for it.
src/cmd/ksh93/bltins/cd_pwd.c: b_cd():
- When determining the old PWD before 'cd', do not trust shp->pwd
but get and validate the current PWD using path_pwd().
