_sfcvt(), "convert a floating point value to ASCII", did not adjust
for negative decimal place movement as what happens with leading
zeroes. This caused ksh's 'printf %f' formatter to fail to round
floating point values correctly.
src/lib/libast/sfio/sfcvt.c:
- Removed constraint of <1e-8 for doubles by matching what was done
for long doubles having <.1.
- Corrected a condition when the next power of 10 occurred and that
new 1 digit was being overwritten by a 0.
src/cmd/ksh93/tests/math.sh:
- Validate that typeset -E/F formatting matches that of their
equivalent printf formatting options as well as checking for
correct float scaling of the fractional parts.
The fix was incomplete: expansions using '?' (${var?w(ord},
${var:?wo)rd}) still did not tolerate parentheses in the word
as regular characters.
It was also possible to simplify the fix by making use of the
ST_BRACE (sh_lexstate7[]) state table. See data/lexstates.c and
include/lexstates.h.
src/cmd/ksh93/sh/lex.c: sh_lex(): case S_MOD1:
- The previous fix tested for modifier operator characters : - + =
as part of the S_MOD2 case, though they are defined as S_MOD1 in
the ST_BRACE state table. It only worked because of the
fallthrough. And it turns out the S_MOD1 case already had a
similar fix, though incomplete. The new fix effectively cancelled
the old one out as any S_MOD1 character eventually led to
'continue'. So it can be simplified by removing most of that
code, without causing any change in behaviour. Only the mode
change to the ST_QUOTE state table followed by 'continue' is
necessary. This also fixes it for the '?' operator as that is
also defined as S_MOD1 in the ST_BRACE state table.
src/cmd/ksh93/sh/macro.c:
- When skipping a ${...} expansion using sh_lexskip(), use the
ST_QUOTE state table if the character c is an S_MOD1 modifier
operator character. This makes it consistent with the S_MOD1
handling in sh_lex().
src/cmd/ksh93/tests/variables.sh:
- Update regression tests to include ? and :? operators.
File descriptors are not properly closed, causing a leak, when
using a process substitution as an argument to a shell function.
See: https://github.com/ksh93/ksh/issues/67
Process substitution uses /dev/fd/NN pseudofiles if the kernel
provides them. This is tested in src/cmd/ksh93/features/options
which causes SHOPT_DEVFD to be defined if /dev/fd/9 can be used.
If not, ksh uses a fallback mechanism involving a temporary FIFO,
which works on all Unix variants.
As it happens, the leak only occurs when using the /dev/fd
mechanism. So, until a fix is found, we can work around the bug by
disabling it. The FIFO mechanism might be slightly less robust,
but it's an improvement over leaking file descriptors. Plus, there
is room for improving it.
src/cmd/ksh93/include/defs.h:
- Unconditionally redefine SHOPT_DEVFD as 0 for now.
src/cmd/ksh93/sh/args.c: sh_argprocsub():
- pathtemp() does appropriate access checks using access(2), but
there is an inherent race condition between calling it and
mkfifo(). Make the FIFO mechanism more robust by handling errors,
trying again if an error occurs that must have resulted from
losing that race, e.g. file name conflict or temp dir
permission/location change.
- Initially create the FIFO without any permissions, then chmod()
the appropriate user read/write permissions. Since mkfifo()
honours the umask and chmod() does not, this ensures that process
substitution continues to work if a shell script sets a umask
that disallows user read or write. (The /dev/fd/ mechanism does
not care about the umask, so neither should the fallback.)
To find the temporary files directory to use, the pathtemp()
function (generate a unique path to a temporary file) first checks
$TMPDIR and $TMPPATH, then falls back to /tmp, then to /usr/tmp as
a last resort. But all systems replaced /usr/tmp by /var/tmp
decades ago to allow mounting /usr as read-only, and a /usr/tmp
compatibility symlink is no longer commonly provided.
src/lib/libast/path/pathtemp.c:
- Change TMP2 definition from "/usr/tmp" to "/var/tmp".
src/lib/libast/features/mmap,
src/lib/libast/features/stdio:
- Change "/usr/tmp" to "/var/tmp" in feature tests.
Following a community discussion, it became clear that 'r' is
particularly problematic as a regular builtin, as the name can and
does conflict with at least one legit external command by that
name. There was a consensus against removing it altogether and
letting users set the alias in their login scripts. However,
aliases are easier to bypass, remove or rename than builtins are.
My compromise is to reinstate 'r' as a preset alias on interactive
shells only, along with 'history', as was done in 17f81ebe before
they were converted to builtins in 03224ae3. So this reintroduces
the notion of predefined aliases to ksh 93u+m, but only for
interactive shells that are not initialised in POSIX mode.
src/cmd/ksh93/Makefile,
src/cmd/ksh93/Mamfile,
src/cmd/ksh93/include/shtable.h,
src/cmd/ksh93/data/aliases.c:
- Restore aliases.c containing shtab_aliases[], a table specifying
the preset aliases.
src/cmd/ksh93/include/shtable.h,
src/cmd/ksh93/sh/init.c:
- Rename inittree() to sh_inittree() and make it extern, because we
need to use it in main.c (sh_main()).
src/cmd/ksh93/sh/main.c: sh_main():
- Init preset aliases from shtab_aliases[] only if the shell is
interactive and not in POSIX mode.
src/cmd/ksh93/bltins/typeset.c,
src/cmd/ksh93/tests/alias.sh:
- unall(): When unsetting an alias, pass on the NV_NOFREE attribute
to nv_delete() to avoid an erroneous attempt to free a preset
alias from read-only memory. See: 5d50f825
src/cmd/ksh93/data/builtins.c:
- Remove "history" and "r" entries from shtab_builtins[].
- Revert changes to inline fc/hist docs in sh_opthist[].
src/cmd/ksh93/bltins/hist.c: b_hist():
- Remove handling for 'history' and 'r' as builtins.
src/cmd/ksh93/sh.1:
- Update accordingly.
Resolves: https://github.com/ksh93/ksh/issues/125
The 'command' name can now result from an expansion, e.g.:
c=command; "$c" ls
set -- command ls; "$@"
both work now. This fixes BUG_CMDEXPAN.
If -o posix is on, 'command' now disables not only the "special"
but also the "declaration" properties of builtin commands that it
invokes. This is because POSIX specifies 'command' as a simple
regular builtin, and any command name following 'command' is just
an argument to the 'command' command, so there is nothing that
allows any further arguments (such as assignment-arguments) to be
treated specially by the parser. So, if and only if -o posix is on:
a. Arguments that start with a variable name followed by '=' are
always treated as regular words subject to normal shell syntax.
b. Since assignment-arguments are not processed as assignments
before the command itself, 'command' can now stop the shell from
exiting (as required by the standard) if a command that it
invokes (such as 'export') tries to modify a readonly variable.
This fixes BUG_CMDSPEXIT.
Most of 'command' is integrated in the parser and parse tree
executer, so that is where it needed fixing.
src/cmd/ksh93/sh/parse.c: simple():
- If the posix option is on, do not skip past SYSCOMMAND so that
any declaration builtin commands that are arguments to 'command'
are not detected and thus not treated specially at parsetime.
src/cmd/ksh93/sh/xec.c: sh_exec():
- When detecting SYSCOMMAND in order to skip past it, not only
compare the Namval_t pointer 'np' to SYSCOMMAND, but also handle
the case where that pointer is NULL, as when the command name
results from an expansion. In that case, search the function tree
shp->fun_tree for the name and see if that yields the SYSCOMMAND
pointer. fun_tree is initialised with a dtview to bltin_tree, so
searching fun_tree instead allows for overriding 'command' with a
shell function (which the POSIX standard requires us to allow).
src/cmd/ksh93/sh.1,
src/cmd/ksh93/data/builtins.c:
- Update documentation to match these changes.
- Various related edits and improvements.
src/cmd/ksh93/tests/builtins.sh:
- Check that 'command' works if resulting from an expansion.
- Check that 'command' can be overridden by a shell function.
The 'exit' and 'return' commands without an argument failed to pass
down the exit status of the last-run command when incorporated in a
block with redirection, &&/|| list, 'case' statement, or 'while',
'until' or 'for' loop.
src/cmd/ksh93/bltins/cflow.c:
- Use $?, which is sh.savexit a.k.a. shp->savexit, as the default
exit status value if there is no argument, instead of
shp->oldexit. This fixes the default exit status behaviour to
match POSIX and other shells.
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/include/shell.h:
- Remove now-unused sh.oldexit (a.k.a. shp->oldexit) private struct
member. It appeared to fulfill the same function as sh.savexit,
but in a slightly broken way.
- Move the savexit/$? declaration from the _SH_PRIVATE part of the
struct definition to the public API part. Since $? uses this,
it's clearly a publicly exposed value already, and this is
generally the one to use. (If anything, it's exitval that should
have been private.) This declares savexit right next to exitval,
rewriting the comments to clarify the difference between them.
src/cmd/ksh93/sh/fault.c,
src/cmd/ksh93/sh/subshell.c,
src/cmd/ksh93/sh/xec.c:
- Remove assignments to shp->oldexit.
src/cmd/ksh93/tests/basic.sh:
- Add thorough regression tests for the default exit status
behaviour of 'return' and 'exit' in various lexical contexts.
- Verify that 'for' and 'case' without any command, as well as a
lone redirection, still correctly reset the exit status to 0.
Fixes: #117
src/cmd/ksh93/include/jobs.h:
- The commit that removed legacy code mistakenly removed the
definition of vmbusy() required for ksh to compile with
-D_std_malloc. Ksh assumes vmbusy is always a macro, even
when _std_malloc is defined. This commit reintroduces the
_std_malloc definition of vmbusy to fix undefined
reference errors.
A coprocess cleanup test could fail on rare occasions because I had
lowered the 'sleep 1' between two test coprocesses to 'sleep .1'.
This increases the sleep to prevent future spurious fails.
Fixes: https://github.com/ksh93/ksh/issues/129
Using a process of elimination I've identified ${.sh.level}
(SH_LEVELNOD) as the cause of the crash. This node apparently
cannot be copied or moved without destabilising the shell. It
contains the current depth of function calls and it cannot be
changed by assignment, so this is not actually a problem.
Meanwhile, this commit re-fixes it for the other three.
src/cmd/ksh93/sh/subshell.c:
- Simplify sh_assignok() by removing special-casing for L_ARGNOD,
SH_SUBSCRNOD and SH_NAMENOD. 'add' now has 3 modes (0, 1, 2).
- The test for a ${ subshare; } was actually wrong. sp->subshare is
a saved backup value. We must test shp->subshare. (re: a9de50bf)
src/cmd/ksh93/bltins/typeset.c:
- setall(): Update the mode 3 sh_assignok() call.
src/cmd/ksh93/tests/variables.sh:
- Regress-test subshell leaks for all special variables except
${.sh.level}.
This reverts commit b3d37b00b0.
While ksh's own regression test suite passed just fine, when
running the modernish[*] regression tests uite, ksh either froze
hard (needing SIGKILL) or threw a spurious syntax error.
Cause unknown, but I'm certainly reverting until I find out.
This reintroduces a subshell leak for four special variables.
[*] https://github.com/modernish/modernish
${var:-wor)d} or ${var+w(ord}. The parentheses now correctly lose
their normal grammatical meaning within the braces. Fix by Eric
Scrivner (@etscrivner) from July 2018 backported from ksh2020.
This fix complies with POSIX:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02
src/cmd/ksh93/sh/lex.c: sh_lex():
- Set the ST_QUOTE state when analysing a modifier with parameter
expansions using operators ':', '-', '+', '='. This state causes
subsequent characters (including parentheses) to be considered
quoted, suppressing their normal grammatical meaning.
src/cmd/ksh93/sh/macro.c: varsub():
- Same for skipping the expansion.
Fixes: https://github.com/ksh93/ksh/issues/126
Prior discussion: https://github.com/att/ast/issues/475
The following special variables leaked out of a subshell:
$_, ${.sh.name}, ${.sh.level}, ${.sh.subscript}.
This was due to a faulty optimisation in sh_assignok().
bd3e2a80 fixed that in part, this fixes the rest.
src/cmd/ksh93/sh/subshell.c:
- Simplify sh_assignok() by removing special-casing for these four
special variables. The 'add' param reverts to a simple boolean.
- The test for a ${ subshare; } was actually wrong. sp->subshare is
a saved backup value. We must test shp->subshare. (re: a9de50bf)
src/cmd/ksh93/bltins/typeset.c:
- setall(), unall(): Update sh_assignok() calls.
src/cmd/ksh93/tests/variables.sh:
- Regress-test subshell leaks for all special variables.
Closes: #122
When exporting variables, ksh exports their attributes (such as
'integer' or 'readonly') in a magic environment variable called
"A__z" (string defined in e_envmarker[] in data/msg.c). Child
shells recognise that variable and restore the attributes.
This little-known feature is risky; the environment cannot
necessarily be trusted and that A__z variable is easy to manipulate
before or between ksh invocations, so you can cause a script's
variables to be of the wrong type, or readonly. Backwards
compatibility requires keeping it, at least for now. But it should
be disabled in the posix mode, as it violates POSIX.
To do this, we have to solve a catch-22 in init.c. We must parse
options to know whether to turn on posix mode; it may be specified
as '-o posix' on the command line. The option parsing loop depends
on an initialised environment[*], while environment initialisation
(i.e., importing attributes) should depend on the posix option.
The catch-22 can be solved because initialising just the values
before option parsing is enough to avoid regressions. Importing the
attributes can be delayed until after option parsing. That involves
basically splitting env_init() into two parts while keeping a local
static state variable between them.
src/cmd/ksh93/sh/init.c:
- env_init():
* Split the function in two stages based on a new
'import_attributes' parameter. Import values in the first
stage; import attributes from A__z in the second (if ever).
Make the 'next' variable static as it keeps a state needed for
the attributes import stage.
* Single point of truth, greppability: don't hardcode "A__z" in
separate character comparisons, but use e_envmarker[].
* Fix an indentation error.
- sh_init(): When initialising the environment (env_init), don't
import the attributes from A__z yet; parse options first, then
import attributes only if posix option is not set.
src/cmd/ksh93/sh/name.c:
- sh_envgen(): Don't export variable attributes to A__z if the
posix option is set.
src/cmd/ksh93/tests/attributes.sh:
- Check that variable attributes aren't imported or exported
if the POSIX option is set.
src/cmd/ksh93/sh.1:
- Update.
This was the last item on the TODO list for -o posix for now.
Closes: #20
[*] If environment initialisation is delayed until after option
parsing, bin/shtests shows various regressions, including:
restricted mode breaks; the locale is not initialised properly
so that multibyte variable names break; $SHLVL breaks.
This commit removes the following standards check on init:
strcmp(astconf("CONFORMANCE",0,0),"standard")==0
This also checks for the POSIXLY_CORRECT variable; the libast
configuration system uses it to set "CONFORMANCE" to "standard",
*but*, only if that parameter wasn't already initialised from the
_AST_FEATURES environment variable (see 'getconf --man').
Problem is, there is a harmful interaction between POSIXLY_CORRECT
and _AST_FEATURES. If the latter exists, it overrides the former.
Not only that, merely querying CONFORMANCE makes astconf create and
export the _AST_FEATURES variable, propagating the current setting
to child ksh processes, which will then ignore POSIXLY_CORRECT.
We could get around this by simply using getenv("POSIXLY_CORRECT").
But then the results may be inconsistent with the AST config state.
The whole thing may not be the best idea anyway. Honouring
POSIXLY_CORRECT at startup introduces a backwards compatibility
issue. Existing scripts or setups may export POSIXLY_CORRECT=y to
put external GNU utilities in standards mode, while still expecting
traditional ksh behaviour from newly initialised shells.
So it's probably better to just get rid of the check. This is not
bash, after all. If ksh is invoked as sh (the POSIX standard
command name), or with '-o posix' on the command line, you get the
standards mode; that ought to be good enough.
src/cmd/ksh93/sh/init.c: sh_init():
- Remove astconf call as per above.
In the SHOPT_BASH code, the -o posix option was given a '\374'
(0xFC, 252) single-letter option character. Reasons unclear. The
'set' builtin doesn't accept it. It can be omitted and the option
still works. And it caused the "$-" expansion (listing active
short-form options) to include that invalid high-bit character if
the -o posix option is active, which is clearly wrong.
src/cmd/ksh93/sh/args.c: optksh[], flagval[]:
- Remove '\374' one-letter option equivalent for SH_POSIX.
src/cmd/ksh93/tests/options.sh:
- Add test verifying that '-o posix' does not affect "$-".
The inclusion of the special parameter expansions ${!} and ${$}
(including the braces) in a here-document caused a syntax error.
Bug reported by @Saikiran-m on Github.
src/cmd/ksh93/data/lexstates.c: sh_lexstate7[]:
- Change the state for ! (33) and $ (36) from S_ERR to 0. State
table 7 is for skipping over ${...}, so this avoids the S_ERR
state being invoked in sh_lex() (lex.c) for these characters
while skipping ${...} in a here-doc.
src/cmd/ksh93/tests/heredoc.sh:
- Test evaluating the braces expansion form for all special
parameters (@ * # ! $ - ? 0) in a here-document.
Fixes: https://github.com/ksh93/ksh/issues/127
Since ksh 93u+m comes bundled with libast 20111111, there's no need
to support older versions, so this is another cleanup opportunity.
src/cmd/ksh93/include/defs.h:
- Throw an #error if AST_VERSION is undefined or < 20111111.
(Note that _AST_VERSION is the same as AST_VERSION, but the
latter is newer and preferred; see src/lib/libast/features/api)
All other changed files:
- Remove legacy code for versions older than the currently used
versions, which are:
_AST_VERSION 20111111
ERROR_VERSION 20100309
GLOB_VERSION 20060717
OPT_VERSION 20070319
SFIO_VERSION 20090915
VMALLOC_VERSION 20110808
SHOPT_ENV is an undocumented compile-time option implementing an
experimental method for handling environment variables, which is
implemented in env.h and env.c. There is no mention in the docs or
Makefile, and no mention in the mailing list archives. It adds no
new functionality, but at first glance it's a clean-looking
interface.
However, unfortunately, it's broken. Compiling with -DSHOPT_ENV
added to CCFLAGS causes bin/shtests to show these regressions:
functions.sh[341]: export not restored name=value function call -- expected 'base', got ''
functions.sh[1274]: Environment variable is not passed to a function
substring.sh[236]: export not restored name=value function call
variables.sh[782]: SHLVL should be 3 not 2
In addition, 'export' stops working on unset variables.
In the 93v- beta this code is still present, unchanged, though 93v-
made lots of incompatible changes. By the time ksh2020 noticed it,
it was no longer compiling, so it probably wasn't compiling in the
93v- beta either. Discussion: https://github.com/att/ast/issues/504
So the experiment was already abandoned by D. Korn and his team.
Meanwhile it was leaving sh/name.c with two versions of several
enviornment-related functions, and it's not clear which one is
actually compiled without doing detective work tracing header files
(most of the code was made conditional on _ENV_H, which is defined
in env.h, which is included by defs.h if SHOPT_ENV is defined).
This actively hinders understanding of the codebase. And any
changes to these functions would need to be implemented twice.
src/cmd/ksh93/include/env.h,
src/cmd/ksh93/sh/env.c:
- Removed.
src/cmd/ksh93/DESIGN,
src/cmd/ksh93/Makefile,
src/cmd/ksh93/Mamfile:
- Update accordingly.
All other changed files:
- Remove deactivated code behind SHOPT_ENV and _ENV_H.
An oops in tests/io.sh (re: c607c48c) wrote temporary files outside
$tmp and into src/cmd/ksh93/tests. Let's fix this properly so it
doesn't happen again.
src/cmd/ksh93/tests/shtests:
- Start each test set in its own temporary directory by default.
src/cmd/ksh93/tests/*.sh:
- Refuse to run if $tmp != $PWD.
- Related cleanups.
On ksh93, 'test -t' is equivalent to 'test -t 1' (and of course
"[ -t ]" is equivalent to "[ -t 1 ]").
This is purely for compatibility with ancient Bourne shell
breakage. No other shell supports this. ksh93 should probably keep
it for backwards compatibility, but it should definitely be
disabled in POSIX mode as it is a violation of the standard; 'test
-t' is an instance of 'test "$string"', which tests if the string
is empty, so it should test if the string '-t' is empty (quod non).
This also replaces the fix for 'test -t 1' in a command
substitution with a better one that avoids forking (re: cafe33f0).
src/cmd/ksh93/sh/parse.c:
- qscan(): If the posix option is active, disable the parser-based
hack that converts a simple "[ -t ]" to "[ -t 1 ]".
src/cmd/ksh93/bltins/test.c:
- e3(): If the posix option is active, disable the part of the
compatibility hack that was used for compound expressions
that end in '-t', e.g. "[ -t 2 -o -t ]".
- test_unop(): Remove the forking fix for "[ -t 1 ]".
src/cmd/ksh93/edit/edit.c:
- tty_check(): This function is used by "[ -t 1 ]" and in other
contexts as well, so a fix here is more comprehensive. Forking
here would cause a segfault, but we don't actually need to. This
adds a fix that simply returns false if we're in a virtual
subshell that is also a command substitution. Since command
substitutions always fork upon redirecting standard output within
them (making them no longer virtual), it is safe to do this.
src/cmd/ksh93/tests/bracket.sh
- Add comprehensive regression tests for test/[/[[ -t variants in
command substitutions, in simple and compound expressions, with
and without redirecting stdout to /dev/tty within the comsub.
- Add tests verifying that -o posix disables the old hack.
- Tweak other tests, including one that globally disabled xtrace.
ksh was enabling POSIX mode on init if it was invoked as any name
that merely started with 'sh' (after parsing initial 'r'). This
included shcomp, which was bad news.
src/cmd/ksh93/sh/init.c: sh_type():
- Check that the 'sh' is at the end of the string by checking
for a final zero byte.
- On Windows (_WINIX, see src/lib/libast/features/common), allow
for a file name extension (sh.exe) by checking for a dot as well.
eeee77ed implemented a POSIX compliance fix that caused a potential
incompatibility with existing ksh scripts; it made the (rarely
used) read/write redirection operator, <>, default to file
descriptor 0 (standard input) as POSIX specified, instead of 1
(standard output) which is traditional ksh93 behaviour. So ksh
scripts needed to change all <> to 1<> to override the new default.
This commit reverts that change, except in the new posix mode.
src/cmd/ksh93/sh/lex.c:
- Make FD for <> default to 0 in POSIX mode, 1 otherwise.
src/cmd/ksh93/tests/io.sh:
- Revert <> regression test changes from 60516872; we no longer
need 1<> instead of <> in ksh code.
If there are file descriptors > 2 opened with 'exec' or 'redirect',
ksh93 has always closed them when invoking another pogram. This is
contrary to POSIX which states:
Utilities other than the special built-ins […] shall be invoked
in a separate environment that consists of the following. The
initial value of these objects shall be the same as that for
the parent shell, except as noted below.
* Open files inherited on invocation of the shell, open files
controlled by the exec special built-in plus any
modifications, and additions specified by any redirections to
the utility
* […]
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12
src/cmd/ksh93/sh/io.c: sh_redirect():
- When flag==2, do not close FDs > 2 if POSIX mode is active.
src/cmd/ksh93/tests/io.sh:
- Regress-test inheriting FD 7 with and without POSIX mode.
src/cmd/ksh93/sh.1:
- Update.
Though the "let" builtin is not itself a POSIX standard command, it
processes standard shell arithmetic, so it should recognise octals
by leading zeros as POSIX requires if the 'posix' option is on.
This overrides the setting of the 'letoctal' option.
Note that none of this applies to the ((...)) arithmetic command,
which has always recognised leading-octal zeros and does not listen
to 'letoctal'. So setting the posix mode makes this consistent.
src/cmd/ksh93/sh/arith.c:
- When running the 'let' builtin, test that both SH_LETOCTAL and
SH_POSIX are off before stripping leading zeros to disable octal
number recognition.
- Cosmetic: fix spurious newline.
src/cmd/ksh93/sh.1:
- Document the change.
src/cmd/ksh93/tests/shtests:
- Make sure to disable posix mode by default for regression tests.
On 16 June there was a call for volunteers to fix the bash
compatibility mode; it has never successfully compiled in 93u+.
Since no one showed up, it is now removed due to lack of interest.
A couple of things are kept, which are now globally enabled:
1. The &>file redirection shorthand (for >file 2>&1). As a matter
of fact, ksh93 already supported this natively, but only while
running rc/profile/login scripts, and it issued a warning. This
makse it globally available and removes the warning, bringing
ksh93 in line with mksh, bash and zsh.
2. The '-o posix' standard compliance option. It is now enabled on
startup if ksh is invoked as 'sh' or if the POSIXLY_CORRECT
variable exists in the environment. To begin with, it disables
the aforementioned &> redirection shorthand. Further compliance
tweaks will be added in subsequent commits. The differences will
be fairly minimal as ksh93 is mostly compliant already.
In all changed files, code was removed that was compiled (more
precisely, failed to compile/link) if the SHOPT_BASH preprocessor
identifier was defined. Below are other changes worth mentioning:
src/cmd/ksh93/sh/bash.c,
src/cmd/ksh93/data/bash_pre_rc.sh:
- Removed.
src/cmd/ksh93/data/lexstates.c,
src/cmd/ksh93/include/shlex.h,
src/cmd/ksh93/sh/lex.c:
- Globally enable &> redirection operator if SH_POSIX not active.
- Remove warning that was issued when &> was used in rc scripts.
src/cmd/ksh93/data/options.c,
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/sh/args.c:
- Keep SH_POSIX option (-o posix).
- Replace SH_TYPE_BASH shell type by SH_TYPE_POSIX.
src/cmd/ksh93/sh/init.c:
- sh_type(): Return SH_TYPE_POSIX shell type if ksh was invoked
as sh (or rsh, restricted sh).
- sh_init(): Enable posix option if the SH_TYPE_POSIX shell type
was detected, or if the CONFORMANCE ast config variable was set
to "standard" (which libast sets on init if POSIXLY_CORRECT
exists in the environment).
src/cmd/ksh93/tests/options.sh,
src/cmd/ksh93/tests/io.sh:
- Replace regression tests for &> and move to io.sh. Since &> is
now for general use, no longer test in an rc script, and don't
check that a warning is issued.
Closes: #9
Progresses: #20
Following a community objection to its removal, the inline 'test'
manual page along with its strange method of invocation is
restored. I've taken the opportunity to correct several mistakes,
add some missing info, do some copy-editing, and document the way
to get these docs in the main (k)sh.1 manual.
Discussion:
https://github.com/ksh93/ksh/commit/fa6a180f#commitcomment-41897553
Did you know that you could get a manual page for the 'test'/'['
builtin command using one of these strange command lines?
test --man --
[ --man -- ]
Neither did I. It's not documented or mentioned anywhere (and this
syntax violates POSIX). So nobody knows about it, which makes that
documentation useless. (The regular --man option doesn't work
because that would break 'test'.) I only found out how to invoke it
when I understood what the uncommented C code handling this does.
The test/[ command's self-documentation is unmaintained since 2003
and somewhat incomplete. It's also mostly redundant with the
documentation on Conditional Expressions in the main (k)sh.1 manual
page. But unlike the latter, this is resident in RAM, wasting
working memory in every shell process.
src/cmd/ksh93/sh.1:
- Add documentation for 'test'/'[' commands (yes, they were not
mentioned in the main manual page until now), describing them
in terms of differences from '[[' and recommending the latter.
src/cmd/ksh93/include/test.h,
src/cmd/ksh93/bltins/test.c,
src/cmd/ksh93/data/testops.c:
- Remove RAM-resident --man doc for test/[ command.
- Remove the bizarre option parsing that allowed invoking it.
Instead of a global 'sh_in_test_builtin' integer flag, it is nicer
to use the mechanism for shell state bits, which was designed for
this sort of thing.
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/sh/defs.c:
- Remove global sh_in_test_builtin integer.
- Define new SH_INTESTCMD state bit.
src/cmd/ksh93/bltins/test.c: _ERROR_exit_b_test(), b_test():
- Use the new state bit.
This removes various blocks of uncommented experimental code that
was disabled using '#if 0' or '#if 1 ... #else' directives. It's
hard or impossible to figure out what the thoughts behind them
might have been, and we can really do without those distractions.
As of 8477d2ce, the mbwide() macro (which tests if we're in a
multibyte locale, i.e. UTF-8) is redefined as a constant 0 if we're
compiling without SHOPT_MULTIBYTE. See src/cmd/ksh93/include/defs.h
The other multibyte macros use mbwide() as well, so they all revert
to the single-byte fallbacks in that case, and the multibyte code
in them is never compiled. See src/lib/libast/include/ast.h
Consequently we can now do a bit of cleanup and get rid of many of
the '#if SHOPT_MULTIBYTE' directives, as the compiler optimiser
will happily remove the multibyte-specific code. This increases the
legibility of the ksh code.
I'm taking the opportunity to fix a few typos and whitespace
formatting glitches as well.
The first of the two multibyte fixes from 8b5f11dc (which was for
using the first character of IFS as an output field separator when
expanding "$*" and similar) had a minor backwards compatibility
problem: if $IFS started with a byte sequence that is not a valid
UTF-8 character, then it treated IFS as empty in UTF-8 locales, so
the fields would be joined without any separator. The expected
behaviour would be for it to fall back to using the first byte of
IFS as it used to (and as bash and zsh do).
The new code handling this was also a bit kludgy and inefficient,
repeating the mbsize() calculation for every byte of the separator
character and for every field joined by the expansion.
src/cmd/ksh93/sh/macro.c: varsub():
- Rewrite code for joining fields for $* in a quoted or scalar
context and $@ in a scalar context, eliminating a confusing 'd'
variable and concentrating the routine in one block.
- When expanding $* with a multibyte separator (first character
of $IFS), only calculate the size in bytes once per expansion.
- If $IFS starts with a byte sequence that represents an invalid
multibyte character, fall back to using the first byte.
src/cmd/ksh93/tests/variables.sh:
- Tweak some regression tests, including one that overwrote $LANG.
- Add test for invalid multibyte character behaviour as per above.
src/cmd/ksh93/sh/path.c: canexecute():
- Close file descriptors inside of the err label. This fixes
a file descriptor leak that occurs when open succeeds but
fstat fails with EIO. The previous code only returned -1
after 'goto err', leaving the opened file descriptor
inaccessible. This bugfix was backported from ksh2020:
https://github.com/att/ast/commit/55cad1d
The package script searches for a good shell to run the build
scripts, preferring a ksh. But it also finds any recently compiled
development version of ksh in arch/*/bin that may be broken, or
have debug code, etc. -- and uses that in preference to anything
else. This is quite capable of breaking the build process.
The way to get around it is to do something like
bin/package make SHELL=/bin/ksh
which is annoying to have to keep doing.
bin/package,
src/cmd/INIT/package.sh:
- When finding a good shell, use the saved user path ($path), not
the current $PATH which includes arch/$HOSTTYPE/bin. Prefix this
temporary path with `getconf PATH`, the system's default path,
so that known-good system shells are found first.
All changed files:
- Put the shell in POSIX mode if it has an '-o posix' option.
- Remove nonsense disabling 'set -x' on bash. It's not broken.
bin/package, src/cmd/INIT/package.sh:
- Add check blocking native zsh mode (e.g., "$path" conflicts).
Using a 'sh -> zsh' symlink works, so recommend that.
- Remove old ksh93 version check for a supposed conflict with
libcmd. It was broken; it would revert to /bin/sh, but on illumos
distributions, /bin/sh is a ksh93 of a version that is supposedly
affected. It builds fine anyway.
- Rewrite checksh() to incorporate the shell compatibility checks
that were previously in two different places in 'package'.
bin/ignore, src/cmd/INIT/ignore.sh,
bin/silent, src/cmd/INIT/silent.sh:
- Change bad check for a full POSIX 'export' command (no, $RANDOM
has nothing to do with that) with a proper feature test.
The shell detection test assumed that any shell that has $RANDOM
and isn't bash must be ksh and have the 'print' built-in command.
This broke iffe on NetBSD 9.0 sh, which has $RANDOM but not 'print'.
src/cmd/INIT/iffe.sh:
- Rewrite shell detection test to test for features actually used
by iffe. "$shell" can now have the values 'bsh' for an ancient
pre-POSIX Bourne shell, 'bash' for bash, 'ksh' for a shell with
'let', 'typeset -u' and 'print' (ksh88, ksh93, mksh, pdksh, zsh),
and 'posix' for another POSIX or POSIX-ish shell with modern
command substitutions and parameter substitutions. (The 'posix'
value currently is not actively checked for anywhere, but avoids
unnecessary use of bsh fallbacks.)
If ksh was compiled with -DSHOPT_REGRESS=1, it would immediately
segfault on init. After fixing that, another segfault remained that
occurred when using the --regress= command line option with an
invalid option-argument.
The __regress__ builtin allows tracing a few things (see
'__regress__ --man' after compiling with -DSHOPT_REGRESS=1, or
usage[] in src/cmd/ksh93/bltins/regress.c). It seems of limited
use, but at least it can be used/tested now.
src/cmd/ksh93/sh/init.c: sh_init():
- Move the call to sh_regress_init() up. The crash on init was
caused by geteuid() being intercepted by regress.c before the
shp->regress (== sh.regress) pointer was initialised.
- The builtin can also be called using a --regress= option-argument
on the ksh command line. Before calling b___regress__() to parse
that, temporarily change error_info.exit so any usage error calls
exit(3) instead of sh_exit(), as the latter assumes a fully
defined shell state and this call is done before the shell is
fully initialised.
The INIT scripts are supposed to be Bourne shell scripts, not
POSIX, so we can't use $( ... ) command substitutions.
bin/package,
src/cmd/INIT/package.sh:
- Change a POSIX command substitution to old-style backticks.
This fixes a hanging bug that could occur on macOS when using the
'read' command to read from a FIFO and encountering end-of-file
without a final newline character. It also makes the 'read' command
perform 15-25% faster on macOS and Linux.
The previous version (ff385e5a) failed on SunOS/Solaris/Illumos
because those systems apparently don't (fully) support the POSIX
standard recv(2) syscall with MSG_PEEK[*], which is the feature
that iffe detects under the 'socket_peek' identifier. On Illumos,
using that methods causes a compilation failure (unknown identifier
MSG_PEEK); on Solaris 11.4, that method causes multiple regressions
in tests/io.sh, suggesting the method compiles but doesn't work at
all. Instead, SunOS/Solaris/Illumos requires the method using
ioctl(2)+I_PEEK and select(2). No other system that ksh currently
builds on requires this method, so it is now only used on
SunOS/Solaris/Illumos.
So far, this version of sfpkrd() has been tested to work correctly
on Linux, macOS, FreeBSD, NetBSD, OpenBSD, HP-UX, Solaris, and
OmniOS (an Illumos distribution).
It still fails to peek on Cygwin, but in the exact same way it
failed before, so that's no loss.
To test, run the 'io' test set: bin/shtests -p io
src/lib/libast/sfio/sfpkrd.c: sfpkrd():
- Remove long-obsolete Mac OS X and Solaris bug workarounds.
- Remove methods that are no longer needed.
On systems with a POSIX compliant recv(2), the only thing that
is required to avoid regressions is the code that was conditional
upon the socket_peek feature test, which tests for the correct
functioning of the recv(2) syscall. This has now been made
mandatory for non-SunOS/Solaris/Illumos systems (using an #error
directive if it is not detected), with the other methods removed.
The result performs 15-25% faster on macOS and Linux while
passing all the regression tests.
On macOS, avoiding the select(2) method fixes the hanging bug.
On SunOS/Solaris/Illumos (the '__sun' identifier), the method
using ioctl(2)+I_PEEK and select(2) (iffe feature IDs:
stream_peek and lib_select) is preserved.
Resolves: https://github.com/ksh93/ksh/issues/118 (again)
[*] https://pubs.opengroup.org/onlinepubs/9699919799/functions/recv.html
