src/lib/libast/misc/optget.c:
- Add screen* (which includes tmux) and dtterm* (CDE terminal) to
the glob pattern deciding whether to use ANSI boldface sequences.
- Don't bother parsing the env var if stderr is not on a terminal.
src/cmd/ksh93/sh.1:
- Extend self-documentation documentation; document how optget(3)
uses the ERROR_OPTIONS env var to control boldface output.
- Tweaks and minor edits.
This fixes a bug in libast optget()'s use of emphasis in the
display of --man(uals) via standard error on a terminal.
Symptom:
$ printf --man 2>&1 | more
(ok; emphasis disabled, no escape codes shown)
$ printf --man
(ok; emphasis correctly displayed)
$ printf --man 2>&1 | more
(whoops; emphasis not disabled; escape codes garble 'more' output)
The problem was that the state.emphasis variable was not
initialised and, when set to one, was never reset again
(except through the use of the --api, --html or --nroff option).
The source code also reveals an undocumented feature: if the
environment variable $ERROR_OPTIONS contains 'noemphasi', emphasis
is forced off, else if it contains 'emphasi', it's forced on.
Other characters (such as the final 's' of emphasis) are ignored.
This was also broken (forcing off didn't work) and is now fixed.
src/lib/libast/misc/optget.c:
- Do not assume that enabling emphasis is forever; re-initialise
the state on every relevant getopts invocation.
- Increase the number of terminals on which emphasis is displayed
using ANSI escape codes. (This is a hack and we should ask the OS
for the correct codes, but never mind -- ANSI is now universal.)
The >;word and <>;word redirection operators cannot be used with
the 'exec' builtin, but the 'redirect' builtin (which used to be
an alias of 'command exec') permitted them. However, they do not
have the documented effect of the added ';'. So this commit blocks
those operators for 'redirect' as they are blocked for 'exec'.
It also tweaks redirect's error message if a non-redirection
argument is encountered.
src/cmd/ksh93/sh/parse.c: simple():
- Set the lexp->inexec flag for SYSREDIR (redirect) as well as
SYSEXEC (exec). This flag is checked for in sh_lex() (lex.c) to
throw a syntax error if one of these two operators is used.
src/cmd/ksh93/sh.1, src/cmd/ksh93/data/builtins.c:
- Documentation tweaks.
src/cmd/ksh93/sh/xec.c, src/cmd/ksh93/bltins/misc.c:
- When 'redirect' gives an 'incorrect syntax' (e_badsyntax) error
message, include the first word that was found not to be a valid
redirection. This is simply the first argument, as redirections
are removed from the arguments list.
src/cmd/ksh93/tests/io.sh:
- Update test to reflect new error message format.
Now that the Make Abstract Machine files are maintained manually
and not generated automatically, unused variables are an annoying
distraction -- and there are many.
But the language/format is very simple and very parseable using
shell, awk, etc. -- so this was easy to automate. All variables are
declared with 'setv' and they are used if an expansion of the form
${varname} exists (the braces are mandatory in Mamfiles).
bin/Mamfile_rm_unused_vars:
- Added for reference and future use.
src/*/*/Mamfile:
- Remove all unused 'setv' variable declarations.
Permanent redirections of that form broke in subshells when used
with the 'redirect' command, because I had overlooked one instance
where the new 'redirect' builtin needs to match the behaviour of
the 'exec' builtin.
src/cmd/ksh93/tests/io.sh: sh_exec():
- Do not restore file descriptors in (virtual) subshells for
'redirect' just as this isn't done for 'exec'.
src/cmd/ksh93/tests/io.sh:
- Add regression test for this bug.
- Complete the test for f9427909 which I committed prematurely.
Fixes: https://github.com/ksh93/ksh/issues/167
This is some nonsense: redirections that store a file descriptor
greater than 9 in a variable, like {var}<&2 and the like, stopped
working if brace expansion was turned off. '{var}' is not a brace
expansion as it doesn't contain ',' or '..'; something like 'echo
{var}' is always output unexpanded. And redirections and brace
expansion are two completely unrelated things. It wasn't documented
that these redirections require the -B/braceexpand option, either.
src/cmd/ksh93/sh/lex.c: sh_lex():
- Remove incorrect check for braceexpand option before processing
redirections of this form.
src/cmd/ksh93/COMPATIBILITY:
- Insert a brief item mentioning this.
src/cmd/ksh93/sh.1:
- Correction: these redirections do not yield a file descriptor >
10, but > 9, a.k.a. >= 10.
- Add a brief example showing how these redirections can be used.
src/cmd/ksh93/tests/io.sh:
- Add a quick regression test.
src/cmd/ksh93/sh/args.c: sh_argprocsub():
- Fix compiler warnings with SHOPT_DEVFD on by including "io.h".
- Without SHOPT_DEVFD, the FIFO code didn't consider that libast's
pathtemp(3) may also fail and return null. Add a check for this.
It was trivial to crash ksh by making an autoloaded function
definition file autoload itself, causing a stack overflow due to
infinite recursion. This commit adds loop detection that stops a
function that is being autoloaded from autoloading itself either
directly or indirectly, without removing the ability of autoloaded
function definition files to autoload other functions.
src/cmd/ksh93/sh/path.c: funload():
- Detect loops by checking if the path of a function to be
autoloaded was already added to a new internal static tree,
and if not, adding it while the function is being loaded.
src/cmd/ksh93/tests/path.sh:
- Add regression test.
- Tweak a couple of others to be freeze- and crash-proof.
NEWS:
- Add this fix + a forgotten entry for the previous fix (6f3b23e6).
Fixes: https://github.com/ksh93/ksh/issues/136
Reproducer from @Saikiran-m:
| ~# sh -c `perl -e 'print "a"x100000'`
| genunix: NOTICE: core_log: sh[1221] core dumped: /var/cores/core.sh.0.1602153496
| Memory fault(coredump)
The crash was in trying to decide whether the name was suitable for
autoloading as a function on $FPATH. This calls strmatch() to check
the name against a regex for valid function name. But the libast
regex code is not designed optimally and uses too much recursion,
limiting the length of the strings it's able to cope with.
src/cmd/ksh93/sh/path.c: path_search():
- Before calling strmatch(), check that the name is shorter than
256 bytes. The maximum length of file names on Linux and macOS is
255 bytes, so an autoload function can't have a name longer than
that anyway.
src/cmd/ksh93/tests/path.sh:
- Add test for this bug.
- Tweak 'command -x' test to not leave a hanging process on Ctrl+C.
Fixes: https://github.com/ksh93/ksh/issues/144
Well, that commit was based on a silly oversight: of course it's
necessary to pass ${KSH_RELFLAGS} to the feature tests too as they
use this flag to determine whether to enable or disable vmalloc.
On further analysis I think the annoying warnings can be solved in
a different way. Quotes (single or double) in 'exec -' commands
don't seem to be special to mamake at all; it looks like they are
passed on to the shell as is. So Mamfile variables are expanded and
the expansions backslash-escaped the same way regardless of quotes.
Which means we can make the shell remove the unwanted level of
backslashes by using double instead of single quotes.
src/*/*/Mamfile:
- On iffe commands, restore ${KSH_RELFLAGS}, using double quotes to
group the compiler command as one argument to iffe.
This reverts an OpenSUSE patch ("libast/comp/conf.sh: apply limits
detection fixes for Linux"). It broke the build on Alpine Linux
with the musl C library (see also e245856f).
This time it was failing on a 64-bit Debian Linux system with very
few and short environment variables. Sigh.
src/cmd/ksh93/sh/path.c:
- Combine the strategy from 63979488 with that of 8f5235a5.
That fix turned out to be insufficient as NixOS has huge
environment variable lists because (due to each software package
being installed in its own directory tree) it has to keep dozens
of directories in variables like XDG_CONFIG_DIRS and others.
The 'command -x' regression test was failing on NixOS.
src/cmd/ksh93/sh/path.c:
- Different strategy. Leave twice the size of the existing
environment free.
Hopefully this will deal with ksh crashing in macOS Terminal.app
once and for all. Trigger: press Command-F to open the find bar,
then press Esc to close it, then press Esc again. Result: crash
somewhere random in the job control code.
Turns out macOS Terminal.app apparently (and wrongly) sends <Esc>
followed by <Ctrl+L> to the terminal, which ksh takes as a sequence
for clearing the screen. The related crash ultimately traced back
to the code for that in emacs.c. The other crash was in the code
for double-ESC file name completion.
This commit also fixes a non-robust invocation of the 'tput'
command by using the direct path found in $(getconf PATH).
src/cmd/ksh93/features/cmds:
- Remove unused tests for the presence of commands
(newgrp,test,id,wc,cut,logname,pfexec).
- Replace 'cmd tput' test by 'pth tput' which will find its path
in $(getconf PATH) and store that path as the macro value.
- Add two tests to determine if 'tput' supports terminfo and/or
termcap codes. (FreeBSD still requires old termcap codes.)
src/cmd/ksh93/edit/emacs.c: escape():
- Fix a crash in the code for double-ESC completion. Check if the
cursor is on a non-zero position; this caused a bus error
(invalid address access) in the subsequent ed_expand call.
- For <Esc><Ctrl+L> (clear screen), fix the strange crash in macOS
Terminal by not using sh_trap() to invoke "tput clear", which
causes ksh itself to invoke that command. ksh apparently doesn't
cope with doing this while SIGWINCH (window size change signal)
is sent by Terminal. The fix is to just use the C standard
system(3) function to invoke tput. This invokes tput via /bin/sh,
but what the hey. (Note that ksh also ran any function or alias
called 'tput' instead of the real command, and that is now also
fixed.)
- Use the new _pth_tput test result to invoke tput with the
hardcoded default system path, increasing robustness.
src/cmd/ksh93/edit/edit.c: ed_setup():
- Use the new _pth_tput test result to invoke tput with the
hardcoded default system path, increasing robustness.
- When getting the escape code for "cursor up", use the new
_tput_terminfo and _tput_termcap test results to determine which
kind of command code to send. This fixes it on FreeBSD.
src/cmd/INIT/iffe.sh:
- Fix "standard system directories" for the cmd test, which were
hardcoded as bin, /etc, /usr/bin, /usr/etc, /usr/ucb. That's both
unportable and antiquated. Replace this with the path output by
'getconf PATH'.
- Add fixes from modernish for 'getconf PATH' output to compensate
for bugs/shortcomigns in NixOS and AIX. Source:
https://github.com/modernish/modernish/blob/9e4bf5eb/lib/modernish/aux/defpath.sh
Ref.: https://github.com/NixOS/nixpkgs/issues/65512
src/lib/libast/comp/conf.tab: PATH:
- Add the NixOS and AIX default path fixes here too; this fixes
'command -p' and the builtin 'getconf PATH' on these systems.
bin/package, src/cmd/INIT/package.sh:
- Re-support being launched with just the command name 'package' in
the command line (if the 'package' command is in $PATH). At least
one other script in the build system does this. (re: 6cc2f6a0)
- Go back three levels (../../..) if we were invoked from
arch/*/bin/package, otherwise we won't find src/cmd/ksh93/SHOPT.sh.
Something similar was previously done in 07cc71b8 from a Debian
patch, and eventually reverted; it redefined the ast atomic
functions asoincint() and asodecint() to be gcc-specific. This
imports the upstream version from the ksh 93v- beta instead.
This commit is based on an OpenSUSE patch:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-joblock.dif
src/cmd/ksh93/include/jobs.h:
- Replace job locking mechanism with the 93v- version which uses
the atomic libast functions asoincint(), asogetint() and
asodecint(). See: src/lib/libast/man/aso.3
src/cmd/ksh93/sh/jobs.c: job_subsave():
- Revert gcc optimiser bug workaround from c258a04f.
It should now be unnecessary.
I got one intermittent regression test failure due to 'argument
list too long' on a Debian x86_64 system.
src/cmd/ksh93/sh/path.c: path_xargs():
- Leave extra argument space for systems that need extra bytes:
1KiB per extra byte, with a minimum of 2KiB (the old value).
From an OpenSUSE patch:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-pathtemp.dif
See src/lib/libast/man/path.3 for pathtemp()
and src/lib/libast/man/sfio.3 for sftmp()
src/lib/libast/path/pathtemp.c:
- Error check fix: add an access check wrapper function that checks
if a path was given and if there is enough free space on the
device, setting errno appropriately in case of trouble.
src/lib/libast/sfio/sftmp.c:
- On Linux, use the /dev/shm shared memory objects for the new
temporary file descriptor -- that is, do not access HD or SSD but
only the memory based tmpfs of the POSIX SHM.
This fixes the function that sets ${.sh.match}. Patch from OpenSUSE:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-limit-name-len.dif
src/cmd/ksh93/sh/init.c: sh_setmatch():
- Fix node size calculation, possibly preventing data corruption.
src/cmd/ksh93/include/ulimit.h: Limit_t:
- Defining the 'name' struct member as 'char name[16]' makes
no sense as the name is being initialised statically in
data/limits.c; just make it a 'char *name' pointer.
This fixes the following regressions marked TODO in attributes.sh:
$ typeset -L 13 bar; readonly bar; typeset -p bar
typeset -r -L 0 foo # exp.: typeset -r -L 13 foo
$ typeset -R 13 bar; readonly bar; typeset -p bar
typeset -r -R 0 bar # exp.: typeset -r -R 13 bar
$ typeset -Z 13 baz; readonly baz; typeset -p baz
typeset -r -Z 0 -R 0 baz # exp.: typeset -r Z 13 -R 13 baz
I've discovered that these were briefly fixed between fdb9781e (Red
Hat patch for typeset -xu/-xl) and 95fe07d8 (reversal of patch,
different -xu/-xl fix, but reintroduced these regressions).
src/cmd/ksh93/sh/name.c: nv_newattr():
- Replace check from 95fe07d8 with a new one that combines its
approach with that of fdb9781e: do not change size (and hence
return early) if NV_RDONLY and/or NV_EXPORT are the only
attributes that are changing.
src/cmd/ksh93/tests/attributes.sh:
- Enable the TODO regression tests.
This commit corrects how shortint was being applied to various
possible typeset variables in error. The short integer option
modifier 'typeset -s' should only be able to be applied if the
the variable is also an integer. Several issues were resolved
with this fix:
- 'typeset -s': created a short integer having an invalid base
of zero. 'typeset -s foo' created 'typeset -s -i 0 foo=0' and
now will result in an empty string.
- 'typeset -sL': previously resulted in a segmentation fault.
The following are the various incorrect 'typeset' instances
that have been fixed:
$ 'export foo; typeset -s foo; readonly foo; typeset -p foo'
(before) typeset -x -r -s -i 0 foo=0
( after) typeset -x -r foo
$ 'typeset -sL foo=1*2; typeset -p foo'
(before) Segmentation fault (core dumped)
( after) typeset -L 3 foo='1*2'
$ 'typeset -sR foo=1*2; typeset -p foo'
(before) typeset -s -i foo=2
( after) typeset -R 3 foo='1*2'
$ 'typeset -sZ foo=1*2; typeset -p foo'
(before) typeset -F 0 foo=2
( after) typeset -Z 3 -R 3 foo='1*2'
src/cmd/ksh93/bltins/typeset.c: b_typeset():
- Add conditional check within the 's' option to only
apply NV_SHORT as well as remove any NV_LONG flag
if NV_INTEGER flag was set.
- Relocate shortint conditional logic to the 'i' option.
src/cmd/ksh93/tests/attributes.sh:
- Adjust regression tests for '-s' and add '-si' check.
This fixes part of https://github.com/ksh93/ksh/issues/87:
Scalar arrays (-a) and associative arrays (-A) of a type created by
'enum' did not consistently block values not specified by the enum
type, yielding corrupted results.
An expansion of type "${array[@]}" yielded random numbers instead
of values for associative arrays of a type created by 'enum'.
This does not yet fix another problem: ${array[@]} does not yield
all values for associative enum arrays.
src/cmd/ksh93/bltins/enum.c: put_enum():
- Always throw an error if the value is not in the list of possible
values for an enum type. Remove incorrect check for the NV_NOFREE
flag. Whatever that was meant to accomplish, I've no idea.
src/cmd/ksh93/sh/array.c: nv_arraysettype():
- Instead of sh_eval()ing a shell assignment, use nv_putval()
directly. Also use the stack (see src/lib/libast/man/stk.3)
instead of malloc to save the value; it's faster and will be
auto-freed at some point. This shortens the function and makes it
faster by not entering into a whole new shell context -- which
also fixes another problem: the error message from put_enum()
didn't cause the shell to exit for indexed enum arrays.
src/cmd/ksh93/sh/name.c: nv_setlist():
- Apply a patch from David Korn that correctly sets the data type
for associative arrays, fixing the ${array[@]} expansion yielding
random numbers. Thanks to @JohnoKing for the pointer.
https://github.com/ksh93/ksh/issues/87#issuecomment-662613887https://www.mail-archive.com/ast-developers@lists.research.att.com/msg00697.html
src/cmd/ksh93/tests/enum.sh:
- Add tests checking that invalid values are correctly blocked for
indexed and associative arrays of an enum type.
Makes progress on: https://github.com/ksh93/ksh/issues/87
Turns out the assumption I was operating on, that Linux and macOS
align arguments on 32 or 64 bit boundaries, is incorrect -- they
just need some extra bytes per argument. So we can use a bit more
of the arguments buffer on these systems than I thought.
src/cmd/ksh93/features/externs:
- Change the feature test to simply detect the # of extra bytes per
argument needed. On *BSD and commercial Unices, ARG_EXTRA_BYTES
shows as zero; on Linux and macOS (64-bit), this yields 8. On
Linux (32-bit), this yields 4.
src/cmd/ksh93/sh/path.c: path_xargs():
- Do not try to calculate alignment, just add ARG_EXTRA_BYTES to
each argument.
- Also add this when substracting the length of environment
variables and leading and trailing static command arguments.
src/cmd/ksh93/tests/path.sh:
- Test command -v/-V with -x.
- Add a robust regression test for command -x.
src/cmd/ksh93/data/builtins.c, src/cmd/ksh93/sh.1:
- Tweak docs. Glob patterns also expand to multiple words.
iffe feature test that add a -D_LARGEFILE64_SOURCE compiler flag to
detect the presence of 64-bit types like off64_t are very
incorrect; they always find the type even if the rest of the source
is not compiled with that flag, causing an inconsistent compilation
environment. This was the cause of mysterious failures to compile
some feature tests on Linux i386 -- it tried to use an off64_t type
that was wrongly detected.
A flag like -D_LARGEFILE64_SOURCE needs to be added to the compiler
flags consistently so it is used for compiling all files and tests.
src/lib/libast/features/dirent,
src/lib/libast/features/fs,
src/lib/libast/features/lib,
src/lib/libast/features/mmap,
src/cmd/ksh93/features/rlimits:
- Remove the -D_LARGEFILE64_SOURCE flag from all the tests that
used it.
- Fix some preprocessor directives for compiling without
_LARGEFILE64_SOURCE. We cannot rely on the result of the _lib_*64
tests because those functions are still found in glibc even if
_LARGEFILE64_SOURCE is not defined; we have to check for the
existence of the type definitions before using them.
src/cmd/INIT/cc.linux.i386,
src/cmd/INIT/cc.linux.i386-icc:
- Add/update compiler wrappers to hardcode -D_LARGEFILE64_SOURCE
in the flags for the default compiler. If it is overriden with
$CC, then it needs to be added manually if desired.
This test depends on the correctness of the locale data provided
by the OS, and some installations are broken. Failures of this test
most likely do not represent a bug in ksh or libast.
This takes another small step towards disentangling the build
system from the old AT&T environment. The USAGE_LICENSE macros with
author and copyright information, which was formerly generated
dynamically for each file from a database, are eliminated and the
copyright/author information is instead inserted into the AST
getopt usage strings directly.
Repetitive license/copyright information is also removed from the
getopt strings in the builtin commands (src/lib/libcmd/*.c and
src/cmd/ksh93/data/builtins.c). There's no need to include 55
identical license/copyright strings in the ksh binary; one (in the
main ksh getopt string, shown by ksh --man) ought to be enough!
This makes the ksh binary about 10k smaller.
It does mean that something like 'enum --author', 'typeset
--license' or 'shift --copyright' will now not show those notices
for those builtins, but I doubt anyone will care.
This commit fixes 'command -x' to adapt to OS limitations with
regards to data alignment in the arguments list. A feature test is
added that detects if the OS aligns the argument on 32-bit or
64-bit boundaries or not at all, allowing 'command -x' to avoid
E2BIG errors while maximising efficiency.
Also, as of now, 'command -x' is a way to bypass built-ins and
run/query an external command. Built-ins do not limit the length of
their argument list, so '-x' never made sense to use for them. And
because '-x' hangs on Linux and macOS on every ksh93 release
version to date (see acf84e96), few use it, so there is little
reason not to make this change.
Finally, this fixes a longstanding bug that caused the minimum exit
status of 'command -x' to be 1 if a command with many arguments was
divided into several command invocations. This is done by replacing
broken flaggery with a new SH_XARG state flag bit.
src/cmd/ksh93/features/externs:
- Add new C feature test detecting byte alignment in args list.
The test writes a #define ARG_ALIGN_BYTES with the amount of
bytes the OS aligns arguments to, or zero for no alignment.
src/cmd/ksh93/include/defs.h:
- Add new SH_XARG state bit indicating 'command -x' is active.
src/cmd/ksh93/sh/path.c: path_xargs():
- Leave extra 2k in the args buffer instead of 1k, just to be sure;
some commands add large environment variables these days.
- Fix a bug in subtracting the length of existing arguments and
environment variables. 'size -= strlen(cp)-1;' subtracts one less
than the size of cp, which makes no sense; what is necessary is
to substract the length plus one to account for the terminating
zero byte, i.e.: 'size -= strlen(cp)+1'.
- Use the ARG_ALIGN_BYTES feature test result to match the OS's
data alignment requirements.
- path_spawn(): E2BIG: Change to checking SH_XARG state bit.
src/cmd/ksh93/bltins/whence.c: b_command():
- Allow combining -x with -p, -v and -V with the expected results
by setting P_FLAG to act like 'whence -p'. E.g., as of now,
command -xv printf
is equivalent to
whence -p printf
but note that 'whence' has no equivalent of 'command -pvx printf'
which searches $(getconf PATH) for a command.
- When -x will run a command, now set the new SH_XARG state flag.
src/cmd/ksh93/sh/xec.c: sh_exec():
- Change to using the new SH_XARG state bit.
- Skip the check for built-ins if SH_XARG is active, so that
'command -x' now always runs an external command.
src/lib/libcmd/date.c, src/lib/libcmd/uname.c:
- These path-bound builtins sometimes need to run the external
system command by the same name, but they did that by hardcoding
an unportable direct path. Now that 'command -x' runs an external
command, change this to using 'command -px' to guarantee using
the known-good external system utility in the default PATH.
- In date.c, fix the format string passed to 'command -px date'
when setting the date; it was only compatible with BSD systems.
Use the POSIX variant on non-BSD systems.
Three OpenSUSE patches from:
https://build.opensuse.org/package/show/shells/ksh
As usual, the relevant bug is not currently public:
https://bugzilla.opensuse.org/show_bug.cgi?id=844071
src/cmd/ksh93/sh/xec.c: sh_debug()/sh_exec():
- Fix stk restoration. [bnc#844071]
src/lib/libast/misc/stk.c:
- Fix stk aliasing code. [bnc#844071]
(ksh93-stkalias.dif)
- Make a unknown location fatal in stkset() so that we get a core
dump right away instead of later in an unrelated part of code.
(ksh93-stkset-abort.dif)
src/lib/libast/man/stk.3,
src/lib/libast/man/stak.3:
- Update manual with new stkset() behaviour. (93u+m addition)
(Note that stak is implemented as macros that translate to stk)
