This fixes a hanging bug that could occur on macOS when using the
'read' command to read from a FIFO and encountering end-of-file
without a final newline character. It also makes the 'read' command
perform 15-25% faster on macOS and Linux and maybe other systems.
src/lib/libast/sfio/sfpkrd.c: sfpkrd():
- Get rid of the optional stuff that uses the poll(2) or select(2)
syscalls. The only thing that is required to avoid regressions is
the code that was conditional upon the socket_peek feature test,
which tests for the correct functioning of the recv(2) syscall.
This has now been made mandatory. The rest now uses what was
previously a fallback in plain C, resulting in a function that is
not only more readable, but actually faster than the syscalls.
Resolves: https://github.com/ksh93/ksh/issues/118
CDE <https://cdesktopenv.sf.net/> developer Chase writes, re dtksh:
| Everything is now completely working, and we are almost ready to
| add ksh93 as a submodule, but I have one last commit to get rid
| of some warnings we are facing. nval.h has some of these
| "compatiblity redefines" that are causing issues whenever we
| include it (warnings about redefining values) [...].
src/cmd/ksh93/include/nval.h:
- Replace ancient compatibility redefines by an unconditional
'#include <hash.h>'; ksh works fine with the "new" hash library.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
If a command's path was previously added to the hash table as a
'tracked alias', then the hash table entry was used, bypassing
the default utility path search activated by 'command -p'.
'command -p' activates a SH_DEFPATH shell state. The bug was caused
by a failure to check for this state before using the hash table.
This check needs to be added in four places.
src/cmd/ksh93/sh/path.c,
src/cmd/ksh93/sh/xec.c:
- path_search(), path_spawn(), sh_exec(), sh_ntfork(): Only consult
the hash table, which is shp->track_tree, if the SH_DEFPATH shell
state is not active.
src/cmd/ksh93/tests/path.sh:
- Add regress tests checking that 'command -p' and 'command -p -v'
still search in the default path if a hash table entry exists for
the command searched.
'command -x' (basically builtin xargs for 'command') worked for
long argument lists on *BSD and HP-UX, but not on macOS and Linux,
where it reliably entered into an infinite loop.
The problem was that it assumed that every byte of the environment
space can be used for arguments, without accounting for alignment
that some OSs do. MacOS seems to be the most wasteful one: it
aligns on 16-byte boundaries and requires some extra bytes per
argument as well.
src/cmd/ksh93/sh/path.c:
- path_xargs(): When calculating how much space to subtract per
argument, add 16 extra bytes to the length of each argument, then
align the result on 16-byte boundaries. The extra 16 bytes is
more than even macOS needs, but hopefully it is future-proof.
- path_spawn(): If path_xargs() does fail, do not enter a retry
loop (which always becomes an infinite loop if the argument list
exceeds OS limitations), but abort with an error message.
Four libast hash functions/macros (which ksh93 doesn't actually use)
were overridden with the following comment:
/*
* These following are for binary compatibility with the old hash library
* They will be removed someday
*/
This has been there for decades, and I just received word that they
cause problems for the dtksh (CDE) developers as dtksh does call
hashlook().
src/cmd/ksh93/sh/name.c:
- Remove 'hashscope', 'hashfree', 'hashname' and 'hashlook'
compatibility overrides.
A memory leak occurred upon leaving a virtual subshell if a
function was defined within it. If this was done more than 32766
(= 2^15-2 = the 'short' max value - 1) times, the shell crashed.
Discussion and reproducer: https://github.com/ksh93/ksh/issues/114
src/cmd/ksh93/sh/subshell.c: table_unset():
- A subshell-defined function was never freed because a broken
check for autoloaded functions (which must not be freed[*]). It
looked for an initial '/' in the canonical path of the script
file that defined the function, but that path is also stored for
regular functions. Now use a check that executes nv_search() in
fpathdict, the same method used in _nv_unset() in name.c for a
regular function unset.
src/cmd/ksh93/bltins/misc.c: b_dot_cmd():
- Fix an additional memory leak introduced in bd88cc7f, that caused
POSIX functions (which are run with b_dot_cmd() like dot scripts)
to leak extra. This fix avoids both the crash fixed there and the
memory leak by introducing a 'tofree' variable remembering the
filename to free. Thanks to Johnothan King for the patch.
src/lib/libast/include/stk.h,
src/lib/libast/misc/stk.c,
src/lib/libast/man/stk.3,
src/lib/libast/man/stak.3:
- Make the stack more resilient by extending the stack reference
counter 'stkref' from (signed) short to unsigned int. On modern
systems with 32-bit ints, this extends the maximum number of
elements on a stack from 2^15-1==32767 to 2^32-1==4294967295.
The ref counter can never be negative, so there is no reason for
signedness. sizeof(int) is defined as the size of a single CPU
word, so this should not affect performance at all.
On a 16-bit system (not that ksh still compiles there), this
doubles the max number of entries to 2^16-1=65535.
src/cmd/ksh93/tests/leaks.sh:
- Add leak regression tests for ksh functions, POSIX functions, dot
scripts run with '.', and dot scripts run with 'source'.
src/cmd/ksh93/tests/path.sh:
- Add an output builtin with a redirect to an autoloaded function
so that a crash[*] is triggered if the check for an autoloaded
function is ever removed from table_unset(), as was done in ksh
93v- (which crashed).
[*] Freeing autoloaded functions after leaving a virtual subshell
causes a crashing bug: https://github.com/att/ast/issues/803
Co-authored-by: Johnothan King <johnothanking@protonmail.com>
Fixes: https://github.com/ksh93/ksh/issues/114
The fix was incomplete because some tests have to unset HISTFILE,
which reverted them to using ~/.sh_history by default.
src/cmd/ksh93/tests/shtests:
- Instead of setting HISTFILE, set HOME to the temporary directory
$tmp, so nothing will write to the real user directory and the
default history file is $tmp/.sh_history.
src/cmd/ksh93/tests/attributes.sh:
- Restore HISTFILE after a test that requires setting HISTFILE=foo.
src/cmd/ksh93/bltins/print.c:
- If in UTF-8 locale, only bother to check for unreserved char if
the character is ASCII (< 128), and write unreserved chars with
a simple stakputc().
Co-authored-by: Martijn Dekker <martijn@inlv.org>
An intermittent crash occurred after running many thousands of
virtual/non-forked subshells. One reproducer is a crash in the
shbench fibonacci.ksh test, as documented here:
https://github.com/ksh-community/shbench/blob/f3d9e134/bench/fibonacci.ksh#L4-L10
The apparent cause was the signed and insufficiently large 'short'
data type of 'curenv' and related variables which wrapped around to
a negative number when overflowing. These IDs are necessary for the
'wait' builtin to obtain the exit status from a background job.
This fix is inspired by a patch based on ksh 93v-:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-longenv.dif?expand=1https://src.fedoraproject.org/rpms/ksh/blob/f24/f/ksh-20130628-longer.patch
However, we change the type to 'unsigned int' instead of 'long'. On
all remotely modern systems, ints are 32-bit values, and using this
type avoids a performance degradation on 32-bit sytems. Making them
unsigned prevents an overflow to negative values.
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/include/jobs.h,
src/cmd/ksh93/include/nval.h,
src/cmd/ksh93/include/shell.h:
- Change the types of the static global 'subenv' and the subshell
structure members 'curenv', 'jobenv', 'subenv', 'p_env' and
'subshell' to one consistent type, unsigned int.
src/cmd/ksh93/sh/jobs.c,
src/cmd/ksh93/sh/macro.c:
src/cmd/ksh93/sh/name.c:
src/cmd/ksh93/sh/nvtype.c,
src/cmd/ksh93/sh/subshell.c:
- Updates to match new variable types.
src/cmd/ksh93/tests/subshell.sh:
- Show wrong exit status in message on failure of 'wait' builtin.
As of aa4669ad, astconf("PATH") is implemented as a hardcoded AST
configuration variable that always has a value, instead of one that
falls back on the OS. Its value is now obtained from the OS (with a
fallback) at configure time and not at runtime. This means that any
fallback for astconf("PATH") is now never used.
src/cmd/ksh93/data/msg.c,
src/cmd/ksh93/include/shell.h:
- Remove e_defpath[]. (The path "/bin:/usr/bin:" made no sense as a
default path anyway, as the final empty element is wrong: default
utilities should never be sought in the current working dir.)
src/cmd/ksh93/sh/path.c,
src/lib/libast/path/pathbin.c:
- abort() if astconf("PATH") returns null.
src/lib/libast/comp/conf.tab: PATH:
- If no 'getconf' utility can be found, use a fallback path that
finds more utilities by also searching in 'sbin' directories.
On some systems, this is needed to find chown(1).
src/cmd/ksh93/sh.1:
- Update doc re default path.
Using the bin/shtests -l/--locale option to run the regression
tests in your own locale broke the tests if you're in a locale that
uses ',' as the radix point, like my nl_NL.UTF-8, unless
LC_NUMERIC=C was exported manually. Let's automate that fix.
src/cmd/ksh93/tests/shtests: --locale:
- If LC_ALL was set, copy it to LANG and unset all LC_* vars.
This allows overriding the radix point with LC_NUMERIC if needed.
- If '1.0' is not a valid shell arithmetic expression, export
LC_NUMERIC=C to fix it.
The entity is not valid in XML, only in HTML. Since we must
be compatible with both, it can't be used. Thanks to Andras Farkas
for the bug report.
In addition, the generation of numeric entities for unprintable
characters was only valid while processing UTF-8 text while in a
UTF-8 locale. In all other conditions it produced invalid results.
This is not worth trying to fix.
Discussion:
https://groups.google.com/d/msgid/korn-shell/CAA0nTRta%3DPbOYduyBv%3DXCzumTcUCU8Lki%3DQQf2O8Erk2BFvO1g%40mail.gmail.com
src/cmd/ksh93/bltins/print.c:
- Remove conversion to entity.
- Remove conversion of non-graph characters to numeric entities.
Convert only the 5 semantically meaningful characters: < > & " '
src/cmd/ksh93/include/defs.h,
src/cmd/ksh93/sh/string.c:
- We don't need sh_isprint() in print.c anymore, so turn it back
into a static function.
src/cmd/ksh93/tests/builtins.sh:
- Update and trim regression tests.
The current fix appears to be only partially successful in
eliminating the intermittent crash, and also breaks '-o notify'
during the 60-second $TMOUT grace period. This replaces it.
The root cause appears to be that the state of job control becomes
somehow inconsistent when running external commands in a command
substitution expanded from the $PS1 prompt. The job_unpost() or
(sometimes) the job_list() function intermittently crash. These are
called if the SH_TTYWAIT state is active:
https://github.com/ksh93/ksh/blob/88e8fa67/src/cmd/ksh93/sh/jobs.c#L463-L469
Temporarily deactivating the SSH_TTYWAIT state while expanding
PS{1..4} prompts appears to fix the problem reliably.
It is quite possible that this fix merely masks a bug in the job
control system, but testing has shown that it stops ksh crashing
without side effects, so I'm calling it good for now.
Thanks to Marc Wilson for many hours of persistent testing.
src/cmd/ksh93/sh/jobs.c:
- Revert changes made in 33858689 and e805c7d9.
src/cmd/ksh93/sh/io.c: io_prompt():
- Save SH_TTYWAIT state and turn it off while expanding prompts.
Resolves: https://github.com/ksh93/ksh/issues/103
Resolves: https://github.com/ksh93/ksh/issues/112
This applies a number of fixes to the printf formatting directives
%H and %#H (as well as their equivalents %(html)q and %(url)q):
1. Both formatters have been made multibyte/UTF-8 aware, and no
longer delete multibyte characters. Invalid UTF-8 byte sequences
are rendered as ASCII question marks.
2. %H no longer wrongly encodes spaces as non-breaking spaces
( ) and instead correctly encodes the UTF-8 non-breaking
space as such.
3. %H now converts the single quote (') to '%#39;' instead of
''' which is not a valid entity in all HTML versions.
4. %#H failed to encode some reserved characters (e.g. '?') while
encoding some unreserved ones (e.g. '~'). It now percent-encodes
all characters except those 'unreserved' as per RFC3986 (ASCII
alphanumeric plus -._~).
Prior discussion:
https://groups.google.com/d/msgid/korn-shell/ce8d1467-4a6d-883b-45ad-fc3c7b90e681%40inlv.org
src/cmd/ksh93/include/defs.h:
src/cmd/ksh93/sh/string.c:
- defs.h: If compiling without SHOPT_MULTIBYTE, redefine the
mbwide() macro (which tests if we're in a multibyte locale) as 0.
This lets the compiler optimiser do the work that would otherwise
require a lot of tedious '#if SHOPT_MULTIBYTE' directives.
- string.c: Remove some now-unneeded '#if SHOPT_MULTIBYTE' stuff.
- defs.h, string.c: Rename is_invisible() to sh_isprint(), invert
the boolean return value, and make it an extern for use in
fmthtml() -- see below. If compiling without SHOPT_MULTIBYTE,
simply #define sh_isprint() as equivalent to isprint(3).
- defs.h: Add URI_RFC3986_UNRESERVED macro for fmthtml() containing
the characters "unreserved" for purposes of URI percent-encoding.
src/cmd/ksh93/bltins/print.c: fmthtml():
- Remove kludge that skipped all multibyte characters (!).
- Complete rewrite to implement fixes described above.
- Don't bother with '#if SHOPT_MULTIBYTE' directives (see above).
src/cmd/ksh93/data/builtins.c:
- sh_optprintf[]: %H: Add single quote to encoded chars doc.
- Edit credits and bump version date.
src/cmd/ksh93/tests/builtins.sh:
- Update and tweak old regression tests.
- Add a number of new tests for UTF-8 HTML and URI encoding, which
are only run when running tests in a UTF-8 locale (shtests -u).
Several regression tests invoke an "interactive" shell using 'ksh
-i'. This records all the commands tested in the shell's history
file. By default, that is the user's history file, ~/.sh_history.
As ksh continuously synchronises history among instances, a ksh
user who ran the regression tests ended up with a number of
mysterious extra commands in their command history.
src/cmd/ksh93/tests/shtests:
- Before running any tests, set and export HISTFILE to a new
history file in the temporary files directory.
There are convincing arguments why including '.' and '..' in the
result of pathname expansion is actively harmful. See:
https://www.austingroupbugs.net/view.php?id=1228https://github.com/ksh93/ksh/issues/58#issuecomment-653716846
pdksh, mksh and zsh already skip these special traversal names
in all cases. This commit makes ksh act like these shells.
Since passing '.' and especially '..' as arguments to commands like
'chmod -R' and 'cp -r' may cause harm, this change seems likely to
fix more legacy scripts than it breaks. I'm unaware of anyone ever
having come up with a concrete use case for the old behaviour.
This change also fixes the bug that '.' and '..' failed to be
ignored as documented if FIGNORE is set.
src/lib/libast/misc/glob.c: glob_dir():
- Explicitly skip any matching '.' and '..' in all cases.
src/cmd/ksh93/tests/glob.sh:
- Add test_glob() tests for '*' and '.*'.
src/cmd/ksh93/sh.1: File Name Generation:
- Update to match new behaviour.
Resolves: https://github.com/ksh93/ksh/issues/58
The 'redirect' builtin command did not error out before executing
any valid redirections. For example, 'redirect ls >foo.txt' issued
an "incorrect syntax" error, but still created 'foo.txt' and left
standard output permanently redirected to it.
src/cmd/ksh93/sh/xec.c: sh_exec():
- If we have redirections (io != NULL), and the command is
SYSREDIR, then check for arguments and error out if there are
any, before calling sh_redirect() to execute redirections.
(Note, the other check for arguments in b_exec() in bltins/misc.c
must be kept, as that applies if there are no redirections.)
src/cmd/ksh93/sh/io.c: sh_redirect():
- Edit comments to better explain what the flag values do.
src/cmd/ksh93/bltins/misc.c:
- Add a dummy b_redirect() function declaration "for the dictionary
generator" as has historically been done for other builtins that
share one C function. I'm not sure what that dictionary generator
is supposed to be, but this also improves greppability.
src/cmd/ksh93/data/builtins.c,
src/cmd/ksh93/sh.1:
- Fix misleading "I/O redirection arguments" term. I/O redirections
are not arguments at all; no argument parser ever sees them.
src/cmd/ksh93/tests/io.sh:
- Test both conditions that should make 'redirect' produce an
"incorrect syntax" error.
- Test that any redirections are not executed if erroneous
non-redirection arguments exist.
src/cmd/ksh93/tests/builtins.sh:
- "... should show usage info on unrecognized options" test:
Because 'redirect' now refuses to process redirections on error,
the error message was not captured. The fix is to run the builtin
in a braces block and add the redirection to the block.
The crash in job_list() or job_unpost() could still occur after the
previous patch if a signal was being handled after $TMOUT was
exceeded and the 60-second grace period was entered.
It *should* work to add a general check for !sh_isstate(SH_GRACE).
We know that the SH_GRACE state is set immediately after printing
the 60 second grace period warning message:
https://github.com/ksh93/ksh/blob/9de65210/src/cmd/ksh93/sh/io.c#L1869-L1870
(and that the crashes occur upon re-evaluating the $PS1 prompt
after setting the SH_GRACE state). We know that the SH_GRACE state
is not turned off again until either the user enters a line:
https://github.com/ksh93/ksh/blob/9de65210/src/cmd/ksh93/sh/main.c#L474
or the shell times out after the grace period:
https://github.com/ksh93/ksh/blob/9de65210/src/cmd/ksh93/sh/io.c#L1861
The SH_GRACE state flag is not used or changed in any other context
(verified with grep -rn SH_GRACE src/cmd/ksh93). So, logically,
this should suffice to make sure the crash stays gone.
src/cmd/ksh93/sh/jobs.c: job_reap():
- Do not list jobs when the SH_GRACE state (the 60 second timeout
grace period after TMOUT was exceeded) is active.
- Keep the previous check for job control just to be sure, and
because it makes sense.
Fixes: https://github.com/ksh93/ksh/issues/103 (again)
This variable is like Bash's $BASHPID, but in virtual subshells
it will retain its previous value as virtual subshells don't fork.
Both $BASHPID and ${.sh.pid} are different from $$ as the latter
is only set to the parent shell's process ID (i.e. it isn't set
to the process ID of the current subshell).
src/cmd/ksh93/include/defs.h:
- Add 'current_pid' for storing the current process ID at a valid
memory address.
- Change 'ppid' from 'int32_t' to 'pid_t', as the return value from
'getppid' is of the 'pid_t' data type.
src/cmd/ksh93/data/variables.c,
src/cmd/ksh93/include/variables.h,
src/cmd/ksh93/sh/init.c,
src/cmd/ksh93/sh/xec.c:
- Add the ${.sh.pid} variable as an alternative to $BASHPID.
The process ID is stored in a struct before ${.sh.pid} is set
as environment variables are pointers that must point to a
valid memory address. ${.sh.pid} is updated by the _sh_fork()
function, which is called when ksh forks a new process with
sh_fork() or sh_ntfork().
src/cmd/ksh93/tests/variables.sh:
- Add ${.sh.pid} to the list of special variables and add three
regression tests for ${.sh.pid}.
src/cmd/ksh93/tests/subshell.sh:
- Update the PATH forking regression test to use ${.sh.pid} and
remove the TODO note.
This bug caused an undefined state, which sometimes crashed the
shell in job_list() or job_unpost(), if $PS1 contains a command
substitution running an external command and the '-b'/'-o notify'
shell option is active. So far the only known way to trigger the
crash is by letting $TMOUT time out the interactive shell. See
https://github.com/ksh93/ksh/issues/103 for details.
src/cmd/ksh93/sh/jobs.c: job_reap():
- The check for the SH_NOTIFY option and the SH_TTYWAIT state
before listing jobs was insufficient. Job control is disabled in
command substitutions, so also check that job control is active
before listing jobs.
src/cmd/ksh93/sh.1:
- Fix TMOUT documentation. The 'read' command in fact only times
out when reading from a terminal, just like 'select'. Also
document the extra 60 second grace period when an interactive
shell prompt reads from a terminal.
Fixes: https://github.com/ksh93/ksh/issues/103
The liblist variable needs to be an extern for dtksh to build.
Quote from CDE developer Chase:
we use an old function that no longer appears in kornshell,
sh_getliblist, it seems to be replaced by the function sh_getlib,
which is fine, but it seems to return a "Shbltin_f" type, which I
can't seem to find any information on what it is. We need the void
pointer dlsym provides for some widget init stuff, I tried making
liblist an extern, but it kept giving me an error about libcomp_t
being undefined.
src/cmd/ksh93/bltins/typeset.c,
src/cmd/ksh93/include/shell.h:
- Fix the compiler error reported above by moving the type definition
for Libcomp_t to shell.h.
- Make liblist an extern since findsym.c in dtksh needs it to build.
The old sh_getliblist function doesn't need to be reintroduced
since the only purpose it served was to workaround the problem
of liblist being a static variable. Now that liblist is an extern,
dtksh fsym can use liblist directly to avoid sh_getliblist.
dtksh findsym.c:
https://sourceforge.net/p/cdesktopenv/code/ci/2.3.2/tree/cde/programs/dtksh/findsym.c
This commit fixes two bugs in the generation of $'...' shellquoted
strings:
1. A bug introduced in f9d28935. In UTF-8 locales, a byte that is
invalid in UTF-8, e.g. hex byte 86, would be shellquoted as
\u[86], which is not the same as the correct quoting, \x86.
2. A bug inherited from 93u+. Single bytes (e.g. hex 11) were
always quoted as \x11 and not \x[11], even if a subsequent
character was a hexadecimal digit. However, the parser reads
past two hexadecimal digits, so we got:
$ printf '%q\n' $'\x[11]1'
$'\x111'
$ printf $'\x111' | od -t x1
0000000 c4 91
0000002
After the bug fix, this works correctly:
$ printf '%q\n' $'\x[11]1'
$'\x[11]1'
$ printf $'\x[11]1' | od -t x1
0000000 11 31
0000002
src/cmd/ksh93/sh/string.c: sh_fmtq():
- Make the multibyte code for $'...' more readable, eliminating the
'isbyte' flag.
- When in a multibyte locale, make sure to shellquote both invalid
multibyte characters and unprintable ASCII characters as
hexadecimal bytes (\xNN). This reinstates 93u+ behaviour.
- When quoting bytes, use isxdigit(3) to determine if the next
character is a hex digit, and if so, protect the quoted byte with
square brackets.
src/cmd/ksh93/tests/quoting2.sh:
- Move the 'printf %q' shellquoting regression tests here from
builtins.sh; they test the shellquoting algorithm, not so much
the printf builtin itself.
- Add regression tests for these bugs.
A segfault happens when an array with an unset method
is turned into a multidimensional array. Reproducer:
function foo {
typeset -a a
a.unset() {
print unset
}
a[3][6][11][20]=7
}
foo
src/cmd/ksh93/sh/nvdisc:
- Fix the multidimensional array unset method crash by
checking if np->nvenv is an array, since multidimensional
arrays need to be handled as arrays. This bugfix was
backported from ksh93v- 2013-10-10-alpha.
src/cmd/ksh93/tests/arrays2.sh:
- Add the reproducer as a regression test for the crash
with multidimensional arrays.
Bug report on the old mailing list:
https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01195.html
The required longjmp used to terminate scripts was not being run
when over-shifting in a POSIX function with a redirection. This
caused scripts to continue after an error in the shift builtin,
which is incorrect since shift is a special builtin. The
interpreter is sent into an indeterminate state that causes
undefined behavior as well:
$ cat reproducer.ksh
some_func() {
shift 10
}
for i in a b c d e f; do
echo "read $i"
[ "$i" != "c" ] && continue
some_func 2>&1
echo "$i = c"
done
$ ksh ./reproducer.ksh
read a
read b
read c
/tmp/k[2]: shift: 10: bad number
c = c
read d
/tmp/k[2]: shift: 10: bad number
d = c
read e
/tmp/k[2]: shift: 10: bad number
e = c
read f
/tmp/k[2]: shift: 10: bad number
f = c
src/cmd/ksh93/sh/xec.c: sh_exec():
- Do the necessary longjmp needed to terminate the script after
over-shifting in a POSIX function when the function call has a
redirection.
src/cmd/ksh93/tests/functions.sh:
- Add the over-shifting regression test from ksh93v- 2013-10-10-alpha.
Bug report and fix on the old mailing list:
https://www.mail-archive.com/ast-developers@lists.research.att.com/msg00732.html
src/lib/libast/tm/tmxfmt.c:
- Making %l and %k aliases to %_I and %_H caused zero padding with
%0l and %0k to fail. Fix that by fully implementing %l and %k
without 'goto push'. This duplicates code from %I and %H, but it
is necessary for these formats to work correctly when zero padded.
src/cmd/ksh93/tests/builtins.sh:
- Add a regression test for manually specifying blank and zero
padding with sixteen different formats.
Some systems disallow executing files in /tmp and there is nothing
regular users can do about it. The build would fail with a
misleading error message about cc being a cross-compiler.
This commit makes the build system consistently use $TMPDIR with
/tmp as a fallback if that variable is not defined. This allows the
user to use another temporary directory with execute permission.
The error message in bin/package is also extended to signal the
possibility of a noexec temp dir.
It was working on Solaris 11.3, but there were still problems
building on Solaris 11.4 with GCC (as on the evaluation VM
downloaded directly from Oracle):
1. ksh immediately segfaulted. Experimenting with the compiler
flags Oracle uses revealed that we need to define _XPG6 for ksh
not to segfault. Why is a mystery.
2. The default path logic used by 'command -p' and the 'getconf
PATH' builtin command was still broken: the result did not
include any of the /usr/xpg?/bin directories where the standard
POSIX utilities actually live. Testing shows that the result of
the C language probe 'confstr(_CS_PATH,name,length)' is broken
on Solaris (it only yields the paths to the historic
non-standard utilities, defeating the purpose) unless _XPG7 is
defined; but the latter makes ksh segfault again. So another
solution is needed.
src/cmd/INIT/package.sh, bin/package:
- Add another hack to add the -D_XPG6 flag to CCFLAGS if we're
running SunOS aka Solaris. (I've tried to add a 'cc.sol11' script
to src/cmd/INIT/ instead, but for some reason that I just don't
have time to figure out, the INIT system ignores that on Solaris
with gcc, so this is the only way I could come up with. Any
patches for less hacky alternatives would be welcome.)
src/lib/libast/comp/conf.sh:
- Sanitise the code for finding the best 'getconf' utility.
src/lib/libast/comp/conf.tab: PATH:
- Since the C-languge getconf(_CS_PATH,...) is broken on Solaris
11.4, replace the C language probe with a shell script probe that
uses the external 'getconf' utility.
- To avoid ksh overriding the result of this probe with the result
of its own getconf(_CS_PATH,...) call, which would make Solaris
use the wrong value again, specify this as an AST configuration
entry instead of a POSIX entry. This should be good enough for
all systems; the OS 'getconf' utility should be reliable and the
default path value is constant for each OS, so can be hardcoded.
src/cmd/ksh93/tests/builtins.sh:
- Add another 'sleep .1' to the 'sleep -s 31' test as it was still
intermittently failing on Solaris and possibly other systems.
Solaris, Illumos distributions, and NetBSD need LDFLAGS set to link
explicitly to libm, otherwise, due to as-yet unknown reasons, the
src/lib/libdll/features/dll fails to write a valid header file and
compilation fails due to unknown identifiers such as Dllscan_t.
This commit adds the flag on those systems.
NixOS is a Linux distro that uses very different paths from the
usual Unix conventions (though it's POSIX compliant), and the
regression tests still needed a lot of tweaks to be compatible.
src/cmd/INIT/package.sh, bin/package:
- On SunOS (Solaris and illumos distros) and NetBSD, add '-lm' to
LDFLAGS before compiling.
src/cmd/INIT/mamprobe.sh, bin/mamprobe,
src/cmd/INIT/execrate.sh, bin/execrate:
- Instead of only in /bin, /usr/bin, /sbin and /usr/sbin, search
utilities in the path given by the OS 'getconf PATH', and use the
user's original $PATH as a fallback.
src/cmd/ksh93/tests/*.sh:
- Miscellaneous portability fixes, mainly elimination of unportable
hardcoded paths to commands.
- basic.sh: Remove test for 'time' keyword millisecond precision.
It was racy and could fail depending on system and system load.
This fixes 'command -p' for systems where getconf(1) lives
somewhere other than in /bin or /usr/bin, i.e. NixOS.
src/lib/libast/comp/conf.tab:
- To determine the default path value for AST 'getconf PATH' and
'command -p', compile a small C program to get the correct local
default path value (_CS_PATH) from the operating system so it
gets hardcoded in the ksh binary. This eliminates the need to to
invoke 'getconf PATH' to get this value, which fixes a catch-22
problem on systems where getconf(1) exists somewhere other than
/bin or /usr/bin.
A multibyte character immediately following an expansion of a
single-character name, e.g. $1 through $9, $?, $-, etc. was
corrupted when in a UTF-8 locale, e.g.:
$ set -- foo; echo "$1テスト"
foo?スト
Prior discussion:
https://www.mail-archive.com/ast-users@lists.research.att.com/msg01060.htmlhttps://bugzilla.redhat.com/show_bug.cgi?id=1256495
src/cmd/ksh93/sh/macro.c:
- Apply a Red Hat patch by Paulo Andrade that avoids calling
fcmbget() if backtracking more than one byte might be required.
src/cmd/ksh93/tests/basic.c:
- Test "テスト" following expansion of "$1", "$?" and "$#".
Co-authored-by: Martijn Dekker <martijn@inlv.org>
Multidimensional associative arrays are created with an extra array
member named '0', which is set to no value. Reproducer:
$ typeset -A foo
$ typeset -A foo[bar]
$ typeset -p foo
typeset -A foo=([bar]=([0]='') )
The bugfix prevents nv_setarray from creating the extra '[0]' member
when an associative array is empty. This bug was discussed on the old
mailing list:
https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01574.html
src/cmd/ksh93/sh/array.c:
- Do not allow the creation of an extra array member when an array
is empty.
src/cmd/ksh93/tests/arrays.sh:
- Add a regression test for creating multidimensional associative
arrays, but use the output from 'typeset -p' instead of fgrep.
When the classic fork/exec mechanism was used (via sh_fork()) to
run an external command from within a non-forking subshell, SIGINT
was blocked until that subshell was exited. If a subsequent loop
was run in the subshell, it became uninterruptible, e.g.:
$ arch/*/bin/ksh -c '(/usr/bin/true; while :; do :; done); exit'
^C^C^C^C^C
src/cmd/ksh93/sh/xec.c:
- sh_fork() did not reset the savesig variable in the parent part
of the fork when running in a virtual subshell. This had the
effect of delaying signal handling until exiting the subshell.
There is no reason for that subshell check that I can discern, so
this removes it.
I've verified that this causes no regression test failures
even when ksh is compiled with -DSHOPT_SPAWN=0 which means the
classic fork/exec mechanism is always used.
Fixes: https://github.com/ksh93/ksh/issues/86
src/cmd/ksh93/tests/builtins.sh:
- Sleep longer after forking a background job to give the OS more
time to launch it; this will hopefully avoid an intermittent
regression test failure on the Github CI runners.
Due to the mysterious workings of vmalloc(3), occasionally a
spurious leak result still showed up. The leak is always smaller
in bytes than the number of test iterations, so it can't be a leak
in the thing tested.
src/cmd/ksh93/tests/leaks.sh:
- Run each test N=512 times.
- Use a 'err_exit_if_leak' function to add a tolerance of N/4 (128)
bytes to each test result check.
Resolves: https://github.com/ksh93/ksh/issues/100
The following is quoted from Marcin Cieślak [*]:
When running under FreeBSD /bin/sh (and not ksh) we get spurious
file named '=' created in the root. This is because the "checksh"
function runs /bin/sh -c '(( .sh.version >= 20111111 ))' which
produces a "=" file with /bin/sh as a side effect.
Fixes https://github.com/ksh93/ksh/issues/13
bin/package,
src/cmd/INIT/package.sh:
- Fix the creation of a spurious '=' file by making sure the shell
has support for (( ... )) expressions.
.gitignore:
- Remove the '=' file entry since it no longer has a purpose.
[*]: https://bsd.network/@saper/103196289917156347
This bugfix is from Marcin Cieślak's fork of the INIT build
system. Before this bugfix, running 'bin/package host cpu'
on FreeBSD would always report one CPU core, even if the CPU
is multi-core:
$ ./bin/package host cpu
1
bin/package,
src/cmd/INIT/package.sh:
- Correctly report the number of CPUs on FreeBSD by using
'sysctl -n hw.ncpu'.
src/cmd/INIT/mamake.c:
- Fix a rare build error by applying Oracle's patch to increase
mamake's buffer size[*]. Description from the original patch:
The build of KornShell might spuriously fail
with the following error.
...
/usr/bin/ksh: line 40: syntax error at line 44: `else unmatched
mamake [lib/libast]: *** exit code 3 making ast.req
mamake: *** exit code 139 making lib/libast
The patch increases the buffer size of mamake to avoid
spurious build failures.
I can't reproduce build error, but this patch should be merged
anyway because OpenSUSE also increases mamake's buffer size
in a patch titled 'workaround-stupid-build-system.diff'[**].
This indicates that the build failure is a heisenbug that can
occur on at least Linux and Solaris.
[*]: 7cad9dae78
[**]: https://build.opensuse.org/package/view_file/shells/ksh/workaround-stupid-build-system.diff?expand=1
src/*/*/Mamfile,
src/lib/libast/Makefile:
- There were a few instances where the CCFLAGS and LDFLAGS were missing
in the Mamfiles and a Makefile. This commit fixes the problem by merging
the changes from Debian's blhc.diff patch:
f8fea737c9/debian/patches/blhc.diff
Related discussion:
https://github.com/ksh93/ksh/issues/95#issuecomment-664010969
src/cmd/ksh93/tests/leaks.sh:
- When ksh is compiled to use the system's malloc(3) instead of AST
vmalloc(3), the vmstate builtin returns either nothing or zero.
Detect this as a regression test failure and refuse to run tests.
- Tweak iterations. Tests don't need 500 or 1000 runs for vmstate.
src/cmd/ksh93/data/builtins.c:
- Do not compile in vmstate builtin when using system's malloc(3).
If an array or upper/lowercase variable was declared with a null
initial value within a virtual/non-forked subshell, like:
( typeset -a foo; ... )
( typeset -A foo; ... )
( typeset -l foo; ... )
( typeset -u foo; ... )
then the type declaration leaked out of the subshell into the
parent shell environment, though without any values that may
subsequently have been assigned.
src/cmd/ksh93/bltins/typeset.c: setall():
- When deciding whether to create a virtual subshell scope for a
variable, use sh_assignok(), which was actually designed for the
purpose, instead of _nv_unset(). This allows getting rid of a
tangled mess of special-casing that never worked quite right.
src/cmd/ksh93/tests/arrays.sh:
- Add regression tests checking that array declarations don't leak
out of virtual subshells.
src/cmd/ksh93/tests/attributes.sh:
- Add regression tests for combining the 'export' and 'readonly'
attributes with every other possible typeset attribute on unset
variables. This also includes a subshell leak test for each one.
Fixes: https://github.com/ksh93/ksh/issues/88
When a builtin is given an unrecognized option, the usage information
for that builtin should be shown as 'Usage: builtin-name options'. The
sleep and suspend builtins were an exception to this. 'suspend' would
not show usage information and sleep wouldn't exit on error:
$ suspend -e
/usr/bin/ksh: suspend: -e: unknown option
$ time sleep -e 1
sleep: -e: unknown option
real 0m1.00s
user 0m0.00s
sys 0m0.00s
src/cmd/ksh93/bltins/sleep.c:
- Show usage information and exit when sleep is given an unknown
option. This bugfix was backported from ksh2020: https://github.com/att/ast/pull/1024
src/cmd/ksh93/bltins/trap.c:
- Use the normal method of parsing options with optget to fix the
suspend builtin's test failure.
src/cmd/ksh93/tests/builtins.sh:
- Add the ksh2020 regression test for getting the usage information
of each builtin. Enable all /opt/ast/bin builtins in a subshell
since those should be tested as well (aside from getconf and uname
because those builtins fallback to the real commands on error).
Add support for multibyte characters to $IFS
This commit fixes BUG_MULTIBIFS, which had two bug reports in the ksh2020 branch.
src/cmd/ksh93/sh/macro.c:
- Backport Eric Scrivner's fix for multibyte IFS characters (slightly modified
for compatibility with C89). Explanation from https://github.com/att/ast/pull/737:
Previously, the varsub method used for the macro expansion of $param, ${param},
and ${param op word} would incorrectly expand the internal field separator (IFS)
if it was a multibyte character. This was due to truncation based on the
incorrect assumption that the IFS would never be larger than a single byte.
This change fixes this issue by carefully tracking the number of bytes that
should be persisted in the IFS case and ensuring that all bytes are written
during expansion and substitution.
Bug report: https://github.com/att/ast/issues/13
- Fixed another bug that caused multibyte characters with the same initial byte
to be treated as the same character by the IFS. This bug was occurring because
the first byte of a multibyte character wasn't being written to the stack when
the IFS delimiter had the same initial byte:
$ IFS=£
$ v='§'
$ set -- $v
$ v="${1-}"
$ echo "$v" | hd # The first byte should be c2, but it isn't due to the bug
00000000 a7 0a |..|
00000002
Bug report: https://github.com/att/ast/issues/1372
src/cmd/ksh93/tests/variables.sh:
- Add (reworked) regression tests from ksh2020 for the multibyte IFS bugs.
- Add a regression test for att/ast#1372 based on the reproducer.
The following explanation is mostly taken from Tomas Klacko's report on
the old mailing list (which also contains a C program reproducer) [*]:
1. When ksh starts a binary, it sets its environment variable "_"
to "*number*/path/to/binary". Where "number" is the pid of the
ksh process.
2. The binary forks and the child executes a suid root shell script
which begins with #!/bin/sh. For this bug to occur, ksh must be /bin/sh.
3. The ksh process interpreting the suid shell script leaves the "_"
variable as not set (nv_getval(L_ARGNOD) returns NULL) because
the "number" from step 1 is not the pid of its parent process.
4-5. Because "_" is not set and the script is suid root, an infinite
loop occurs because when the SHELL environment variable contains
"/bin/sh" pathshell() returns "/bin/sh". This becomes an infinite
loop of /bin/sh /dev/fd/3 executing /bin/sh /dev/fd/3.
src/cmd/ksh93/sh/init.c: get_lastarg():
- Disable the check for if the "number" refers to the process id of
the parent process.
src/cmd/ksh93/sh/main.c: sh_main():
- Prevent an infinite loop when '$_' is not passed in from the environment.
Solaris applies this bugfix to their version of ksh:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/190-17432413.patch
[*]: https://www.mail-archive.com/ast-developers@lists.research.att.com/msg01680.html
