_ _ ___ _____ ___ ___ ___
| | _____| |__ / _ \___ / _ _ _ _ __ ___ / / | / _ \ / _ \
| |/ / __| '_ \ | (_) ||_ \| | | |_| |_| '_ ` _ \ / /| || | | | | | |
| <\__ \ | | | \__, |__) | |_| |_ _| | | | | |/ / | || |_| | |_| |
|_|\_\___/_| |_| /_/____/ \__,_| |_| |_| |_| |_/_/ |_(_)___(_)___/
It may have taken exactly a decade, but here we are... a proper new
ksh release. :) Many thanks to all contributors for their hard work!
Compared to an unpatched 93u+, this release has roughly a thousand
bugs fixed. It incorporates a fair number of enhancements as well.
Not all known bugs have been worked out yet; see the TODO file. Let's
hope this release will rekindle interest and attract more bug hunters.
This commit also makes some very minor fixes in comments. Notable:
src/cmd/ksh93/sh/arith.c: sh_strnum():
- Update a security-related comment. As of b48e5b33, evaluating
untrusted arithmetic expressions from the environment should no
longer cause CVE-2019-14868. But let's keep disallowing it anyway.
Resolves: https://github.com/ksh93/ksh/issues/491
These are a few minor edits to the man page for readability and consistency:
* Add missing "in" to or else a fifo in a temporary directory
* Some paragraphs had missing spacing between them
* Some lists were missing spacing before the following paragraph
* Use bullet points to list the rksh restrictions
* Indent the list of characters in the "Quoting" section
This now allows easy installation via:
bin/package install DESTINATION_DIRECTORY [ COMMAND ... ]
Commands are installed in bin and share/man subdirectories.
If no command is specified, ksh and shcomp are assumed.
[This commit was previously reverted because it seemed to cause the
build to fail on Cygwin. But I just re-tested it, and it's fine. It
may be that my Cygwin installation at the time was defective.]
Something similar was previously done in 07cc71b8 from a Debian
patch, and eventually reverted; it redefined the ast atomic
functions asoincint() and asodecint() to be gcc-specific. This
imports the upstream version from the ksh 93v- beta instead.
This commit is based on an OpenSUSE patch:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-joblock.dif
src/cmd/ksh93/include/jobs.h:
- Replace job locking mechanism with the 93v- version which uses
the atomic libast functions asoincint(), asogetint() and
asodecint(). See: src/lib/libast/man/aso.3
src/cmd/ksh93/sh/jobs.c: job_subsave():
- Revert gcc optimiser bug workaround from c258a04f.
It should now be unnecessary.
There were still intermittent leak test failures on Debian.
They're clearly spurious as a different test fails each time.
My testing on a Debian 11.4.0 VM showed that it takes not a
doubling, but a quadrupling of the maximum number of test
iterations (from 16384==2**14 to 65536==2**16) for that system's
memory state to stabilise enough to avoid spurious test failurese.
Resolves: https://github.com/ksh93/ksh/issues/501
When converting from -Z to another attribute, if the value of the
-Z variable was empty (so -Z0 and an empty value), the loop that
skips initial zeros may read before the beginning of the buffer:
2968: if(nv_isattr(np,NV_ZFILL))
2969: {
2970: while(*sp=='0') sp++; /* skip initial zeros */
2971: if(!*sp) sp--; /* if number was 0, leave one zero */
2972: }
If the *sp value is empty (just a terminating zero byte), line 2970
does nothing, but line 2971 still decrases the pointer, to before
the beginning of the buffer.
The fix is to check for an initial zero before running that block.
Reproducer (fails intermittently, depending on garbage before *sp):
typeset -Z foo=
typeset -i foo
macOS doesn't like it if ast.h is included in vmmopen.c
(conflicting type declarations for brk() and sbrk() between that
file and the OS header), so only include ast.h when we're actually
using the the NoN stub macro from it.
Thanks to Mark Wilson for the build failure report.
Resolves: https://github.com/ksh93/ksh/issues/499
This commit:
1. Updates the COPYRIGHT file with all the contributors to ksh
93u+m since the reboot, based on the commit history, sorted by
number of commits in descending order. It also separates the
authors of backported contributions from direct contributors.
2. Adds missing contributors of source each file to that file's
copyright header based on that file's commit history.
The script used to maintain this is now up at:
https://github.com/ksh93/ksh/wiki/update-copyright.sh
In addition, the original ast copyright notice is restored to
its EPL 1.0 version -- that is what AT&T Intellectual Property
originally licensed this code under, and we are the ones who
upgraded it to EPL 2.0 (as allowed by EPL 1.0) in 441dcc04.
The history should be accurate.
- Fixed a few minor typos and updated the list of tested systems in
src/cmd/ksh93/README.
- vmmopen.c: Include the ast.h header to fix a build error on Haiku
caused by the otherwise undefined NoN macro (re: 05f0c1b1).
In the documentation on the .unset discipline, this removes the
sentence: "The variable will not be unset unless it is unset
explicitly from within this discipline function."
Every ksh93 version published since at least 1995 unsets a variable
if 'unset' is run, regardless of whether any .unset discipline
function itself unsets the variable or not.
But all the documentation and books ever published (man page,
Bolsky, O'Reilly, IBM AIX docs) claim that an .unset discipline
needs to unset the variable again in order for it to be unset.
Bolsky (1995) p. 79, says:
> unset
> Called when the variable is unset. The variable will not be
> unset unless it is unset inside the dicipline function.
Other similar claims:
https://github.com/ksh93/ksh/issues/419#issuecomment-1199773432
So, ALL OF IT IS WRONG, believe it or not. Welcome to the
ksh93 twilight zone, where nothing is as it seems.
There are other bugs, see #419. But for now, other than correcting
the man page, I'm not touching this mess with a ten-foot pole.
The callback function for malloc(3) failure in the stk(3) routines
(which can be specified using stkinstall() and is set to nomemory()
in init.c) uses a size argument of type int. That's a type mismatch
with all the other size arguments and variables in stk(3) which use
size_t, an unsigned type that may be larger than int.
This is all quite inconsequential as nothing in our code base (or
in the complete old AT&T AST code base) actually uses that size for
anything, but it's still wrong, so this corrects the interface.
With this very minor API change, let's bump the libast API version
to 20220801, the date of the upcoming ksh 93u+m 1.0.0 release. :)
The ksh93/sh/init.c nomemory() function now reports the size that
could not be allocated, just because it can.
The bug is that "$*", and related expansions such as "${arr[*]}",
etc., do pattern matching if the first character of $IFS is a
wildcard. For example, the following:
IFS=*
set -- F ''
case BUGFREE in
BUG"$*") echo bug ;;
esac
outputs 'bug'. This bug can be reproduced in every other glob
pattern matching context as well, but not in pathname expansion.
src/cmd/ksh93/sh/macro.c: varsub():
- When joining fields into one for a "$*"-type expansion, check if
a glob pattern matching operation follows (mp->pattern is set).
If so, write a preceding backslash to escape the separator.
Resolves: https://github.com/ksh93/ksh/issues/489
Resolves: https://github.com/att/ast/issues/12
EPL 1.0 says, in section 7: "The Program (including Contributions)
may always be distributed subject to the version of the Agreement
under which it was received. In addition, after a new version of
the Agreement is published, Contributor may elect to distribute the
Program (including its Contributions) under the new version."
The Eclipse Foundation also encourage everyone to upgrade:
https://www.eclipse.org/legal/epl-2.0/faq.php#h.60mjudroo8e5https://www.eclipse.org/legal/epl-2.0/faq.php#h.tci84nlsqpgw
Unfortunately the new Secondary License option is not available to
us as we're not the original copyright holders and don't have the
legal power to add one. So, no GPL compatibility. Sorry.
Attempting to tab-complete the only member of a directory while in
vi mode switches to "control" under the following conditions:
1. pwd is outside of the directory
2. No part of the basename has been typed in (e.g., starting with
vim TEST/t will autocomplete to vim TEST/test, but starting
with vim TEST/ will not).
Also, trying to type with "input" or "replace" fails afterwards
until a new line has been generated (either with ctrl + c or the
j/k keys in control mode).
src/cmd/ksh93/edit/vi.c: textmod():
- The strange completion behavior in vi mode for single-member
directories appears to be at least partially due to completion
failing to use \ or * mode from outside of pwd. Checking for
e_nlist directly from vi.c — 0 unless in = mode — allows for
entering the else loop at line 2575 to reach APPEND mode.
Resolves: https://github.com/ksh93/ksh/issues/485
Since we now have sh.current_ppid, we might as well point a shell
variable to it, as the cost is nil.
Together, ${.sh.pid} and ${.sh.ppid} (updated when a virtual
subshell forks) form a logical counterpart pair to $$ and $PPID
(never updated in subshells).
This commit also adds a section to the manual page that hopefully
does away with the depressingly widespread subshell/child shell
confusion once and for all... :P
The undocumented .sh.dollar variable was a hack used to change the
value of $$ (main shell's PID); any value assigned to .sh.dollar
overrides the value of $$. This was used by libcoshell and by the
pre-fork(2) code to update the value of $$ in child processes
initialised by internally generated shell scripts. Both were
removed long ago, so we don't need this.
The Sfio code uses its own _has_multibyte macro and that was not
being disabled by AST_NOMULTIBYTE, so some of its multibyte code
was still getting compiled in. This is easily fixed in sfhdr.h.
But that exposed another issue: the Sfio stdio wrappers didn't
know about the _has_multibyte macro. So this adds the necessary
directives to keep their multibyte functions from compiling.
For $PPID and ${.sh.pid}, ksh has simply been assuming that pid_t
is a 32-bit integer. In POSIX, the size of pid_t is not specified
and operating systems are free to vary it. On any system where
sizeof(pid_t) != 4, these variables produce undefined results.
(Note that this is not a problem with $$ as it is converted to a
string value directly from pid_t in special() in macro.c.)
src/cmd/ksh93/features/options:
- Add feature test that produces NV_PID, a name-value attribute
macro that is set to NV_INTEGER|NV_SHORT (for int16_t),
NV_INTEGER (for int32_t) or NV_INTEGER|NV_LONG (for Sflong_t
a.k.a. intmax_t), depending on the size of pid_t. These are the
three integer size attributes supported by nv_getval(). The build
will abort on any (hypothetical?) system where pid_t has a size
other than these three. In that case, the test and nv_getval()
will need to have a new integer size attribute added.
src/cmd/ksh93/data/variables.c:
- Use NV_PID instead of NV_INTEGER for $PPID and ${.sh.pid}. This
fixes the issue.
Other changed files:
- Fix the typecasts to various fmtbase() calls; the first argument
is of type intmax_t (a.k.a. Sflong_t), not long.
- When outputting PIDs with sfprintf(), typecast to Sflong_t and
use %lld to account for possible systems with very large PIDs.
(Though %lld is not in C89/C90, Sfio supports it anyway.)
I made a mistake in sh.reinit() which caused $PPID to be set to the
new process ID, not the parent process ID. This commit fixes it by
introducing, updating and using sh.current_ppid, so we continue to
minimise context switches due to getpid(2)/getppid(2) system calls.
Thanks to Geoff Clare for the report.
The arguments to the binary numerical comparison operators (-eq,
-gt, etc.) in the [[ and test/[ commands are treated as arithmetic
expressions, even if $((...)) is not used. But there is some
seriously incorrect behaviour:
Reproducers (all should output 0/true):
$ [[ 0x0A -eq 10 ]]; echo $?
1
$ [[ 1+0x0A -eq 11 ]]; echo $?
0
$ (set --posix; [[ 010 -eq 8 ]]); echo $?
1
$ (set --posix; [[ +010 -eq 8 ]]); echo $?
0
$ [[ 0xA -eq 10 ]]; echo $?
1
$ xA=10; [[ 0xA -eq 10 ]]; echo $?
0
$ xA=WTF; [[ 0xA -eq 10 ]]; echo $?
ksh: WTF: parameter not set
(POSIX mode enables the recognition of the initial 0 as a prefix
indicating an octal number in arithmetic expressions.)
The cause is the two 'while' loops in this section in test_binop()
in src/cmd/ksh93/bltins/test.c:
502: if(op&TEST_ARITH)
503: {
504: while(*left=='0')
505: left++;
506: while(*right=='0')
507: right++;
508: lnum = sh_arith(left);
509: rnum = sh_arith(right);
510: }
So initial zeros are unconditionally skipped. Ostensibly this is to
disable the recognition of the initial zero as an octal prefix as
well as 0x as a hexadecimal prefix. This would be okay for
enforcing a decimal-only limitation for simple numbers, but to do
this for arithmetic expressions is flagrantly incorrect, to say the
least. (insert standard rant about AT&T quality standards here)
The fix for '[[' is simply to delete the two 'while' loops. But
that creates a problem for the deprecated-but-standard 'test'/'['
command, which also uses the test_binop() function. According to
POSIX, test/[ only parses simple decimal numbers, so octal, etc. is
not a thing. But, after that fix, 'test 08 -eq 10' in POSIX mode
yields true, which is unlike every other shell. (Note that this is
unlike [[ 08 -eq 10 ]], which yields true on bash because '[['
parses operands as arithmetic expressions.)
For test/[ in non-POSIX mode, we don't need to change anything. For
POSIX mode, we should only parse literal decimal numbers for these
operators in test/[, disallowing unexpanded arithmetic expressions.
This makes ksh's POSIX-mode test/[ act like every other shell and
like external .../bin/test commands shipped by OSs.
src/cmd/ksh93/bltins/text.c: test_binop():
- Correct a type mismatch. The left and right hand numbers should
be Sfdouble_t, the maximum double length on the current system,
and the type that sh_arith() returns. Instead, long double
(typeset -lF) values were reduced to double (typeset -F) before
comparing!
- For test/[ in POSIX, only accept simple decimal numbers: use
strtold() instead of sh_arith(). Do skip initial zeros here as
this disables the recognition of the hexadecimal prefix. Throw an
error on an invalid decimal number. Floating point constants are
still parsed, but that's fine as this does not cause any
incompatibility with POSIX.
- For code legibility, move handling of TEST_EQ, etc. to within the
if(op&TEST_ARITH) block. This also allows lnum and rnum to be
local to that block.
We're nearly there!
I intend to release ksh 93u+m/1.0.0 on 2022-08-01, precisely ten
years after the last canonical 93u+ release.
We have a week until then, so here's a release candidate. Please
try as hard as you can to break it, and to help fix known bugs.
As of this commit, the 1.0 branch is feature-frozen and will only
get bugfixes.
src/cmd/ksh93/fun/man:
- Last-minute fix: .man.try_os_man(): do not look for arguments
with / in section 1 and 8; this can cause false positives.
Not only is it C++-specific (and we no longer pretend to be
compatible with C++), it's also z/OS-specific. It seems that
access to z/OS for testing is not available to mere mortals.
The Solaris 10.1 man command happily exits with status 0 on error
and even prints error messages to standard output. This was fixed
in later versions of Solaris. but we should still fix the function
for portability as it should work almost anywhere.
.man.try_os_man() now checks that the standard output of a man
command includes at least three newlines; that's probably a
reliable-enough indicator that it's a man page and not an error
message. Thankfully, all man commands do seem to deactivate the
pager when standard output is not on a terminal (e.g. when catching
standard output with a command substitution).
Somewhat notable changes:
- Remove pointless test commands from Mamfiles.
- Consistent use of NoN macro instead of manual void _STUB_foo(){}
(this is to silence "foo.o has no symbols" warnings from ld).
- Remove src/lib/libast/comp/transition.c; obsolete, does nothing.
- xec.c: Fix off-by-one error in sigreset() used by sh_ntfork():
it tried to (re)set signal 0, which is harmless, but wrong.
(note that sh.st.trapmax is the current max trapped sig + one!)
This autoloadable function activates a feature similar to 'shopt -s
autocd' in bash: type only a directory name to change directory.
It uses the DEBUG trap to work. See the file for details.
This adds a new 'man' function in src/cmd/ksh93/fun/man, meant for
autoload. This integrates the --man self-documentation of ksh
built-in and external AST commands with your system's 'man' command
so you can conveniently use 'man' for all commands, whether
built-in or external. See the file for details.
Since 'command -x' provides xargs-like functionality to repeatedly
run external commands if the argument list is too long for one
invocation, it makes little sense to use with built-ins. So I
decided that 'command -x' should double as a way to guarantee
running an external command. However, it was only bypassing plain
built-ins and not path-bound builtins (the ones that show up with a
virtual directory path name in the output of 'builtin').
src/cmd/ksh93/sh/path.c:
- Before looking for a path-bound built-in, check that the SH_XARG
state bit is not on. This needs to be done in path_absolute() as
well as in path_spawn().
Even tough sh_subtmpfile() should only be relevant to command
substitutions, checking the sh.comsub flag instead of sh.subshell
before calling is not valid in all cases; subshells of command
substitutions run into problems in some oddly specific cases, e.g.,
both 'eval' and an external command must be involved. The ksh
regression tests didn't detect a problem, but both modernish[*] and
shellspec[*] have one regression test failure after that change.
Minimal reproducer, assuming a cat at /bin/cat:
v=$(eval 'print output | /bin/cat')
print -r "v=[$v]"
Actual output:
output
v=[]
Expected output:
v=[output]
This commit makes three interrelated changes.
First, the code for erasing the command line before redrawing it
upon a window size change is simplified and modernised. Instead of
erasing the line with lots of spaces, it now uses the sequence
obtained from 'tput ed' (usually ESC, '[', 'J') to "erase to the
end of screen". This avoids messing up the detection and automatic
redrawing of wrapped lines on terminals such as Apple Terminal.app.
Second, the -b/--notify option is made more usable. When it is on
and either the vi or emacs/gmacs line editor is in use, 'Done' and
similar notifications are now buffered and trigger a command line
redraw as if the window size changed, and the redraw routine prints
that notify buffer between erasing and redrawing the commmnd line.
The effect is that the notification appears to magically insert
itself directly above the line you're typing. (The notification
behaviour when not in the shell line editor, e.g. while running
commands such as external editors, is unchanged.)
Third, a bug is fixed that caused -b/--notify to only report on one
job when more than one terminated at the same time. The rest was
reported on the next command line as if -b were not on. Reproducer:
$ set -b; sleep 1 & sleep 1 & sleep 1 &
This commit also includes a fair number of other window size and
$COLUMNS/$LINES handling tweaks that made all this easier, not all
of which are mentioned below.
src/cmd/ksh93/include/fault.h,
src/cmd/ksh93/sh/fault.c:
- Replace sh_update_columns_lines with a new sh_winsize() function.
It calls astwinsize() and is to be used instead of it, and
instead of nv_getval(LINES) and nv_getval(COLUMNS) calls. It:
- Allows passing one or neither of lines or cols pointers.
- Updates COLUMNS and LINES, but only if they actually changed
from the last values. This makes .set discipline functions
defined for these variables more useful.
- Sets the sh.winch flag, but only if COLUMNS changes. If only
the height changes, the command line does not need redrawing.
src/cmd/ksh93/include/io.h:
- Add sh_editor_active() that allows checking whether one of vi,
emacs or gmacs is active without onerous #if SHOPT_* directives.
src/cmd/ksh93/sh/jobs.c: job_reap():
- Remove the fix backported in fc655f1a, which was really just a
workaround that papered over the real bug.
- Move a check for errno==ECHILD so it is only done when waitpid()
returns an error (pid < 0); the check was not correct because C
library functions that do not error out also do not change errno.
- Move the SH_NOTIFY && SH_TTYWAIT code section to within the
while(1) loop so it is run for each job, not only the
last-processed one. This fixes the bug where only one job was
notified when more than one ended at the same time.
- In that section, check if an editor is active; if so, set the
output file for job_list() to sh.notifybuffer instead of standard
error, list the jobs without the initial newline (JOB_NLFLAG),
and trigger a command line redraw by setting sh.winch.
src/cmd/ksh93/edit/edit.c:
- Obtain not just CURSOR_UP but also ERASE_EOS (renamed from
KILL_LINE) using 'tput'. The latter had the ANSI sequence
hardcoded. Define a couple of TPUT_* macros to make it easier to
deal with terminfo/termcap codes.
- Add get_tput() to make it easier to get several tput values
robustly (with SIGINT blocked, trace disabled, etc.)
- ed_crlf(): Removed. Going by those ancient #ifdefs, nothing that
93u+m will ever run on requires a '\r' before a '\n' to start a
new line on the terminal. Plus, as of 93u+, there were already
several spots in emacs.c and vi.c where it printed a sole '\n'.
- ed_read():
- Simplify/modernise command line erase using ERASE_EOS.
- Between erasing and redrawing, print the contents of the notify
buffer. This has the effect of inserting notifications above
the command line while the user is typing.
src/cmd/ksh93/features/cmds:
- To detect terminfo vs termcap codes, use all three codes we're
currently using. This matters on at least on my system (macOS
10.14.6) in which /usr/bin/tput has incomplete terminfo support
(no 'ed') but complete termcap support!
Reproducer:
$ /bin/sleep 100
^Z[1] + Stopped /bin/sleep 100
$ kill %% <--- no notification shown
$ jobs <--- nothing: it was in fact killed
Expected behaviour:
$ /bin/sleep 100
^Z[1] + Stopped /bin/sleep 100
$ kill %%
[1] + Terminated /bin/sleep 100
In the reproducer, the job in fact gets killed, but no notification
is printed of this fact. This is because notifications now require
the P_BG (background job) flag, and it is not set if a foreground
job is stopped. It should be: stopping moves it to the background.
When the 'sleep' builtin is used instead of the external command,
the notice says 'Done' instead of 'Terminated' because SIGTERM is
being handled via sh_fault() instead of set to SIG_DFL. It remains
to be considered if anything needs to change about that.
src/cmd/ksh93/sh/jobs.c: job_reap():
- Also set a job's P_BG bit when it was stopped.
After that commit, iousepipe(), sh_iounpipe(), and supporting flags
were all broken and dead code. Since all command substitutions use
temp files now, they're unused; remove them.
I'm experimenting with reintroducing pipes to command substitutions
in a consistent way, as this is needed for them to wait for
grandchildren. If and when I ever manage to figure out how to do
that in a way that doesn't cause severe hanging and crashing bugs,
these functions may return in some form.
Related: https://github.com/ksh93/ksh/issues/124
Notable changes:
- Tie up some loose ends re: 3de4da5a and 7ba2c685.
- comp/omitted.c: Header include fix for Cygwin.
- misc/optget.c:
- args(): When printing options for the uage line, use a local
pointer for the 'if' block instead of reusing the 'b' pointer.
That variable is used to output blanks later.
- The above fix allows re-enabling the AST translation-aware
macros and deleting the astsa fallback without causing usage
message corruption in multibyte locales. Maybe someday we'll
make ksh actually translatable.
- Remove code to reinitialise _error_infop_ and _opt_info_
'because these are not initialised by all DLLs'. In 2022,
hopefully the buggy dynamic linkers are fixed. If not, we're
not going to find out by keeping the workaround. I suspect that
those bugs may have been triggered by the Microsoft/Cygwin
import/export obfuscation removed in 3de4da5a.
- ksh93:
- Remove unused sh.st.var_local variable. This was a leftover of
a 93v- attempt to implement the bash 'local' command. It used
static scoping, so it's not actually compatible.
- Add a few regression tests for miscellaneous breakage that I
caused in experiments (the breakage never made it to git; the
tests are just to keep it that way).
Windows/Cygwin requires onerous special handling and the definition
of additional _imp__* symbols to import/export symbols between
dynamically linked binaries. Its support in AST used a lot of
macros and code obfuscation. In the features/common test for this,
AT&T called this the "Microsoft import/export nonsense".
They're right, it's nonsense. Somehow, Microsoft's POSIX layer,
SFU/Interix, always managed without it. No one has time to maintain
this (especially considering how incredibly sluggish Cygwin is).
And in fact, it had already fallen victim to bit rot; I confirmed
this in my early experiments with reintroducing dynamic library
support. No one has time to fix it, either.
So, my apologies to any Cygwin fans; ksh 93u+m will never support
dynamically loadable built-ins on Cygwin, even when I do manage to
reintroduce dynamic linking properly.
This removes support for the (AFAIK, completely unavailable) UWIN
preprocessor, which used directives that nothing else understands,
hidden behind the following macros:
__STDPP__
__STDPP__directive
__STDPP__hide
This mailing list post from 2001 says that __STDPP__ is UWIN:
https://lists.gnu.org/archive/html/bug-groff/2001-04/msg00030.html
This also removes code hidden by the __EMX__ symbol. EMX (Eberhard
Mattes eXtender) was a programming environment and POSIX interface
for MS-DOS and OS/2. The last release was in 1998.
https://en.wikipedia.org/wiki/EMX_%28programming_environment%29
UWIN was David Korn's UNIX emulation layer for Microsoft Windows.
It was never very well known, certainly not like Cygwin or
Microsoft SFU/Interix. It was a very interesting system that
exposed the Windows registry to the file system, making it
UNIX-like, and that natively used ksh and all the AST utilities.
Regrettably, it appears to be dead and buried. Only 32-bit binaries
can still be found in the wild, as well as the source code at:
https://github.com/att/uwin
The latter does not seem to be usable since (as far as I can tell)
it requires a UWIN environment with a compiler to build, and UWIN
binaries with a compiler are simply nowhere to be found.
The activity level on that repo (which is zero) also shows how much
interest there still is in this project. And of course the
supporting code in this repo is almost certainly broken by now as
we've never been able to test it on a UWIN system.
The AST team clearly cared about it since roughly 8k lines of code
are dedicated to its support, disabled (directly or indirectly) on
non-UWIN systems via the _UWIN macro. This removes all that.
astsa(3) is a small stand-alone version of the ast library that
only requires cdt(3) and sfio(3). We're not using it. Nothing in
the original AT&T code base was using it, either. It wasn't being
compiled. Bit rot is inevitable, so remove.
An old annoyance of mine: when an error occurs in a ksh function,
the function name is reported in the error message, but the same is
not done for POSIX functions.
Since POSIX functions are treated as glorified dot scripts,
b_dot_cmd() needs an errorpush() call and an assignment to
error_info.id to make this happen for both POSIX functions and dot
scripts. This is the same thing that sh_funscope() does for ksh
functions. There is now no difference from ksh functions in how
these report errors.
Note that the sh_popcontext() macro includes an errorpop() call, so
that does not need to be added. See fault.h and error.h.
In main.c:
158: if(sh.ppid==1)
159: sh.login_sh++;
If that was ever valid, it certainly is not now. As far as I know,
there is no currently existing system where PID 1 (init or systemd
or whatever) is the parent shell of the login shell, even straight
after bootup; login shells are invoked via a program like login(1).
Plus, there is no guarantee the init process actually has PID 1.
This invalidates all use of login_sh that couldn't be replaced by
checks for the login_shell option, so this commit does just that.
src/cmd/ksh93/include/shell.h:
- Remove login_sh flag.
src/cmd/ksh93/sh/init.c:
- If a login shell was detected, just set the login_shell option.
- Remove obsolete check for #! setuid scripts. This was meant to
guard against a symlink called '-i' to a setuid script with a
hashbang path, which used to give users a root shell. All modern
Unixes ignore the setuid bit when they detect a hashbang path.
src/cmd/ksh93/SHOPT.sh:
- By default, let's require the -p/--privileged invocation option
for the setuid/setgid bit on the shell binary to be respected,
for all user IDs (>= 0). This is what bash and mksh do, and
it seems sensible. (See init.c 1475-1483)
The pseudorandom generator generates a reproducible sequnece of
values after seeding it with a known value. But:
$ (RANDOM=1; echo $RANDOM; echo $RANDOM)
2100
18270
$ (RANDOM=1; echo $RANDOM; ls >/dev/null; echo $RANDOM)
2100
30107
Since RANDOM was seeded with a specific value, the two command
lines should output the same pair of numbers. Running 'ls' in the
middle should make no difference.
The cause is a nv_getval(RANDNOD) call in xec.c, case TFORK, that
is run for all TFORK cases, in the parent process -- including
background jobs and external commands. What should happen instead
is that $RANDOM is reseeded in the child process.
This bug is in every version of ksh93 since before 1995.
There is also an opportunity for optimisation. As of 396b388e, the
RANDOM seed may be invalidated by setting rand_last to -2,
triggering a reseed the next time a $RANDOM value is obtained. This
was done to optimise the virtual subshell mechanism. But that can
also be used to eliminate unconditional reseeding elsewhere. So as
of this commit, RANDOM is never (re)seeded until it's used.
src/cmd/ksh93/include/variables.h,
src/cmd/ksh93/sh/subshell.c:
- Add RAND_SEED_INVALIDATED macro, a single source of truth for the
value that triggers a reseeding in sh_save_rand_seed().
- Add convenient sh_invalidate_rand_seed() function macro.
src/cmd/ksh93/sh/init.c,
src/cmd/ksh93/sh/xec.c:
- Optimisation: invalidate seed instead of reseeding directly.
- sh_exec(): case TFORK: Delete the nv_getval(RANDNOD) call. Add a
sh_invalidate_rand_seed() to the child part. This fixes the bug.
Before the fix, a file descriptor leak could occur on command not
found because sh_ntfork() saves sh.topfd on function entry (as part
of the sh_pushcontext() macro expansion) and uses that value to
sh_iorestore(). Process substitution arguments will already have
been processed by then and their file descriptors opened, so those
were not closed on restore. To compensate, the committed fix added
a second sh_iorestore() call using the topfd variable, in which
sh_exec) saves the value of sh.topfd on its function entry. A more
elegant fix is to pass that value to sh_ntfork() as an argument so
its own sh_iorestore() call does the right thing.
In the olden days, ksh used the hash(3) library to store variables,
aliases, functions, etc. For many years, it's been using the cdt(3)
library instead. But the low-level nv_search() name-value tree
lookup function was still repurposing some bit flags from the old
hash API for its options, though that API is otherwise unused.
So we were still including the entire obsolete <hash.h> API just
to use two repurposed HASH_* bit flags for nv_search(). This commit
makes nv_search() repurpose some flags from <nval.h> instead.
This commit should not change ksh's behaviour.
src/cmd/ksh93/sh/nvdisc.c:
- Make nv_search() use NV_NOSCOPE instead of HASH_NOSCOPE and
NV_REF instead of HASH_BUCKET.
- The HASH_SCOPE flag was also passed to some nv_search() calls,
but nv_search() ignores it, so that flag is deleted from those.
- Document nv_search() in a comment.
src/cmd/ksh93/include/name.h:
- Move NV_UNATTR to nval.h to join the other nv_open() options
there. (re: 1184b2ad)
src/cmd/ksh93/include/nval.h:
- Since we no longer use HASH_* macros, do not include <hash.h>.
- Remove unused NV_NOASSIGN macro, defined to 0. This was there
for "backward compatibility" since before 1995; long enough.
src/cmd/ksh93/include/shell.h:
- Bump SH_VERSION due to the nv_search() API change (even though no
changes were made to the APIs documented in nval.3 or shell.3).
All other changed files:
- Update to match the flaggery changes.
*.getn discipline functions cause .sh.value to have a float type
for arithmetic expressions that get the value of foo, avoiding the
problem of having to convert between floats and strings (e.g.
rounding errors). There is no corresponding .setn discipline.
A search in the ast-open-archive repo reveals that the getn
discipline was quietly added in version 2009-08-21 93t+, with not
even a mention in the RELEASE file.
The one available mention on the internet is this old thread:
https://www.mail-archive.com/ast-users@research.att.com/msg00601.html
Apparently a setn discipline *was* planned, but never implemented.
getn discipline functions may also crash in several ways. I've been
unsuccessful at solving all the crashes, particularly as one of
them is intermittent. This should not be in the 1.0 release.
Further discussion: https://github.com/ksh93/ksh/issues/435
This now allows our little make utility to run when compiled with
AddressSanitizer on gcc on Linux; it no longer aborts the build on
exit with a complaint about (very small) memory leaks.
The change in .github/workflows/ci.yml (use 'bin/package test' to
run the iffe regression tests as well) caused the GitHub workflow
to fail immediately with a syntax error. This is because iffe.tst
is a ksh93 script, but the runner does not have a ksh93 installed
as a system package.
As of e08ca80d, package prefers a known-good system shell over the
newly compiled shell to stop builds from failing when I break ksh,
making it convenient to fix it. But that change should not apply to
the regression tests; they should use the newly compiled shell.
src/cmd/INIT/package.sh, bin/package:
- Set KEEP_SHELL to 2 when given a SHELL=* argument.
- Before running the regression tests, override the known-good
shell with the compiled one if $KEEP_SHELL < 2, ensuring that all
tests, including the iffe ones, are run with the compiled shell.
This allows 'bin/package test' to run if the known-good shell is
not a ksh93, while testing that the compiled shell successfully
runs iffe. (re: e08ca80d)
- Standardise 'test' command use: -a and -o are deprecated.
- Remove some more unused cruft. (re: 6137b99a)
src/cmd/ksh93/Mamfile:
- Do not override SHELL when running shtests; this is now set
correctly in 'bin/package test'.
src/cmd/INIT/rt.sh:
- Removed. This regression test output filter was only used with
nmake, which we deleted. (re: 2940b3f5, 6cc2f6a0, aa601a39)
