Changes:
- Fixed two xtrace test failures introduced in commit cfc8744c.
- The definition of _use_ntfork_tcpgrp in xec.c is now dependent on
SHOPT_SPAWN being defined (re: 8e9ed5be).
- Removed many unnecessary newlines and fixed various typos.
Building ksh with the tcc (tinycc) compiler failed as of glibc
commit 5d98a7da. The NEWS file in that commit adds:
+* When _DYNAMIC_STACK_SIZE_SOURCE or _GNU_SOURCE are defined,
+ PTHREAD_STACK_MIN is no longer constant and is redefined to
+ sysconf(_SC_THREAD_STACK_MIN).
The tcc build failure seeminly had nothing to do with that --
however, deleting the PTHREAD_STACK_MIN entry and its dependent
THREAD_STACK_MIN entry from conf.tab fixes the build failure.
For reasons unknown, gcc didn't have a problem with it. However,
these config identifiers aren't used anywhere in the ast codebase
(including the full ast-open-history repo) so it should be fine to
just get rid of them; ksh is not and will not be threaded.
NOTE: To build ksh with tcc, you need to build the latest tcc code
from <https://repo.or.cz/tinycc>. The tcc release packages in OS
distributions are typically too old and will not work.
Resolves: https://github.com/ksh93/ksh/issues/437
Thanks to @JohnoKing for the report.
This commit implements support for the glibc 2.35
posix_spawn_file_actions_addtcsetpgrp_np(3) extension[2][3],
updating spawnveg(3) to use the new function for setting the
terminal group. This was done with the intention of improving
performance in interactive shells without reintroducing previous
race conditions[4][5].
[1]: https://sourceware.org/pipermail/libc-alpha/2022-February/136040.html
[2]: https://sourceware.org/git/?p=glibc.git;a=commit;h=342cc934
[3]: https://sourceware.org/git/?p=glibc.git;a=commit;h=6289d28d
[4]: https://github.com/ksh93/ksh/issues/79
[5]: https://www.mail-archive.com/ast-developers@research.att.com/msg00717.html
src/cmd/ksh93/sh/path.c:
- Tell spawnveg(3) to set the terminal process group when launching
a child process in an interactive shell.
src/cmd/ksh93/sh/xec.c:
- If posix_spawn_file_actions_addtcsetpgrp_np(3) is available,
allow use of spawnveg(3) (via sh_ntfork()) even with job control
active.
- sh_ntfork(): Reimplement most of the SIGTSTP handling code
removed in commit 66c37202.
src/lib/libast/comp/spawnveg.c,
src/lib/libast/misc/procopen.c,
src/lib/libast/features/sys:
- Add support for posix_spawn_file_actions_addtcsetpgrp_np(3).
- Allow spawnveg to set the terminal process group when pgid == 0.
This was necessary to avoid race conditions when using the new
function.
src/lib/libast/features/lib:
- Detect posix_spawn_file_actions_addtcsetpgrp_np(3).
- Do not detect an OS spawnveg(3). With the API changes to spawnveg
in this pull request ksh probably can't use the OS's spawnveg
function anymore. (That's assuming anything else even provides a
spawnveg function to begin with, which is unlikely.)
src/lib/libast/features/api,
src/cmd/ksh93/include/defs.h:
- Bump libast version (20220101 => 20220201) due to the spawnveg(3)
API change.
src/lib/libast/man/spawnveg.3:
- Document the changes to spawnveg(3) in the corresponding man
page. Currently, it will only use the new tcfd argument if
posix_spawn_file_actions_addtcsetpgrp_np(3) is supported. This
could also be implemented for the fork(2) fallback, but for now
I've avoided changing that since actually using it in the fork
code would likely require a lot of hackery to avoid attempting
tcsetpgrp with vfork (the behavior of tcsetpgrp after vfork is
not portable) and would only benefit systems that don't have
posix_spawn and vfork (I can't recall any off the top of my head
that would fall under that category).
- Updated the man page to account for spawnveg's change in
behavior.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
This commit implements support for POSIX_SPAWN_SETSID in
spawnveg(3). The fork/vfork fallback for spawnveg already attempts
to use setsid in the manner described by the man page, so the
posix_spawn implementation should also do so.
src/lib/libast/comp/spawnveg.c:
- Add support for POSIX_SPAWN_SETSID to the posix_spawn version of
spawnveg.
- Minor extra: Remove dead code that's never used (the
_lib_posix_spawn < 2 code block is inside of _lib_posix_spawn >
1, plus when it's manually enabled by changing the previous #if
condition you'll find it causes many regression test failures (at
least on OpenBSD)).
src/lib/libast/man/spawnveg.3:
- Document that spawnveg cannot make the new process a session
leader if the operating system doesn't support POSIX_SPAWN_SETSID
and the new process was created using posix_spawn.
Turns out that the standards macros set by features/standards (such
as _GNU_SOURCE for Linux or _DARWIN_SOURCE for macOS) were still
*not* included for most C source files! Instead, they were
selectively included for some files only, sometimes via
FEATURE/standards (the output of features/standards), sometimes
via ast_standards.h which is copied from FEATURE/standards.
Consequently, there were still inconsistencies in the system header
interfaces exposed on Linux, macOS, Solaris, et al. It's no wonder
it sometimes took so much hackery to keep everything building.
Of course, making this consistent had to break things somewhere.
Breakage occurred on 32-bit Linux due to a lot of ugly hackery
involving direct use of internal GNU types like off64_t and
functions like fseek64(). This is now all removed and they are
activated by setting the appropriate feature macro instead, so
these types and functions can be used with their standard names
(off_t, fseek, etc.)
Before committing I've tested these changes on the following
i386/x86_64 systems: Linux (glibc 32 and 64 bit, musl libc 64 bit),
Solaris (32 and 64 bit), illumos (32 and 64 bit), FreeBSD (64 bit),
macOS (64 bit), Cygwin (32 bit), and Haiku (64 bit).
(Note: ast_standards.h is copied from FEATURE/standards, whereas
ast_common.h is copied from FEATURE/common.)
src/lib/libast/include/ast_std.h,
src/lib/libast/stdio/stdhdr.h:
- Include <ast_standards.h> first. This should cause all the AST
and dependent code (such as ksh) to get the standards macros.
src/lib/libast/features/standards:
- For GNU (glibc), #define _FILE_OFFSET_BITS 64 to get large file
support with 64-bit offsets.
- Stop GNU and Cygwin <string.h> form defining the GNU version of
basename(3); on Cygwin, that declaration conflicts with the AST
version (and with POSIX) by using a const char* argument instead
of char*. It is deactivated by defining the macro 'basename' (as
'basename'); this causes GNU string.h to consider it to be
already defined by the standard libgen.h header.
All other changed files:
- Remove direct use of *64* types and functions and a lot of
related hackery.
These are minor things I accumulated over the last month or so.
Notable changes:
src/lib/libast/features/api,
src/lib/libast/misc/state.c,
src/lib/libast/comp/conf.tab,
src/cmd/ksh93/include/defs.h:
- Bump internal libast version to 20220101L. We've made a few
additions to the API, at least pathicase (see 71934570, ca3ec200)
and astconf_long (see c2ac69b2), so this should have been done
already. This also updates '/opt/ast/bin/getconf _AST_VERSION'.
- Use AST_VERSION instead of outdated _AST_VERSION.
- In state.c, use AST_VERSION instead of hardcoding the version.
src/cmd/ksh93/sh/xec.c:
- Remove 'restorefd' variable, unused as of 42becab6.
- Remove 'cmdrecurse' function and SH_RUNPROG macro; this was once
used by a few libcmd commands, but ast-open-archive reveals it's
unused as of ast 1999-12-25.
src/cmd/ksh93/sh/*.c:
- Where available, use e_dot instead of "." for consistency; it is
defined as an extern so we might as well use it.
src/cmd/ksh93/tests/*.sh:
- When reporting signal names in fails, include the SIG prefix.
- Fix a broken process hang test in subshell.sh.
src/lib/libast/man/sfdisc.3:
- Removed. The interfaces described here never made it out of AT&T;
they do not exist in any libast version in ast-open-archive.
Resolves: https://github.com/ksh93/ksh/issues/426
This commit makes various different improvements to the documentation:
- sh.1: Backported (with changes) mandoc warning fixes from ksh2020
for the ksh93(1) man page: <https://github.com/att/ast/pull/1406>
- Removed unnecessary spaces at the end of lines to fix a few other
mandoc warnings.
- Fixed various typos and capitalization errors in the documentation.
- ANNOUNCE: Document the addition of the ${.sh.pid} variable
(re: 9de65210).
- libast/man/str*: Update the man pages for the libast str* functions
to improve how accurately each function is described.
- ksh93/README: Update regression test/compatibility notes to include
OpenBSD 7.0, FreeBSD 13.0 and WSL running Ubuntu 20.04.
- Change a few places to store the return value from strlen in a
size_t variable rather than signed int.
- comp/setlocale.c: To avoid confusion of two separate variables named
lang, the function local variable has been renamed to langidx.
This takes another step towards cleaning up the build system. We
now do not even pretend to be theoretically compatible with
pre-1989 K&R C compilers or with C++ compilers. In practice, this
had already been broken for many years due to bit rot.
Commit 46593a89 already removed the license handling enormity that
depended on proto, so now we can cleanly remove it altogether. But
we do need to leave some backwards compatibility stubs to keep the
build system compatible with older AST code; it should remain
possible to build older ksh versions with the current build system
(the bin/ and src/cmd/INIT/ directories) for testing purposes.
So as of now there is no more __MANGLE__d rubbish in your generated
header files. This is only about a quarter of a century overdue...
This commit also includes a huge amount of code cleanup to remove
thousands of unused K&R C fallbacks and other cruft, particularly
in libast. This code base should now be a little easier to
understand for people who are familiar with a modern(ish) C
standard.
ratz is now also removed; this was a standalone and simplified 2005
version of gunzip. As of 6137b99a, none of our code uses it, even
theoretically. And the real g(un)zip is now everywhere.
src/cmd/INIT/proto.c, src/cmd/INIT/ratz.c:
- Removed.
COPYRIGHT:
- Remove zlib license; this only applied to ratz.
bin/package, src/cmd/INIT/package.sh:
- Related cleanups.
- Unset LC_ALL before invoking a new shell, respecting the user's
locale again and avoiding multibyte character corruption on the
command line.
src/cmd/INIT/proto.sh:
- Add stub for backwards compatibility with Mamfiles that depend on
proto. It does nothing but pass input without modification and is
now installed as the new arch/*/bin/proto by src/cmd/INIT/Mamfile.
src/cmd/INIT/iffe.sh:
- Ignore the proto-related -e (--package) and -p (--prototyped)
options; keep parsing them for backwards compatibility.
- Trim the macros passed to every test to their standard C
versions, removing K&R C and C++ versions. These are now
considered to be for backwards compatibility only.
src/cmd/INIT/iffe.tst:
- Remove proto(1) mangling code.
By the way, iffe can be regression-tested as follows:
$ bin/package use # set up environment in a child shell
$ regress src/cmd/INIT/iffe.tst
$ exit # leave package environment
src/cmd/INIT/make.probe, src/cmd/INIT/probe.win32:
- Remove code to handle C++.
src/lib/libast/features/common:
- As in iffe.sh above, trim macros designed for compatibility with
C++ and ancient C compilers to their standard C versions and
comment that they are for backwards compatibility with AST code.
This is needed to keep all the old ast and ksh code compiling.
src/cmd/ksh93/sh/init.c,
src/cmd/ksh93/sh/name.c:
- Clarify libshell ABI compatibility function versions of macros.
A "proto workaround" comment in the original code mislead me into
thinking this had something to do with the removed proto(1), but
it's unrelated. Call the workaround macro BYPASS_MACRO instead.
src/cmd/ksh93/include/defs.h:
- sh_sigcheck() macro: allow &sh as an argument: parenthesise shp.
src/cmd/ksh93/sh/nvtype.c:
- Remove unused nv_mkstruct() function. (re: d0a5cab1)
**/features/*:
- Remove obsolete iffe 'set prototyped' option.
**/Mamfile:
- Remove all references to the ast/prototyped.h header.
- Remove all use of the proto command. Simply copy instead.
*** 850-ish source files: ***
- Remove all '#pragma prototyped' directives.
- Remove all C++ compat code conditional upon defined(__cplusplus).
- Remove all use of the _ARG_ macro, which on standard C expands to
its argument:
#define _ARG_(x) x
(on K&R C, it expanded to nothing)
- Remove all use of _BEGIN_EXTERNS_ and _END_EXTERNS_ macros (empty
on standard C; this was for C++ compatibility)
- Reduce all #if __STD_C (standard code) #else (K&R code) #endif
blocks to the standard code only, without use of the macro.
- Same for _STD_ macro which seems to have had the same function.
- Change all instances of 'Void_t' to standard 'void'.
This commit fixes an issue with how ksh was obtaining the value of
NGROUPS_MAX. On some systems this setting can be changed (e.g., on
illumos adding 'set ngroups_max=32' to /etc/system then rebooting
changes NGROUPS_MAX from 16 to 32). Ksh was using NGROUPS_MAX with
the assumption it's a static value, which could cause issues on
systems where it isn't static. This bugfix is inspired by the one
from <b1362c3a5>, although it
has been expanded a bit to account for OPEN_MAX as well.
src/cmd/ksh93/sh/init.c, src/lib/libcmd/fds.c:
- Rename the getconf() macro to astconf_long() and move it to ast.h
to prevent redundancy. Other sections of the code have been
modified to use this macro for astconf() to account for
dynamic settings.
- An equivalent macro for unsigned long values (astconf_ulong) has
been added.
- Prefer sysconf(3) where available. It has better performance as it
returns a numeric value directly instead of via string
conversion.
- The astconf_long and astconf_ulong macros have been documented in
the ast(3) man page.
List of changes:
- Fixed some -Wuninitialized warnings and removed some unused variables.
- Removed the unused extern for B_login (re: d8eba9d1).
- The libcmd builtins and the vmalloc memfatal function now handle
memory errors with 'ERROR_SYSTEM|ERROR_PANIC' for consistency with how
ksh itself handles out of memory errors.
- Added usage of UNREACHABLE() where it was missing from error handling.
- Extend many variables from short to int to prevent overflows (most
variables involve file descriptors).
- Backported a ksh2020 patch to fix unused value Coverity issues
(https://github.com/att/ast/pull/740).
- Note in src/cmd/ksh93/README that ksh compiles with Cygwin on
Windows 10 and Windows 11, albeit with many test failures.
- Add comments to detail some sections of code. Extensive list of
commits related to this change:
ca2443b5, 7e7f1372, 2db9953a, 7003aba4, 6f50ff64, b1a41311,
222515bf, a0dcdeea, 0aa9e03f, 61437b27, 352e68da, 88e8fa67,
bc8b36fa, 6e515f1d, 017d088c, 035a4cb3, 588a1ff7, 6d63b57d,
a2f13c19, 794d1c86, ab98ec65, 1026006d
- Removed a lot of dead ifdef code.
- edit/emacs.c: Hide an assignment to avoid a -Wunused warning. (See
also https://github.com/att/ast/pull/753, which removed the assignment
because ksh2020 removed the !SHOPT_MULTIBYTE code.)
- sh/nvdisc.c: The sh_newof macro cannot return a null pointer because
it will instead cause the shell to exit if memory cannot be allocated.
That makes the if statement here a no-op, so remove it.
- sh/xec.c: Fixed one unused variable warning in sh_funscope().
- sh/xec.c: Remove a fallthrough comment added in commit ed478ab7
because the TFORK code doesn't fall through (GCC also produces no
-Wimplicit-fallthrough warning here).
- data/builtins.c: The cd and pwd man pages state that these builtins
default to -P if PATH_RESOLVE is 'physical', which isn't accurate:
$ /opt/ast/bin/getconf PATH_RESOLVE
physical
$ mkdir /tmp/dir; ln -s /tmp/dir /tmp/sym
$ cd /tmp/sym
$ pwd
/tmp/sym
$ cd -P /tmp/sym
$ pwd
/tmp/dir
The behavior described by these man pages isn't specified in the ksh
man page or by POSIX, so to avoid changing these builtin's behavior
the inaccurate PATH_RESOLVE information has been removed.
- Mamfiles: Preserve multi-line errors by quoting the $x variable.
This fix was backported from 93v-.
(See also <a7e9cc82>.)
- sh/subshell.c: Remove set but not used sp->errcontext variable.
src/lib/libast/features/standards:
- Add heuristic (u_long availability) for systems that hide rather
than reveal functionality in the presence of _POSIX_SOURCE, etc.
- Define _DARWIN_C_SOURCE, like _GNU_SOURCE, to enable the full
range of definitions on macOS systems.
- Due to the above, remove MACH (macOS)-specific hack.
- These changes ported from https://github.com/att/ast/pull/1492 -
thanks to Lev Kujawski (@lkujaw). His PR indicates that this
fixes the standards macros on UnixWare, too. Therefore, no longer
exclude UnixWare from standards macros (re: ff70c27f).
src/lib/libast/comp/conf.sh:
- Promote the 'op' member in Conf_t (struct Conf_s) from short to
int. This allows some Darwin/macOS values, now exposed, to fit
that would otherwise be truncated, namely:
_CS_DARWIN_USER_CACHE_DIR 65538
_CS_DARWIN_USER_DIR 65536
_CS_DARWIN_USER_TEMP_DIR 65537
Thus, the following AST getconf values are now correct on macOS:
$ /opt/ast/bin/getconf | grep ^DARWIN
DARWIN_USER_CACHE_DIR=/var/folders/nx/(REDACTED)/C/
DARWIN_USER_DIR=/var/folders/nx/(REDACTED)/0/
DARWIN_USER_TEMP_DIR=/var/folders/nx/(REDACTED)/T/
src/lib/libast/features/tty:
- Include <sys/ioctl.h> if available. This silences a compiler
warning in src/lib/libast/misc/procopen.c about an invalid
implicit declaration of ioctl(2).
This commit ports over two of Andy Fiddaman's bugfixes to conf.sh
on illumos:
- The compiler isn't passed on to an invocation of iffe. The bugfix is
from this commit: <63563232>
- The getconf builtin is missing several parameters on illumos.
Reproducer:
$ /opt/ast/bin/getconf ADDRESS_WIDTH
getconf: Invalid argument (ADDRESS_WIDTH) # Should output '64'
This bug occurs because conf.sh expects GNU sed and fails to work
properly with other sed implementations. The bugfix and original bug
report can be found here:
https://www.illumos.org/issues/14044ba443cfd
Ksh segfaults on M1 Macs, apparently because Apple's compiler
optimizer can't cope with overriding bzero(3) with libast's
implementation (though it's nothing more than "memset(b, 0, n);").
Apple has disabled libast's bzero function since 2018-12-04:
https://opensource.apple.com/source/ksh/ksh-27/patches/src__lib__libast__comp__omitted.c.diff.auto.html
src/lib/libast/comp/omitted.c:
- Only fall back to the libast bzero function if the OS lacks an
implementation of bzero.
- Remove the bzero undef since this fallback is only reached if the
OS doesn't have bzero.
NEWS:
- Add compat info for macOS on ARM64. This notes that macOS
Monterey can now compile and run ksh, although there is one
regression test failure:
builtins.sh[345]: printf %H: invalid UTF-8 characters
(expected %3F%C2%86%3F%3F%3F; got %3F%C2%86%3F%3Fv%3F%3F)
Thanks to @DesantBucie for the report and the testing.
Resolves: https://github.com/ksh93/ksh/issues/329
This commit fixes three problems with getconf pathbound builtin:
1. The -l/--lowercase option did not change all variable names to
lower case.
2. The -q/--quote option now quotes all string values. Previously,
it only quoted string values that had a space or other
non-shellsafe character.
3. The -c/--call, -n/--name and -s/--standard options matched all
variable names provided by 'getconf -a', even if none were
actual matches.
Additionally, references to the confstr and sysconf functions have
been updated to reference section 3 of the man pages instead of
section 2.
src/lib/libast/port/astconf.c:
- Previously, only values that had spaces in them were quoted. Change
that behavior to quote all string values by using the FMT_ALWAYS
flag. Bug report: https://github.com/att/ast/issues/1173
- Not all variable names were printed in lowercase by 'getconf -l'.
Fix it by adding a few missing instances of fmtlower.
Bug report: https://github.com/att/ast/issues/1171
- Add the missing code to the '#if _pth_getconf_a' block to handle
-c/-n/-s while parsing the OS's native 'getconf -a' output. This
approach reuses code for name matching from other parts of
astconflist(). Resolves: https://github.com/ksh93/ksh/issues/279
src/lib/libcmd/getconf.c:
- Update the documentation to note the -q flag only quotes strings.
src/cmd/ksh93/tests/bulitins.sh:
- Add regression tests for the getconf bugs fixed in this commit.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
This is an update to one of Red Hat's patches. The strdup change is
from CentOS:
https://git.centos.org/rpms/ksh/blob/c8s/f/SOURCES/ksh-20120801-annocheck.patch
The reason why gcc (and also clang) optimize out the null check is
because the glibc string.h header gives 's' a nonnull attribute (in
other words, this is a glibc compatibility bug, not a compiler bug).
Clang gives the following informative warning when compiling strdup:
> /home/johno/GitRepos/KornShell/ksh/src/lib/libast/string/strdup.c:66:10: warning: nonnull parameter 's' will evaluate to 'true' on
> return (s && (t = oldof(0, char, n = strlen(s) + 1, 0))) ? (char*)memcpy(t, s, n) : (char*)0;
> ^ ~~
> /usr/include/string.h:172:35: note: declared 'nonnull' here
> __THROW __attribute_malloc__ __nonnull ((1));
> ^
> /usr/include/sys/cdefs.h:303:44: note: expanded from macro '__nonnull'
> # define __nonnull(params) __attribute__ ((__nonnull__ params))
The proper fix is to rename the function in strdup.c to
'_ast_strdup'. This avoids the string.h conflict and fixes the Red
Hat bug. I've also made a similar change to getopt.c, since clang
was throwing a nonnull warning there as well.
src/lib/libast/features/map.c (which generates FEATURE/map which is
indirectly included by everything) is updated to always map getopt
to _ast_getopt and strdup to _ast_strdup.
This commit adds an UNREACHABLE() macro that expands to either the
__builtin_unreachable() compiler builtin (for release builds) or
abort(3) (for development builds). This is used to mark code paths
that are never to be reached.
It also adds the 'noreturn' attribute to functions that never
return: path_exec(), sh_done() and sh_syntax(). The UNREACHABLE()
macro is not added after calling these.
The purpose of these is:
* to slightly improve GCC/Clang compiler optimizations;
* to fix a few compiler warnings;
* to add code clarity.
Changes of note:
src/cmd/ksh93/sh/io.c: outexcept():
- Avoid using __builtin_unreachable() here since errormsg can
return despite using ERROR_system(1), as shp->jmplist->mode is
temporarily set to 0. See: https://github.com/att/ast/issues/1336
src/cmd/ksh93/tests/io.sh:
- Add a regression test for the ksh2020 bug referenced above.
src/lib/libast/features/common:
- Detect the existence of either the C11 stdnoreturn.h header or
the GCC noreturn attribute, preferring the former when available.
- Test for the existence of __builtin_unreachable(). Use it for
release builds. On development builds, use abort() instead, which
crahses reliably for debugging when unreachable code is reached.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
This commit adds '/* FALLTHROUGH */' comments to fix many
GCC warnings when compiling with -Wimplicit-fallthrough.
Additionally, the existing fallthrough comments have been
changed for consistency.
I grepped for #include changes in all the commits and compared
that to the changes in the Mamfiles. I found 7 commits that don't
update the Mamfiles with the appropriate dependencies while
adding #includes, as I only learned how this works after having
worked with this code for some time.
This commit adds the missing Mamfile updates for the
corresponding #include changes in the following commits:
06e721c3, 65d363fd, 70fc1da7, 79d19458, b1a41311, bb4d6a2e,
db71b3ad, and this commit.
Additionally:
src/lib/libast/comp/setlocale.c:
- Change include errno.h to error.h to use EILSEQ fallback if
needed; remove corresponding #ifdef (re: 4dcf5c50, 71bfe028).
src/cmd/ksh93/Mamfile:
- Fix a broken dependency on libast FEATURE/float (re: 72968eae).
We can't use 'prev' for a file that was not mentioned before in
the same Mamfile, we have to use a 'make'...'done' on the first
mention. Add subdependencies matching those in libast/Mamfile.
This allows ksh to be compiled with versions of tcc that define
__dso_handle in libtcc1.a, i.e., versions as of this commit:
https://repo.or.cz/tinycc.git/commit/dd60b20c
Older versions of tcc still fail to compile ksh, although now they
fail after reaching the libdll feature test. I'm not sure if fixing
that is feasible since even if I hack out the failing libdll
feature test, ksh fails to link with a '__dso_handle' error.
src/lib/libast/comp/atexit.c,
src/lib/libast/features/lib,
src/lib/libast/vmalloc/vmexit.c:
- From what I've been able to gather the only OSes with support
for on_exit are Linux and SunOS 4. However, on_exit takes two
arguments, so the macro that defines it as taking one argument
is incorrect. Since Solaris (SunOS 5) no longer has this call
and the macro breaks on Linux, the clean fix is to remove it
(atexit(3) is used instead).
src/lib/libast/include/ast.h:
- When compiling with tcc on FreeBSD, pretend to be gcc 2.95.3
instead of gcc 9.3.0. This stops /usr/include/math.h from
activating gcc 3.0+ math compiler builtins that don't exist on
tcc, while still identifying as gcc which is needed to avoid
other FreeBSD system header breakage.
src/cmd/builtin/Mamfile,
src/cmd/builtin/features/pty,
src/lib/libdll/Mamfile,
src/lib/libdll/features/dll:
- tcc forbids combining the -c compiler flag with -l* linker flags.
Use the -lm flag in the iffe feature tests instead of the
Mamfiles. This avoids iffe combining -lm with the -c flag.
src/lib/libast/vmalloc/malloc.c:
- Fix failure to compile with -D_std_malloc.
This patch is from OpenSUSE:
https://build.opensuse.org/package/view_file/shells/ksh/ksh93-malloc-hook.dif
As it turns out tcc needs this change to build ksh with
-D_std_malloc, so it has been applied.
Co-authored-by: Martijn Dekker <martijn@inlv.org>
Resolves: https://github.com/ksh93/ksh/issues/232
In various places in libast and libcmd there are preprocessor
fallbacks like this, for systems that don't define all the commonly
used errno value IDs:
#ifndef ENOSYS
#define ENOSYS EINVAL
#endif
and many others. It is better to have these all in one place so
they are not duplicated and we don't risk inconsistencies when
adding new code.
src/lib/libast/include/error.h includes the OS's <errno.h>, so it
is the logical file to move all these fallbacks into.
Quite possibly there is no remotely current system that needs any
of these, but they won't do any harm either.
Most files already use <error.h> directly or indirectly. Four
needed new #include <error.h> directives to use the fallbacks if
needed. The libast Mamfile is updated to make those files depend on
that header.
These are minor fixes I've accumulated over time. The following
changes are somewhat notable:
- Added a missing entry for 'typeset -s' to the man page.
- Add strftime(3) to the 'see also' section. This and the date(1)
addition are meant to add onto the documentation for 'printf %T'.
- Removed the man page the entry for ksh reading $PWD/.profile on
login. That feature was removed in commit aa7713c2.
- Added date(1) to the 'see also' section of the man page.
- Note that the 'hash' command can be used instead of 'alias -t' to
workaround one of the caveats listed in the man page.
- Use an 'out of memory' error message rather than 'out of space'
when memory allocation fails.
- Replaced backticks with quotes in some places for consistency.
- Added missing documentation for the %P date format.
- Added missing documentation for the printf %Q and %p formats
(backported from ksh2020: https://github.com/att/ast/pull/1032).
- The comments that show each builtin's options have been updated.
Most of these changes remove unused variables, functions and labels
to fix -Wunused compiler warnings. Somewhat notable changes:
src/cmd/ksh93/bltins/print.c:
- Removed the unused 'neg' variable.
Patch from ksh2020: https://github.com/att/ast/pull/725
src/cmd/ksh93/bltins/sleep.c:
- Initialized ns to fix three -Wsometimes-uninitialized warnings.
src/cmd/ksh93/edit/{emacs,vi}.c:
- Adjust strncpy size to fix two -Wstringop-truncation warnings.
src/cmd/ksh93/include/shell.h:
- The NOT_USED macro caused many -Wunused-value warnings,
so it has been replaced with ksh2020's macro:
19d0620a
src/cmd/ksh93/sh/expand.c:
- Removed an unnecessary 'ap = ' since 'ap' is never read
between stakseek and stakfreeze.
src/cmd/ksh93/edit/vi.c: refresh():
- Undef this function's 'w' macro at the end of it to stop it
potentially interfering with future code changes.
src/cmd/ksh93/sh/nvdisc.c,
src/lib/libast/misc/magic.c,
src/lib/libast/regex/regsubexec.c,
src/lib/libast/sfio/sfpool.c,
src/lib/libast/vmalloc/vmbest.c:
- Fixed some indentation to silence -Wmisleading-indentation
warnings.
src/lib/libast/include/ast.h:
- For clang, now only suppress hundreds of -Wparentheses warnings
as well as a few -Wstring-plus-int warnings.
Clang's -Wparentheses warns about things like
if(foo = bar())
which assigns to foo and checks the assigned value.
Clang wants us to change this into
if((foo = bar()))
Clang's -Wstring-plus-int warns about things like
"string"+x
where x is an integer, e.g. "string"+3 represents the string
"ing". Clang wants us to change that to
"string"[3]
The original versions represent a perfectly valid coding style
that was common in the 1980s and 1990s and is not going to change
in this historic code base. (gcc does not complain about these.)
Co-authored-by: Martijn Dekker <martijn@inlv.org>
Compilers like GCC are capable of optimizing away calls like
pow(1,inf), which caused the IEEE compliance feature test within
libast to incorrectly succeed on platforms with non-IEEE behavior.
This is arguably a bug within GCC, as floating point optimizations
should never alter the behavior of code unless IEEE compliance is
explicitly disabled via a flag like -ffast-math. Programs in which
only some calls to pow are optimized away are liable to severely
malfunction.
Thanks to Martijn Dekker for pointing this issue out and the kind
operators of polarhome.com for permitting me gratis use of their
Unix systems.
src/lib/libast/comp/omitted.c:
- Add IEEE compliant functions that wrap powf, pow, and powl.
src/lib/libast/features/float:
- Look for powf, pow, and powl in the C library.
- For compilers that do the right thing, like the native toolchains
of Solaris and UnixWare, use lightweight macros to wrap the pow
functions.
- Use a volatile function pointer through which to access the C
library's pow function in an attempt to defeat code optimization.
- For these overzealous compilers, define pow to _ast_pow so that
the same technique can be used within the above functions.
This makes ksh 93u+m build on the following system:
$ uname -a
QNX qnx 6.5.0 2010/07/09-14:44:03EDT x86pc x86
Thanks to polarhome.com for providing the QNX shell account.
There are a number of regressions left to work out:
arrays.sh[636]: copying a large array fails
bracket.sh[129]: /tmp/ksh93.shtests.1753215026.6923/bracket.C/original should be older than /tmp/ksh93.shtests.1753215026.6923/bracket.C/newer
bracket.sh[132]: /tmp/ksh93.shtests.1753215026.6923/bracket.C/newer should be newer than /tmp/ksh93.shtests.1753215026.6923/bracket.C/original
builtins.sh[683]: real_t1 not found after parent directory renamed in subshell
functions.sh[1023]: cannot handle comsub depth > 256 in function
io.sh[252]: <# not working for pipes
io.sh[337]: read -n3 from pipe not working
io.sh[346]: read -n3 from fifo failed -- expected 'a', got 'abc'
io.sh[349]: read -n1 from fifo failed -- expected 'b', got 'd'
io.sh[379]: should have timed out
io.sh[380]: line1 should be 'prompt1: '
io.sh[381]: line2 should be line2
io.sh[382]: line3 should be 'prompt2: '
io.sh[406]: LC_ALL=C read -n2 from pipe 'a bcd' failed -- expected 'a bcd', got 'ab cd'
io.sh[406]: LC_ALL=C.UTF-8 read -n2 from pipe 'a bcd' failed -- expected 'a bcd', got 'ab cd'
jobs.sh[86]: warning: skipping subshell job control test due to non-compliant 'ps'
pty.sh[105]: POSIX sh 026(C): line 120: expected "(Stopped|Suspended)", got EOF
pty.sh[128]: POSIX sh 028(C): line 143: expected "(Stopped|Suspended) \(SIGTTIN\)", got EOF
pty.sh[151]: POSIX sh 029(C): line 166: expected "(Stopped|Suspended) \(SIGTTOU\)", got EOF
signal.sh[310]: kill -TERM $$ failed, required termination by signal 'EXIT'
signal.sh[310]: kill -VTALRM $$ failed, required termination by signal 'EXIT'
signal.sh[310]: kill -PIPE $$ failed, required termination by signal 'EXIT'
(The io.sh failures mean libast sfpkrd() is not working.)
src/lib/libast/obsolete/spawn.c:
- Removed. Didn't compile due to wrong number of arguments to
spawnve(2), but is obsolete and unused.
src/lib/libast/comp/localeconv.c:
- The initialisation of two static 'struct lconv' variables was
done in a way that depended on OS headers declaring the struct
members in a certain order. This holds on most systems, but not
on QNX, and POSIX does not actually specify the order at all:
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/locale.h.html
So each member must be initialised by name. But C89 does not
support initialising struct members by name, so we have to do it
using an initialiser function that simply assigns the values.
src/lib/libast/comp/spawnveg.c:
- Fix for systems without either P_DETACH or _P_DETACH.
src/lib/libast/features/vmalloc,
src/lib/libast/vmalloc/vmmopen.c,
src/lib/libast/Mamfile:
- Add test for sys/shm.h header. If it doesn't exist, as it doesn't
on QNX, use the stub vmmapopen() as the real one won't compile.
(Mamfile: Add dependency on FEATURE/vmalloc to vmmopen.c.)
src/lib/libast/vmalloc/malloc.c:
- Remove superfluous externs that are already provided by either
AST or system headers. The 'void cfree' extern caused a build
failure on QNX because cfree() is of type int on QNX.
src/lib/libast/comp/conf.tab:
- Remove check for _map_spawnve; src/lib/libast/RELEASE says it was
removed.
This reverts an OpenSUSE patch ("libast/comp/conf.sh: apply limits
detection fixes for Linux"). It broke the build on Alpine Linux
with the musl C library (see also e245856f).
src/cmd/INIT/iffe.sh:
- Fix "standard system directories" for the cmd test, which were
hardcoded as bin, /etc, /usr/bin, /usr/etc, /usr/ucb. That's both
unportable and antiquated. Replace this with the path output by
'getconf PATH'.
- Add fixes from modernish for 'getconf PATH' output to compensate
for bugs/shortcomigns in NixOS and AIX. Source:
9e4bf5eb/lib/modernish/aux/defpath.sh
Ref.: https://github.com/NixOS/nixpkgs/issues/65512
src/lib/libast/comp/conf.tab: PATH:
- Add the NixOS and AIX default path fixes here too; this fixes
'command -p' and the builtin 'getconf PATH' on these systems.
bin/package, src/cmd/INIT/package.sh:
- Re-support being launched with just the command name 'package' in
the command line (if the 'package' command is in $PATH). At least
one other script in the build system does this. (re: 6cc2f6a0)
- Go back three levels (../../..) if we were invoked from
arch/*/bin/package, otherwise we won't find src/cmd/ksh93/SHOPT.sh.
Same idea as in the referenced commit.
src/lib/libast/comp/conf.sh:
- If an output header file has not changed after rerunning conf.sh,
still update the output file's timestamp using touch(1) to signal
that the test has already been run.
AIX on ibm.risc comes with a broken version of ksh88 as /bin/sh
where the following causes breakage in the parser (spurious syntax
errors):
(set -o posix) 2>/dev/null && set -o posix
However, prefixing it with 'command' (while keeping the subshell)
circumvents the problem. So, why not.
(command set -o posix) 2>/dev/null && set -o posix
This now makes ksh build on DragonFly BSD.
bin/package, src/cmd/INIT/package.sh:
- DragonFly also needs the -lm hack for LDFLAGS.
src/cmd/ksh93/sh/main.c, src/cmd/ksh93/tests/basic.sh:
- fixargs() doesn't work on DragonFly either
(re: 9b7c392a, 159fb9ee, cefe087d).
The following are backported from:
https://github.com/att/ast/issues/26#issuecomment-313927854https://github.com/att/ast/pull/19
src/lib/libast/comp/setlocale.c:
- Add missing #include <errno.h> since errno is used.
src/lib/libast/features/standards:
- Do not set any standards macros (_POSIX_SOURCE etc) on FreeBSD or
DragonflyBSD; they disable too much functionality on those.
src/lib/libast/features/wchar:
- Set _STDFILE_DECLARED on DragonFly, too.
src/lib/libast/include/sfio.h, src/lib/libast/include/sfio_t.h,
src/lib/libast/sfio/_sfopen.c, src/lib/libast/sfio/sfclrlock.c,
src/lib/libast/sfio/sfhdr.h, src/lib/libast/sfio/sfnew.c,
src/lib/libast/sfio/sfset.c:
- Rename SF_* macros to SFIO_* to avoid a conflict with system
headers.
src/lib/libast/string/strexpr.c:
- Rename error() to err() to avoid a conflict.
That patch didn't work for non-gcc, non-clang compilers -- at least
Solaris Studio cc. It doesn't prefix error messages with "error:".
As a result, it caused the build to fail on Solaris with native cc.
src/lib/libast/comp/conf.sh:
- Use a sed formula that should catch error messages prefixed by
"line xx:" while still removing warnings and suggestions. This
works on at least clang, gcc, Solaris Studio cc.
conf.sh checks for undefined symbols by parsing compiler output and
looking for strings of capital letters and underscores. Modern gcc
produces suggestions for replacement variables too, for example:
error: '_SC_CLOCKRES_MIN' undeclared here (not in a function); did you mean _POSIX_CLOCKRES_MIN?
_SC_CLOCKRES_MIN,
^~~~~~~~~~~~~~~~
_POSIX_CLOCKRES_MIN
This causes good variables to be excluded along with bad, causing differences
between the builtin and system getconf commands.
src/lib/libast/comp/conf.sh:
- Only use lines containing 'error:' and ignore everything starting
from 'did you mean:'. (Note this scripts sets the locale to C.)
Patch from @citrus-it:
061a4b1da1
This upstreams a Solaris patch:
https://github.com/oracle/solaris-userland/blob/master/components/ksh93/patches/050-CR7065478.patch
src/lib/libast/comp/setlocale.c:
- Add wide_wctomb() wrapper for wctomb(3). It changes an invalid
character (wctomb returns -1) to a single byte with length 1.
- set_ctype(): Use wide_wctomb() instead of wctomb(3) as the
conversion discipline function (ast.mb_conv). Effectively this
means there are no invalid characters. Perhaps this is necessary
for compatibility with ASCII. Sadly, no public info available.
All changed files:
- Put the shell in POSIX mode if it has an '-o posix' option.
- Remove nonsense disabling 'set -x' on bash. It's not broken.
bin/package, src/cmd/INIT/package.sh:
- Add check blocking native zsh mode (e.g., "$path" conflicts).
Using a 'sh -> zsh' symlink works, so recommend that.
- Remove old ksh93 version check for a supposed conflict with
libcmd. It was broken; it would revert to /bin/sh, but on illumos
distributions, /bin/sh is a ksh93 of a version that is supposedly
affected. It builds fine anyway.
- Rewrite checksh() to incorporate the shell compatibility checks
that were previously in two different places in 'package'.
bin/ignore, src/cmd/INIT/ignore.sh,
bin/silent, src/cmd/INIT/silent.sh:
- Change bad check for a full POSIX 'export' command (no, $RANDOM
has nothing to do with that) with a proper feature test.
As of aa4669ad, astconf("PATH") is implemented as a hardcoded AST
configuration variable that always has a value, instead of one that
falls back on the OS. Its value is now obtained from the OS (with a
fallback) at configure time and not at runtime. This means that any
fallback for astconf("PATH") is now never used.
src/cmd/ksh93/data/msg.c,
src/cmd/ksh93/include/shell.h:
- Remove e_defpath[]. (The path "/bin:/usr/bin:" made no sense as a
default path anyway, as the final empty element is wrong: default
utilities should never be sought in the current working dir.)
src/cmd/ksh93/sh/path.c,
src/lib/libast/path/pathbin.c:
- abort() if astconf("PATH") returns null.
src/lib/libast/comp/conf.tab: PATH:
- If no 'getconf' utility can be found, use a fallback path that
finds more utilities by also searching in 'sbin' directories.
On some systems, this is needed to find chown(1).
src/cmd/ksh93/sh.1:
- Update doc re default path.
It was working on Solaris 11.3, but there were still problems
building on Solaris 11.4 with GCC (as on the evaluation VM
downloaded directly from Oracle):
1. ksh immediately segfaulted. Experimenting with the compiler
flags Oracle uses revealed that we need to define _XPG6 for ksh
not to segfault. Why is a mystery.
2. The default path logic used by 'command -p' and the 'getconf
PATH' builtin command was still broken: the result did not
include any of the /usr/xpg?/bin directories where the standard
POSIX utilities actually live. Testing shows that the result of
the C language probe 'confstr(_CS_PATH,name,length)' is broken
on Solaris (it only yields the paths to the historic
non-standard utilities, defeating the purpose) unless _XPG7 is
defined; but the latter makes ksh segfault again. So another
solution is needed.
src/cmd/INIT/package.sh, bin/package:
- Add another hack to add the -D_XPG6 flag to CCFLAGS if we're
running SunOS aka Solaris. (I've tried to add a 'cc.sol11' script
to src/cmd/INIT/ instead, but for some reason that I just don't
have time to figure out, the INIT system ignores that on Solaris
with gcc, so this is the only way I could come up with. Any
patches for less hacky alternatives would be welcome.)
src/lib/libast/comp/conf.sh:
- Sanitise the code for finding the best 'getconf' utility.
src/lib/libast/comp/conf.tab: PATH:
- Since the C-languge getconf(_CS_PATH,...) is broken on Solaris
11.4, replace the C language probe with a shell script probe that
uses the external 'getconf' utility.
- To avoid ksh overriding the result of this probe with the result
of its own getconf(_CS_PATH,...) call, which would make Solaris
use the wrong value again, specify this as an AST configuration
entry instead of a POSIX entry. This should be good enough for
all systems; the OS 'getconf' utility should be reliable and the
default path value is constant for each OS, so can be hardcoded.
src/cmd/ksh93/tests/builtins.sh:
- Add another 'sleep .1' to the 'sleep -s 31' test as it was still
intermittently failing on Solaris and possibly other systems.
Solaris, Illumos distributions, and NetBSD need LDFLAGS set to link
explicitly to libm, otherwise, due to as-yet unknown reasons, the
src/lib/libdll/features/dll fails to write a valid header file and
compilation fails due to unknown identifiers such as Dllscan_t.
This commit adds the flag on those systems.
NixOS is a Linux distro that uses very different paths from the
usual Unix conventions (though it's POSIX compliant), and the
regression tests still needed a lot of tweaks to be compatible.
src/cmd/INIT/package.sh, bin/package:
- On SunOS (Solaris and illumos distros) and NetBSD, add '-lm' to
LDFLAGS before compiling.
src/cmd/INIT/mamprobe.sh, bin/mamprobe,
src/cmd/INIT/execrate.sh, bin/execrate:
- Instead of only in /bin, /usr/bin, /sbin and /usr/sbin, search
utilities in the path given by the OS 'getconf PATH', and use the
user's original $PATH as a fallback.
src/cmd/ksh93/tests/*.sh:
- Miscellaneous portability fixes, mainly elimination of unportable
hardcoded paths to commands.
- basic.sh: Remove test for 'time' keyword millisecond precision.
It was racy and could fail depending on system and system load.
This fixes 'command -p' for systems where getconf(1) lives
somewhere other than in /bin or /usr/bin, i.e. NixOS.
src/lib/libast/comp/conf.tab:
- To determine the default path value for AST 'getconf PATH' and
'command -p', compile a small C program to get the correct local
default path value (_CS_PATH) from the operating system so it
gets hardcoded in the ksh binary. This eliminates the need to to
invoke 'getconf PATH' to get this value, which fixes a catch-22
problem on systems where getconf(1) exists somewhere other than
/bin or /usr/bin.
Support for the long-dead 3DFS userland versioning file system was
already removed from ksh93 (although I'd overlooked some minor
things), but libast still supported it. This removes that too.
src/lib/libast/include/fs3d.h,
src/lib/libast/man/fs3d.3,
src/lib/libast/misc/fs3d.c:
- Removed.
bin/package,
src/cmd/INIT/package.sh:
- Remove attempted use of removed vpath builtin.
src/cmd/ksh93/*:
- Remove minor 3dfs vestiges.
src/lib/lib{ast,cmd,coshell}/*:
- Remove code supporting 3dfs.
If the processing of a multibyte character was interrupted in UTF-8
locales, e.g. by reading just one byte of a two-byte character 'ü'
(\303\274) with a command like:
print -nr $'\303\274' | read -n1 g
then the shellquoting algorithm was corrupted in such a way that
the final quote in simple single-quoted string was missing. This
bug may have had other, as yet undiscovered, effects as well. The
problem was with corrupted multibyte character processing and not
with the shell-quoting routine sh_fmtq() itself.
Full trace and discussion at: https://github.com/ksh93/ksh/issues/5
(which is also an attempt to begin to understand the esoteric
workings of the libast mb* macros that process UTF-8 characters).
src/lib/libast/comp/setlocale.c: utf8_mbtowc():
- If called from the mbinit() macro (i.e. if both pointer
parameters are null), reset the global multibyte character
synchronisation state variable. This fixes the problem with
interrupted processing leaving an inconsistent state, provided
that mbinit() is called before processing multibyte characters
(which it is, in most (?) places that do this). Before this fix,
calling mbinit() in UTF-8 locales was a no-op.
src/cmd/ksh93/sh/string.c: sh_fmtq():
- Call mbinit() before potentially processing multibyte characters.
Testing suggests that this could be superfluous, but at worst,
it's harmless; better be sure.
src/cmd/ksh93/tests/builtins.sh:
- Add regression test for shellquoting with 'printf %q' after
interrupting the processing of a multibyte characeter with
'read -n1'. This test only fails in a UTF-8 locale, e.g. when
running: bin/shtests -u builtins SHELL=/buggy/ksh-2012-08-01
Fixes#5.
- fix to buildin.c to prevent compiler from optimizing it out
- fix to realpath.c to handle case where 'path' is NULL
- fix to emacs.c (I think from dgk)
- fix to file.c (I think from gsf)
- fix to tail.c (I think from gsf)
- fix to code setting the malloc init hook (for vmalloc)
- changed memcpy to memmove in several places in sfio because src and dst
might overlap
- fixes to macros like FILE_defined to fix header file issues
- fixes to expr.h to handle macro definition issue
