Fix line continuation within command substitutions

In command substitutions of the $(standard) and ${ shared state; }
form, backslash line continuation is broken.

Reproducer:

	echo $(
	echo one two\
	three
	)

Actual output (ksh93, all versions):

	one two\ three

Expected output (every other shell, POSIX spec):

	one twothree

src/cmd/ksh93/sh/lex.c: sh_lex(): case S_REG:
- Do not skip new-line joining if we're currently processing a
  command substitution of one of these forms (i.e., if the
  lp->lexd.dolparen level is > 0).

Background info/analysis:

comsub() is called from sh_lex() when S_PAR is the current state.
In src/cmd/ksh93/data/lexstates.c, we see that S_PAR is reached in
the ST_DOL state table at index 40. Decimal 40 is ( in ASCII. So,
the previous skipping of characters was done according to the
ST_DOL state table, and the character that stopped it was (. This
means we have $(.

Alternatively, comsub() may be called from sh_lex() by jumping to
the do_comsub label. In brief, that is the case when we have ${.

Regardless of which it is from the two, comsub() is now called from
sh_lex(). In comsub(), lp->lexd.dolparen is incremented at the
beginning and decremented at the end. Between them, we see that
sh_lex() is called. So, lp->lexd.dolparen in sh_lex() indicates the
depth of nesting $( or ${ statements we're in. Thus, it is also the
number of comsub() invocations seen in a backtrace taken in
sh_lex().

The codepath for `...` is different (and never had this bug).

Co-authored by: Martijn Dekker <martijn@inlv.org>
Resolves: https://github.com/ksh93/ksh/issues/367

NEWS

@@ -3,11 +3,17 @@ For full details, see the git log at: https://github.com/ksh93/ksh/tree/1.0
 
 Any uppercase BUG_* names are modernish shell bug IDs.
 
+2022-05-21:
+
+- Fixed a bug, present since the beginning of ksh93, that broke backslash line
+  continuation within $(standard) and ${ shared-state; } command substitutions.
+  Thanks to atheik for the analysis and the fix. (BUG_CSUBLNCONT)
+
 2022-05-20:
 
 - Fixed an ancient bug that caused a spurious syntax error when using double
   quotes within a `backtick` command substitution within double quotes.
-  Thanks to atheik for the analysis and the fix.
+  Thanks to atheik for the analysis and the fix. (BUG_CSUBBTQUOT)
 
 2022-03-10:
 

src/cmd/ksh93/include/version.h

@@ -21,7 +21,7 @@
 
 #define SH_RELEASE_FORK	"93u+m"		/* only change if you develop a new ksh93 fork */
 #define SH_RELEASE_SVER	"1.0.0-beta.2"	/* semantic version number: https://semver.org */
-#define SH_RELEASE_DATE	"2022-05-20"	/* must be in this format for $((.sh.version)) */
+#define SH_RELEASE_DATE	"2022-05-21"	/* must be in this format for $((.sh.version)) */
 #define SH_RELEASE_CPYR	"(c) 2020-2022 Contributors to ksh " SH_RELEASE_FORK
 
 /* Scripts sometimes field-split ${.sh.version}, so don't change amount of whitespace. */

src/cmd/ksh93/sh/lex.c

@@ -671,8 +671,8 @@ int sh_lex(Lex_t* lp)
 				if(mode==ST_BEGIN)
 				{
 				do_reg:
-					/* skip new-line joining */
+					/* skip new-line joining if not called from comsub() */
-					if(c=='\\' && fcpeek(0)=='\n')
+					if(c=='\\' && fcpeek(0)=='\n' && !lp->lexd.dolparen)
 					{
 						sh.inlineno++;
 						fcseek(1);

src/cmd/ksh93/tests/quoting2.sh

@@ -295,5 +295,28 @@ got="${var:+'}text between expansions${var:+'}"
 got=$(eval 'foo="`: "^Exec(\[[^]=]*])?="`"' 2>&1) || err_exit "Backtick command substitutions can't nest double quotes" \
 	"(got $(printf %q "$got"))"
 
+# ======
+# https://github.com/ksh93/ksh/issues/367
+exp='one twothree'
+got=$(
+echo one two\
+three
+)
+[[ $got == "$exp" ]] || err_exit "Line continuation broken within standard command substitution" \
+	"(expected $(printf %q "$exp"), got $(printf %q "$got"))"
+got=${
+echo one two\
+three
+}
+[[ $got == "$exp" ]] || err_exit "Line continuation broken within shared-state command substitution" \
+	"(expected $(printf %q "$exp"), got $(printf %q "$got"))"
+# backticks did not have this bug but let's test them anyway
+got=`
+echo one two\
+three
+`
+[[ $got == "$exp" ]] || err_exit "Line continuation broken within backtick command substitution" \
+	"(expected $(printf %q "$exp"), got $(printf %q "$got"))"
+
 # ======
 exit $((Errors<125?Errors:125))