Improve TON build scripts and some tests (#855)

* fix macOS github actions

* fix android tonlib GH action;

* fixing wasm GH action

* strip binaries

* fix randomly failing ubuntu and wasm GH actions

* fix randomly failing ubuntu and wasm GH actions

* revert some changes

* adding more nix scripts and automated native build scripts;
debug static ton compilation

* minor fix

* do not use pkg_config if path specified

* move wasm script, run with sudo action script

* weird, fixing - cp: missing destination file operand after 'assembly/native/build-ubuntu-20.04-shared.sh'

* weird, fixing - cp: missing destination file operand after 'assembly/native/build-ubuntu-20.04-shared.sh'

* ok

* some adjustments for android and win builds

* some adjustments for android and win builds

* moving stripping inside the build script

* access rights handling; adding simple binaries' tests

* make lite-client-common, fift-lib and smc-envelope deliberately static;
add -a (artifacts) flag to build scripts;

* minor wasm build fix;
create separate tonlib android build script;
remove outdated __has_trivial_copy(T)

* add windows build - WIP

* adjust android build;
improve win build;

* adjust sodium paths for android build; use proper compiler for windows build;

* add github windows build auxiliary file

* adjust wasm build

* add portable ubuntu build

* exclude some unstable tests for some time

* compile portable binaries on ubuntu-20.04

* exclude some unstable tests

* include static gsl

* restart builds

* restart builds

* restart builds

* remove libreadline, gsl and blas dependencies in linux build

* add macos build script

* install missing autoconf in macos builds

* enable all tests and see what fails

* enable win tests and restart others

* enable win tests and fix test-smartcont.cpp

* enable win tests

* use clang-16 on mac builds, add blockchain-explorer for ubuntu builds, add portable macos build

* move sudo part outside a build scripts

* move sudo part outside a build scripts

* run llvm install with sudo

* remove libgnutls28-dev before ubuntu static compilation, include blockchain-explorer into artifacts;
remove warning: definition of implicit copy constructor for 'Stat' is deprecated because it has a user-declared copy assignment operator [-Wdeprecated-copy]

* rework wrong decision, put back system gnutls, but compile libmicrohttpd with --disable-https

* add jenkins pipeline sceleton

* WIP jenkins pipeline sceleton

* WIP jenkins pipeline changes

* WIP jenkins pipeline: add stage timout, zip and group artifacts

* WIP jenkins pipeline: macos portable build fix

* WIP jenkins pipeline: wording

* WIP jenkins pipeline: add android tonlib

* WIP jenkins pipeline: add was binaries

* WIP jenkins pipeline: add TOTAL_MEMORY 1.5gb to funcfiftlib wasm linking

* WIP jenkins pipeline: add nix build on linux aarch64

* WIP jenkins pipeline: funcfiftlib compilation fails that 16mb mem is not enough, increase to 32mb

* WIP jenkins pipeline: enable test in nix build

* WIP jenkins pipeline: add linux x86-64 nix build

* WIP jenkins pipeline: include libs in nix build

* WIP jenkins pipeline: include libs in nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: include mac nix build

* WIP jenkins pipeline: nix linux arm64 with openssl 1.1 for now

* WIP jenkins pipeline: working ubuntu arm64 libtonjson

* WIP jenkins pipeline: working ubuntu arm64 libtonjson + minor fix

* WIP jenkins pipeline: working ubuntu arm64 libtonjson + minor fix 2

* WIP jenkins pipeline: merry christmas

* WIP jenkins pipeline: merry christmas 2

* WIP jenkins pipeline: remove native static builds

* WIP jenkins pipeline: enable more tests

* WIP jenkins pipeline: zip artifacts better

* WIP jenkins pipeline: get rid of path in the final zip

* WIP jenkins pipeline: minor fix, include lib and smartcont folders

* WIP jenkins pipeline: minor fix, include lib and smartcont folders into nix artifacts also

* WIP jenkins pipeline: minor fix

* WIP jenkins pipeline: minor fix

* adjust github actions for new nix builds

* cleanup

* cleanup

* cleanup

* cleanup

* rename libtonlibjson.so.0.5 to libtonlibjson.so

* Add TON build instructions to README.md

* simplify

* fix test-tonlib-offline

* set timeout per test of 300 sec

* set timeout per test of 600 sec for non nix builds

* increase test timeout to 900 sec; minor changes

* use MS VS 2022 for win TON compilation; update README.md

* use MS VS 2022 for win TON compilation; update README.md

* change path to MSVC in github workflow

* change path to MSVC in groovy pipeline

* compile ton on win, with msvc 2022 community and enterprise versions

* minor fixes

* improve network tests

* remove TON compilation against macos-11 github runner

* add `choco feature enable -n allowEmptyChecksums` since pkg-config-lite-0.28-1 does not have a checksum

* abort win compilation if 3pp can't be downloaded

* increase test timeout to 30 min

* improving test-catchain

assembly/wasm/fift-func-wasm-build-ubuntu.sh

@@ -0,0 +1,159 @@
+# The script builds funcfift compiler to WASM
+
+# Execute these prerequisites first
+# sudo apt update
+# sudo apt install -y build-essential git make cmake ninja-build clang libgflags-dev zlib1g-dev libssl-dev \
+#                    libreadline-dev libmicrohttpd-dev pkg-config libgsl-dev python3 python3-dev python3-pip \
+#                    nodejs libsecp256k1-dev libsodium-dev automake libtool
+
+# wget https://apt.llvm.org/llvm.sh
+# chmod +x llvm.sh
+# sudo ./llvm.sh 16 all
+
+with_artifacts=false
+
+while getopts 'a' flag; do
+  case "${flag}" in
+    a) with_artifacts=true ;;
+    *) break
+       ;;
+  esac
+done
+
+export CC=$(which clang-16)
+export CXX=$(which clang++-16)
+export CCACHE_DISABLE=1
+
+cd ../..
+rm -rf openssl zlib emsdk secp256k1 libsodium build
+echo `pwd`
+
+git clone https://github.com/openssl/openssl.git
+cd openssl
+git checkout checkout openssl-3.1.4
+./config
+make -j16
+OPENSSL_DIR=`pwd`
+cd ..
+
+git clone https://github.com/madler/zlib.git
+cd zlib
+ZLIB_DIR=`pwd`
+cd ..
+
+git clone https://github.com/bitcoin-core/secp256k1.git
+cd secp256k1
+./autogen.sh
+SECP256K1_DIR=`pwd`
+cd ..
+
+git clone https://github.com/jedisct1/libsodium --branch stable
+cd libsodium
+SODIUM_DIR=`pwd`
+cd ..
+
+mkdir build
+cd build
+cmake -GNinja -DCMAKE_BUILD_TYPE=Release \
+-DCMAKE_CXX_STANDARD=17 \
+-DOPENSSL_FOUND=1 \
+-DOPENSSL_ROOT_DIR=$OPENSSL_DIR \
+-DOPENSSL_INCLUDE_DIR=$OPENSSL_DIR/include \
+-DOPENSSL_CRYPTO_LIBRARY=$OPENSSL_DIR/libcrypto.so \
+-DOPENSSL_SSL_LIBRARY=$OPENSSL_DIR/libssl.so \
+-DTON_USE_ABSEIL=OFF ..
+
+test $? -eq 0 || { echo "Can't configure TON build"; exit 1; }
+
+ninja fift smc-envelope
+
+test $? -eq 0 || { echo "Can't compile fift "; exit 1; }
+
+rm -rf *
+
+cd ..
+
+git clone https://github.com/emscripten-core/emsdk.git
+cd emsdk
+./emsdk install 3.1.19
+./emsdk activate 3.1.19
+EMSDK_DIR=`pwd`
+ls $EMSDK_DIR
+
+. $EMSDK_DIR/emsdk_env.sh
+export CC=$(which emcc)
+export CXX=$(which em++)
+export CCACHE_DISABLE=1
+
+cd ../openssl
+
+make clean
+emconfigure ./Configure linux-generic32 no-shared no-dso no-engine no-unit-test
+sed -i 's/CROSS_COMPILE=.*/CROSS_COMPILE=/g' Makefile
+sed -i 's/-ldl//g' Makefile
+sed -i 's/-O3/-Os/g' Makefile
+emmake make depend
+emmake make -j16
+test $? -eq 0 || { echo "Can't compile OpenSSL with emmake "; exit 1; }
+
+cd ../zlib
+
+emconfigure ./configure --static
+emmake make -j16
+test $? -eq 0 || { echo "Can't compile zlib with emmake "; exit 1; }
+ZLIB_DIR=`pwd`
+
+cd ../secp256k1
+
+emconfigure ./configure --enable-module-recovery
+emmake make -j16
+test $? -eq 0 || { echo "Can't compile secp256k1 with emmake "; exit 1; }
+
+cd ../libsodium
+
+emconfigure ./configure --disable-ssp
+emmake make -j16
+test $? -eq 0 || { echo "Can't compile libsodium with emmake "; exit 1; }
+
+cd ../build
+
+emcmake cmake -DUSE_EMSCRIPTEN=ON -DCMAKE_BUILD_TYPE=Release \
+-DZLIB_FOUND=1 \
+-DZLIB_LIBRARIES=$ZLIB_DIR/libz.a \
+-DZLIB_INCLUDE_DIR=$ZLIB_DIR \
+-DOPENSSL_FOUND=1 \
+-DOPENSSL_ROOT_DIR=$OPENSSL_DIR \
+-DOPENSSL_INCLUDE_DIR=$OPENSSL_DIR/include \
+-DOPENSSL_CRYPTO_LIBRARY=$OPENSSL_DIR/libcrypto.a \
+-DOPENSSL_SSL_LIBRARY=$OPENSSL_DIR/libssl.a \
+-DCMAKE_TOOLCHAIN_FILE=$EMSDK_DIR/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake \
+-DCMAKE_CXX_FLAGS="-sUSE_ZLIB=1" \
+-DSECP256K1_FOUND=1 \
+-DSECP256K1_INCLUDE_DIR=$SECP256K1_DIR/include \
+-DSECP256K1_LIBRARY=$SECP256K1_DIR/.libs/libsecp256k1.a \
+-DSODIUM_INCLUDE_DIR=$SODIUM_DIR/src/libsodium/include \
+-DSODIUM_LIBRARY_RELEASE=$SODIUM_DIR/src/libsodium/.libs/libsodium.a \
+-DSODIUM_USE_STATIC_LIBS=ON ..
+
+test $? -eq 0 || { echo "Can't configure TON with emmake "; exit 1; }
+cp -R ../crypto/smartcont ../crypto/fift/lib crypto
+
+emmake make -j16 funcfiftlib func fift tlbc emulator-emscripten
+
+test $? -eq 0 || { echo "Can't compile TON with emmake "; exit 1; }
+
+if [ "$with_artifacts" = true ]; then
+  echo "Creating artifacts..."
+  cd ..
+  rm -rf artifacts
+  mkdir artifacts
+  ls build/crypto
+  cp build/crypto/fift* artifacts
+  cp build/crypto/func* artifacts
+  cp build/crypto/tlbc* artifacts
+  cp build/emulator/emulator-emscripten* artifacts
+  cp -R crypto/smartcont artifacts
+  cp -R crypto/fift/lib artifacts
+fi
+
+