The heartbeat of SRS is a timer that requests an HTTP URL. We can use
this heartbeat to report the necessary information for registering the
backend server with the proxy server.
```text
SRS(backend) --heartbeat---> Proxy server
```
A proxy server is a specialized load balancer for media servers. It
operates at the application level rather than the TCP level. For more
information about the proxy server, see issue #4158.
Note that we will merge this PR into SRS 5.0+, allowing the use of SRS
5.0+ as the backend server, not limited to SRS 7.0. However, the proxy
server is introduced in SRS 7.0.
It's also possible to implement a registration service, allowing you to
use other media servers as backend servers. For example, if you gather
information about an nginx-rtmp server and register it with the proxy
server, the proxy will forward RTMP streams to nginx-rtmp. The backend
server is not limited to SRS.
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
1. Do not create a source when mounting FLV because it may not unmount
FLV when freeing the source. If you access the FLV stream without any
publisher, then wait for source cleanup and review the FLV stream again,
there is an annoying warning message.
```bash
# View HTTP FLV stream by curl, wait for stream to be ready.
# curl http://localhost:8080/live/livestream.flv -v >/dev/null
HTTP #0 127.0.0.1:58026 GET http://localhost:8080/live/livestream.flv, content-length=-1
new live source, stream_url=/live/livestream
http: mount flv stream for sid=/live/livestream, mount=/live/livestream.flv
# Cancel the curl and trigger source cleanup without http unmount.
client disconnect peer. ret=1007
Live: cleanup die source, id=[], total=1
# View the stream again, it fails.
# curl http://localhost:8080/live/livestream.flv -v >/dev/null
HTTP #0 127.0.0.1:58040 GET http://localhost:8080/live/livestream.flv, content-length=-1
serve error code=1097(NoSource)(No source found) : process request=0 : cors serve : serve http : no source for /live/livestream
serve_http() [srs_app_http_stream.cpp:641]
```
> Note: There is an inconsistency. The first time, you can access the
FLV stream and wait for the publisher, but the next time, you cannot.
2. Create a source when starting to serve the FLV client. We do not need
to create the source when creating the HTTP handler. Instead, we should
try to create the source in the cache or stream. Because the source
cleanup does not unmount the HTTP handler, the handler remains after the
source is destroyed. The next time you access the FLV stream, the source
is not found.
```cpp
srs_error_t SrsHttpStreamServer::hijack(ISrsHttpMessage* request, ISrsHttpHandler** ph) {
SrsSharedPtr<SrsLiveSource> live_source;
if ((err = _srs_sources->fetch_or_create(r.get(), server, live_source)) != srs_success) { }
if ((err = http_mount(r.get())) != srs_success) { }
srs_error_t SrsBufferCache::cycle() {
SrsSharedPtr<SrsLiveSource> live_source = _srs_sources->fetch(req);
if (!live_source.get()) {
return srs_error_new(ERROR_NO_SOURCE, "no source for %s", req->get_stream_url().c_str());
}
srs_error_t SrsLiveStream::serve_http(ISrsHttpResponseWriter* w, ISrsHttpMessage* r) {
SrsSharedPtr<SrsLiveSource> live_source = _srs_sources->fetch(req);
if (!live_source.get()) {
return srs_error_new(ERROR_NO_SOURCE, "no source for %s", req->get_stream_url().c_str());
}
```
> Note: We should not create the source in hijack, instead, we create it
in cache or stream:
```cpp
srs_error_t SrsHttpStreamServer::hijack(ISrsHttpMessage* request, ISrsHttpHandler** ph) {
if ((err = http_mount(r.get())) != srs_success) { }
srs_error_t SrsBufferCache::cycle() {
SrsSharedPtr<SrsLiveSource> live_source;
if ((err = _srs_sources->fetch_or_create(req, server_, live_source)) != srs_success) { }
srs_error_t SrsLiveStream::serve_http(ISrsHttpResponseWriter* w, ISrsHttpMessage* r) {
SrsSharedPtr<SrsLiveSource> live_source;
if ((err = _srs_sources->fetch_or_create(req, server_, live_source)) != srs_success) { }
```
> Note: This fixes the failure and annoying warning message, and
maintains consistency by always waiting for the stream to be ready if
there is no publisher.
3. Fail the http request if the HTTP handler is disposing, and also keep
the handler entry when disposing the stream, because we should dispose
the handler entry and stream at the same time.
```cpp
srs_error_t SrsHttpStreamServer::http_mount(SrsRequest* r) {
entry = streamHandlers[sid];
if (entry->disposing) {
return srs_error_new(ERROR_STREAM_DISPOSING, "stream is disposing");
}
void SrsHttpStreamServer::http_unmount(SrsRequest* r) {
std::map<std::string, SrsLiveEntry*>::iterator it = streamHandlers.find(sid);
SrsUniquePtr<SrsLiveEntry> entry(it->second);
entry->disposing = true;
```
> Note: If the disposal process takes a long time, this will prevent
unexpected behavior or access to the resource that is being disposed of.
4. In edge mode, the edge ingester will unpublish the source when the
last consumer quits, which is actually triggered by the HTTP stream.
While it also waits for the stream to quit when the HTTP unmounts, there
is a self-destruction risk: the HTTP live stream object destroys itself.
```cpp
srs_error_t SrsLiveStream::serve_http(ISrsHttpResponseWriter* w, ISrsHttpMessage* r) {
SrsUniquePtr<SrsLiveConsumer> consumer(consumer_raw); // Trigger destroy.
void SrsHttpStreamServer::http_unmount(SrsRequest* r) {
for (;;) { if (!cache->alive() && !stream->alive()) { break; } // A circle reference.
mux.unhandle(entry->mount, stream.get()); // Free the SrsLiveStream itself.
```
> Note: It also introduces a circular reference in the object
relationships, the stream reference to itself when unmount:
```text
SrsLiveStream::serve_http
-> SrsLiveConsumer::~SrsLiveConsumer -> SrsEdgeIngester::stop
-> SrsLiveSource::on_unpublish -> SrsHttpStreamServer::http_unmount
-> SrsLiveStream::alive
```
> Note: We should use an asynchronous worker to perform the cleanup to
avoid the stream destroying itself and to prevent self-referencing.
```cpp
void SrsHttpStreamServer::http_unmount(SrsRequest* r) {
entry->disposing = true;
if ((err = async_->execute(new SrsHttpStreamDestroy(&mux, &streamHandlers, sid))) != srs_success) { }
```
> Note: This also ensures there are no circular references and no
self-destruction.
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
Edge FLV is not working because it is stuck in an infinite loop waiting.
Previously, there was no need to wait for exit since resources were not
being cleaned up. Now, since resources need to be cleaned up, it must
wait for all active connections to exit, which causes this issue.
To reproduce the issue, start SRS edge, run the bellow command and press
`CTRL+C` to stop the request:
```bash
curl http://localhost:8080/live/livestream.flv -v >/dev/null
```
It will cause edge to fetch stream from origin, and free the consumer
when client quit. When `SrsLiveStream::do_serve_http` return, it will
free the consumer:
```cpp
srs_error_t SrsLiveStream::do_serve_http(ISrsHttpResponseWriter* w, ISrsHttpMessage* r) {
SrsUniquePtr<SrsLiveConsumer> consumer(consumer_raw);
```
Keep in mind that in this moment, the stream is alive, because only set
to not alive after this function return:
```cpp
alive_viewers_++;
err = do_serve_http(w, r); // Free 'this' alive stream.
alive_viewers_--; // Crash here, because 'this' is freed.
```
When freeing the consumer, it will cause the source to unpublish and
attempt to free the HTTP handler, which ultimately waits for the stream
not to be alive:
```cpp
SrsLiveConsumer::~SrsLiveConsumer() {
source_->on_consumer_destroy(this);
void SrsLiveSource::on_consumer_destroy(SrsLiveConsumer* consumer) {
if (consumers.empty()) {
play_edge->on_all_client_stop();
void SrsLiveSource::on_unpublish() {
handler->on_unpublish(req);
void SrsHttpStreamServer::http_unmount(SrsRequest* r) {
if (stream->entry) stream->entry->enabled = false;
for (; i < 1024; i++) {
if (!cache->alive() && !stream->alive()) {
break;
}
srs_usleep(100 * SRS_UTIME_MILLISECONDS);
}
```
After 120 seconds, it will free the stream and cause SRS to crash
because the stream is still active. In order to track this potential
issue, also add an important warning log:
```cpp
srs_warn("http: try to free a alive stream, cache=%d, stream=%d", cache->alive(), stream->alive());
```
SRS may crash if got this log.
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
If SRS responds with this empty data packet, FFmpeg will receive an
empty stream, like `Stream #0:0: Data: none` in following logs:
```bash
ffmpeg -i rtmp://localhost:11935/live/livestream
# Stream #0:0: Data: none
# Stream #0:1: Audio: aac (LC), 44100 Hz, stereo, fltp, 30 kb/s
# Stream #0:2: Video: h264 (High), yuv420p(progressive), 768x320 [SAR 1:1 DAR 12:5], 212 kb/s, 25 fps, 25 tbr, 1k tbn
```
This won't cause the player to fail, but it will inconvenience the user
significantly. It may also cause FFmpeg slower to analysis the stream,
see #3767
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
When stopping the stream, it will wait for the HTTP Streaming to exit.
If the HTTP Streaming goroutine hangs, it will not exit automatically.
```cpp
void SrsHttpStreamServer::http_unmount(SrsRequest* r)
{
SrsUniquePtr<SrsLiveStream> stream(entry->stream);
if (stream->entry) stream->entry->enabled = false;
srs_usleep(...); // Wait for about 120s.
mux.unhandle(entry->mount, stream.get()); // Free stream.
}
srs_error_t SrsLiveStream::serve_http(ISrsHttpResponseWriter* w, ISrsHttpMessage* r)
{
err = do_serve_http(w, r); // If stuck in here for 120s+
alive_viewers_--; // Crash at here, because stream has been deleted.
```
We should notify http stream connection to interrupt(expire):
```cpp
void SrsHttpStreamServer::http_unmount(SrsRequest* r)
{
SrsUniquePtr<SrsLiveStream> stream(entry->stream);
if (stream->entry) stream->entry->enabled = false;
stream->expire(); // Notify http stream to interrupt.
```
Note that we should notify all viewers pulling stream from this http
stream.
Note that we have tried to fix this issue, but only try to wait for all
viewers to quit, without interrupting the viewers, see
https://github.com/ossrs/srs/pull/4144
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
1. Remove the srs_global_dispose, which causes the crash when still
publishing when quit.
2. Always call _srs_thread_pool->initialize for single thread.
3. Support `--signal-api` to send signal by HTTP API, because CLion
eliminate the signals.
---
Co-authored-by: Jacob Su <suzp1984@gmail.com>
By setting the env `ASAN_OPTIONS=halt_on_error=0`, we can ignore memory
leaks, see
https://github.com/google/sanitizers/wiki/AddressSanitizerFlags
By setting env `ASAN_OPTIONS=detect_leaks=0`, we can disable memory
leaking detection in parent process when forking for daemon.
For coroutine, we should use `__sanitizer_start_switch_fiber` which
similar to`VALGRIND_STACK_REGISTER`, see
https://github.com/google/sanitizers/issues/189#issuecomment-1346243598
for details. If not fix this, asan will output warning:
```
==72269==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x00016f638000; bottom 0x000106bec000; size: 0x000068a4c000 (1755627520)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
```
It will cause asan failed to get the stack, see
`research/st/asan-switch.cpp` for example:
```
==71611==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x000103600733 at pc 0x0001009d3d7c bp 0x000100b4bd40 sp 0x000100b4bd38
WRITE of size 1 at 0x000103600733 thread T0
#0 0x1009d3d78 in foo(void*) asan-switch.cpp:13
```
After fix this issue, it should provide the full stack when crashing:
```
==73437==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x000103300733 at pc 0x000100693d7c bp 0x00016f76f550 sp 0x00016f76f548
WRITE of size 1 at 0x000103300733 thread T0
#0 0x100693d78 in foo(void*) asan-switch.cpp:13
#1 0x100693df4 in main asan-switch.cpp:23
#2 0x195aa20dc (<unknown module>)
```
For primordial coroutine, if not set the stack by
`st_set_primordial_stack`, then the stack is NULL and asan can't get the
stack tracing. Note that it's optional and only make it fail to display
the stack information, no other errors.
---
Co-authored-by: john <hondaxiao@tencent.com>
Improvements for ST(State Threads):
1. ST: Use g++ for CXX compiler.
2. ST: Remove macros for clist.
3. ST: Remove macros for global thread and vp.
4. ST: Remove macros for vp queue operations.
5. ST: Remove macros for context switch.
6. ST: Remove macros for setjmp/longjmp.
7. ST: Remove macro for stack pad.
8. ST: Refine macro for valgrind.
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
New features for valgrind:
1. ST: Support /api/v1/valgrind for leaking check.
2. ST: Support /api/v1/valgrind?check=full|added|changed|new|quick
To use Valgrind to detect memory leaks in SRS, even though Valgrind
hooks are supported in ST, there are still many false positives. A more
reasonable approach is to have Valgrind report incremental memory leaks.
This way, global and static variables can be avoided, and detection can
be achieved without exiting the program. Follow these steps:
1. Compile SRS with Valgrind support: `./configure --valgrind=on &&
make`
2. Start SRS with memory leak detection enabled: `valgrind
--leak-check=full ./objs/srs -c conf/console.conf`
3. Trigger memory detection by using curl to access the API and generate
calibration data. There will still be many false positives, but these
can be ignored: `curl http://127.0.0.1:1985/api/v1/valgrind?check=added`
4. Perform load testing or test the suspected leaking functionality,
such as RTMP streaming: `ffmpeg -re -i doc/source.flv -c copy -f flv
rtmp://127.0.0.1/live/livestream`
5. Stop streaming and wait for SRS to clean up the Source memory,
approximately 30 seconds.
6. Perform incremental memory leak detection. The reported leaks will be
very accurate at this point: `curl
http://127.0.0.1:1985/api/v1/valgrind?check=added`
> Note: To avoid interference from the HTTP request itself on Valgrind,
SRS uses a separate coroutine to perform periodic checks. Therefore,
after accessing the API, you may need to wait a few seconds for the
detection to be triggered.
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
I did some preliminary code inspection. The two playback endpoints share
the same `SrsLiveStream` instance. After the first one disconnects,
`alive_` is set to false.
```
alive_ = true;
err = do_serve_http(w, r);
alive_ = false;
```
In the `SrsHttpStreamServer::http_unmount(SrsRequest* r)` function,
`stream->alive()` is already false, so `mux.unhandle` will free the
`SrsLiveStream`. This causes the other connection coroutine to return to
its execution environment after the `SrsLiveStream` instance has already
been freed.
```
// Wait for cache and stream to stop.
int i = 0;
for (; i < 1024; i++) {
if (!cache->alive() && !stream->alive()) {
break;
}
srs_usleep(100 * SRS_UTIME_MILLISECONDS);
}
// Unmount the HTTP handler, which will free the entry. Note that we must free it after cache and
// stream stopped for it uses it.
mux.unhandle(entry->mount, stream.get());
```
`alive_` was changed from a `bool` to an `int` to ensure that
`mux.unhandle` is only executed after each connection's `serve_http` has
exited.
---------
Co-authored-by: liumengte <liumengte@visionular.com>
Co-authored-by: winlin <winlinvip@gmail.com>
1. don't use static variable to store the result;
2. add more UT to handle the multi value and values with whitespaces;
related to #409216e569d823/trunk/src/app/srs_app_config.cpp (L71-L82)
`static SrsConfDirective* dir` removed, this static var here is to avoid
the memory leak, I add the `SrsConfDirective` instance to the `env_dirs`
directive container, which will destroy itself inside `SrsConfig`
destructor.
---------
Co-authored-by: winlin <winlinvip@gmail.com>
When unpublishing, the handler callback that will stop the coroutine:
```cpp
_can_publish = true;
handler->on_unpublish(req);
```
In this handler, the `http_unmount` will be called:
```cpp
void SrsHttpStreamServer::http_unmount(SrsRequest* r)
cache->stop();
```
In this `http_unmount` function, there could be context switching. In
such a situation, a new connection might publish the stream while the
unpublish process is freeing the stream, leading to a crash.
To prevent a new publisher, we should change the state only after all
handlers and hooks are completed.
---------
Co-authored-by: liumengte <liumengte@visionular.com>
Co-authored-by: winlin <winlinvip@gmail.com>
1. add on_connect & on_close directives to conf/full.conf;
2. let http_hooks env overwrite support multi values; e.g.
SRS_VHOST_HTTP_HOOKS_ON_CONNECT="http://127.0.0.1/api/connecthttp://localhost/api/connect"
related to
https://github.com/ossrs/srs/issues/1222#issuecomment-2170424703
Above comments said `http_hook` env may not works as expected, as I
found there are still has some issue in `http_hooks` env configuration,
but this PR may not target above problem.
---------
Co-authored-by: winlin <winlinvip@gmail.com>
Sorry this is another pull request with same intention. But there is
more variants of h264 und h265 codecs and I think it is good to support
them all.
---------
Co-authored-by: winlin <winlinvip@gmail.com>
Currently only libx264 ffmpeg encoder is supported. This pull request
add also h264_qsv. But maybe a more generic solution with oder encoders
would be useful to.
---------
Co-authored-by: winlin <winlinvip@gmail.com>
For #3369 to support an external powerful SIP server, do not use the
embedded SIP server of SRS.
For more information, detailed steps, system architecture, and
background explanation, please see
https://ossrs.net/lts/zh-cn/docs/v6/doc/gb28181#external-sip
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
Co-authored-by: winlin <winlinvip@gmail.com>
The session HLS manifest file lacks a terminating newline in the final
line.
This may cause strict players to reject it.
---------
Co-authored-by: winlin <winlinvip@gmail.com>
the html img tag occupy 20px size in safari.
427104f1da/trunk/research/players/rtc_player.html (L19)
> <img width="1011" alt="Screenshot 2024-04-17 at 9 17 07 AM"
src="79a4edf8-5bbf-4300-8817-039088f13283">
(ignore the img css warning: `auto\9;` it's another problem, I will file
another PR.)
but, the empty img tag just occupy 1px size in chrome. So I guess it's a
html compatible problem.
> <img width="608" alt="Screenshot 2024-04-17 at 9 46 33 AM"
src="40cb2eb6-3a6d-4bb7-9b17-51c5fd6d2272">
---------
`TRANS_BY_GPT4`
---------
Co-authored-by: winlin <winlinvip@gmail.com>
## Describe ##
http_remux feature support config `has_audio`, `has_video` &
`guess_has_av` prop.
282d94d7bb/trunk/src/app/srs_app_http_stream.cpp (L630-L632)
Take `http_flv` as example, `srs` can accept both RTMP streams with only
audio, only video or both audio and video streams. It is controlled by
above three properties.
But `guess_has_av` is not implemented by `http_ts`. The problem is that
if I want publish a RTMP stream with audio or video track, the
`has_audio` and `has_video`, which are default true/on, must to be
config to match the RTMP stream, otherwise the `mpegts.js` player can't
play the `http-ts` stream.
## How to reproduce ##
1. `export SRS_VHOST_HTTP_REMUX_HAS_AUDIO=on; export
SRS_VHOST_HTTP_REMUX_HAS_VIDEO=on; export
SRS_VHOST_HTTP_REMUX_GUESS_HAS_AV=on; ./objs/srs -c
conf/http.ts.live.conf`
2. publish rtmp stream without video: `ffmpeg -re -stream_loop -1 -i
srs/trunk/doc/source.200kbps.768x320.flv -vn -acodec copy -f flv
rtmp://localhost/live/livestream`
3. open chrome browser, open
`http://localhost:8080/players/srs_player.html?schema=http`, go to
`LivePlayer`, input URL: `http://localhost:8080/live/livestream.ts`,
click play.
4. the `http://localhost:8080/live/livestream.ts` can not play.
## Solution ##
Let `http-ts` support `guess_has_av`, `http-flv` already supported. The
`guess_has_av` default value is ture/on, so the `http-ts|flv` can play
any streams with audio, video or both.
---------
Co-authored-by: Winlin <winlinvip@gmail.com>
This not only silences a deprecation warning by docker build, but also
makes it consistent as the other ENV statement already uses the new
syntax.
---------
Co-authored-by: winlin <winlinvip@gmail.com>
1. Should always stop coroutine before close fd, see #511, #1784
2. When edge forwarder coroutine quit, always set the error code.
3. Do not unpublish if invalid state.
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
