[droidvenc] Performance tuning?

asked 2020-05-10 19:59:06 +0200

takimata
496 ●8 ●16 ●20

My final goal is to use the camera in a Xperia X as a webcam on another computer. My current approach is to connect the phone via USB to the computer, enable developer mode to get a network connection between both and just transfer the data via UDP using a custom gstreamer pipeline written in C.

(This pipeline can't be composed on the command line for the reasons detailed in https://together.jolla.com/question/224002/internal-error-in-gstreamer-element-droidcamsrc/)

However, I get a latency of approx. 1-2 seconds, which is apparently unacceptable for a webcam (caching is disabled on the receiver side).

I also started my application with GST_TRACERS="latency(flags=pipeline)" GST_DEBUG=GST_TRACER:7 which, I think, also tells me a latency of 1 sec, so the problem must be on the sender side.

This is my pipeline (which is based on the example at https://github.com/sailfishos/gst-droidcamsrc/tree/master/test):

#define CAMERA_SRC "droidcamsrc"

TestPipeline *pipeline = test_pipeline_new(argc, argv);
if (!pipeline) {
    return 1;
}

pipeline->src = gst_element_factory_make(CAMERA_SRC, NULL);
if (!pipeline->src) {
    g_printerr("Failed to create %s element", CAMERA_SRC);
    return 1;
}

g_object_set(pipeline->src, "mode", 2, NULL); // mode: (1: image or 2: video)
g_object_set(pipeline->src, "camera-device", 0, NULL); // (0): primary,  (1): secondary


GstElement *fakesink = gst_element_factory_make("fakesink", NULL);
GstElement *droidvenc = gst_element_factory_make("droidvenc", NULL);
g_object_set(droidvenc, "target-bitrate", 1 * 1024 * 512, NULL);

GstElement *rtppay = gst_element_factory_make("rtpmp4vpay", NULL);
g_object_set(rtppay, "config-interval", 2, NULL);

GstElement *udpsink = gst_element_factory_make("udpsink", NULL);
g_object_set(udpsink, "host", dstIpAddress, NULL);
g_object_set(udpsink, "port", 5004, NULL);

gst_bin_add_many(GST_BIN(pipeline->pipeline), pipeline->src, fakesink, droidvenc, rtppay, udpsink, NULL);

GstCaps *caps;
caps = gst_caps_from_string("video/x-raw(memory:DroidVideoMetaData),framerate=30/1");

if (gst_element_link_pads_filtered(pipeline->src, "vidsrc", droidvenc, "sink", caps) != TRUE) {
    g_printerr("Failed to link droidvenc");
    return 1;
}
gst_caps_unref(caps);

caps = gst_caps_from_string("video/mpeg, mpegversion=4");
if (gst_element_link_pads_filtered(droidvenc, "src", rtppay, "sink", caps) != TRUE) {
    g_printerr("Failed to link rtppay");
    return 1;
}
gst_caps_unref(caps);

// ! connecting the viewfinder source is important, otherwise it will stop due to pushing a buffer into an unlinked pad !
// -> Apparently everything is designed for a mandatory use of a viewfinder.
if (gst_element_link_pads(pipeline->src, "vfsrc", fakesink, "sink") != TRUE) {
    g_printerr("Failed to link fakesink");
    return 1;
}

if (gst_element_link_many(rtppay, udpsink, NULL) != TRUE) {
    g_printerr("Failed to link elements");
    return 1;
}

See https://github.com/sailfishos/gst-droidcamsrc/blob/master/test/test.h#L28 for the definition of TestPipeline.

What am I doing wrong? The CPU usage is at ~19% while this pipeline is running but https://sailfishos.org/wiki/Multimedia says that the encoding should be hardware accelerated... Maybe the Xperia X with Sailfish does not support MPEG encoding in hardware at all?