We have moved to a new Sailfish OS Forum. Please start new discussions there.
4

[droidvenc] Performance tuning?

asked 2020-05-10 19:59:06 +0300

takimata gravatar image

My final goal is to use the camera in a Xperia X as a webcam on another computer. My current approach is to connect the phone via USB to the computer, enable developer mode to get a network connection between both and just transfer the data via UDP using a custom gstreamer pipeline written in C.

(This pipeline can't be composed on the command line for the reasons detailed in https://together.jolla.com/question/224002/internal-error-in-gstreamer-element-droidcamsrc/)

However, I get a latency of approx. 1-2 seconds, which is apparently unacceptable for a webcam (caching is disabled on the receiver side).

I also started my application with GST_TRACERS="latency(flags=pipeline)" GST_DEBUG=GST_TRACER:7 which, I think, also tells me a latency of 1 sec, so the problem must be on the sender side.

This is my pipeline (which is based on the example at https://github.com/sailfishos/gst-droidcamsrc/tree/master/test):

#define CAMERA_SRC "droidcamsrc"

TestPipeline *pipeline = test_pipeline_new(argc, argv);
if (!pipeline) {
    return 1;
}

pipeline->src = gst_element_factory_make(CAMERA_SRC, NULL);
if (!pipeline->src) {
    g_printerr("Failed to create %s element", CAMERA_SRC);
    return 1;
}

g_object_set(pipeline->src, "mode", 2, NULL); // mode: (1: image or 2: video)
g_object_set(pipeline->src, "camera-device", 0, NULL); // (0): primary,  (1): secondary


GstElement *fakesink = gst_element_factory_make("fakesink", NULL);
GstElement *droidvenc = gst_element_factory_make("droidvenc", NULL);
g_object_set(droidvenc, "target-bitrate", 1 * 1024 * 512, NULL);

GstElement *rtppay = gst_element_factory_make("rtpmp4vpay", NULL);
g_object_set(rtppay, "config-interval", 2, NULL);

GstElement *udpsink = gst_element_factory_make("udpsink", NULL);
g_object_set(udpsink, "host", dstIpAddress, NULL);
g_object_set(udpsink, "port", 5004, NULL);

gst_bin_add_many(GST_BIN(pipeline->pipeline), pipeline->src, fakesink, droidvenc, rtppay, udpsink, NULL);

GstCaps *caps;
caps = gst_caps_from_string("video/x-raw(memory:DroidVideoMetaData),framerate=30/1");

if (gst_element_link_pads_filtered(pipeline->src, "vidsrc", droidvenc, "sink", caps) != TRUE) {
    g_printerr("Failed to link droidvenc");
    return 1;
}
gst_caps_unref(caps);

caps = gst_caps_from_string("video/mpeg, mpegversion=4");
if (gst_element_link_pads_filtered(droidvenc, "src", rtppay, "sink", caps) != TRUE) {
    g_printerr("Failed to link rtppay");
    return 1;
}
gst_caps_unref(caps);

// ! connecting the viewfinder source is important, otherwise it will stop due to pushing a buffer into an unlinked pad !
// -> Apparently everything is designed for a mandatory use of a viewfinder.
if (gst_element_link_pads(pipeline->src, "vfsrc", fakesink, "sink") != TRUE) {
    g_printerr("Failed to link fakesink");
    return 1;
}

if (gst_element_link_many(rtppay, udpsink, NULL) != TRUE) {
    g_printerr("Failed to link elements");
    return 1;
}

See https://github.com/sailfishos/gst-droidcamsrc/blob/master/test/test.h#L28 for the definition of TestPipeline.

What am I doing wrong? The CPU usage is at ~19% while this pipeline is running but https://sailfishos.org/wiki/Multimedia says that the encoding should be hardware accelerated... Maybe the Xperia X with Sailfish does not support MPEG encoding in hardware at all?

edit retag flag offensive close delete

1 Answer

Sort by » oldest newest most voted
4

answered 2020-05-15 21:27:17 +0300

Andy Branson gravatar image

What a cool thing to do. Glad those docs have been useful! You might have more luck if you use the h264 or mpeg4 caps of droidcamsrc vidsrc rather than using droidvenc - that's what the Camera app does. In this mode, a Recorder mode is used that directly connects the camera to the codec in a much more efficient way.

edit flag offensive delete publish link more

Comments

Thanks for the hint!

Actually, droidvenc and the h264 pad of droidcamsrc perform roughly equally well. The culprit here was the MPEG part, which performs considerably slower for both ways (Xperia X). Anyway, droidvenc's CPU usage is ~ 2 percentage points higher and draws ~ 130 mA more current (measured with Battery Log), so it really is more efficient to use the h264 pad.

Unfortunately, the power consumption of both is too high for a standard 500 mA USB port on my computer - so running it drains the battery. Not much, though, but it can't run forever, which means I need plug it into a better charger and then transfer the image over WiFi (which therefore requires some encryption)...

edit: I should mention that the official Camera App also consumes that much power, so it's a general problem; I guess there's not much I can do against that.

takimata ( 2020-05-19 19:59:04 +0300 )edit
Login/Signup to Answer

Question tools

Follow
3 followers

Stats

Asked: 2020-05-10 19:59:06 +0300

Seen: 404 times

Last updated: May 15 '20