[MPlayer-users] MPlayer memory leak with lavc output

[Michael Kohne]

I'm seeing a memory leak with mencoder when capturing data from a USB
video camera and outputting to a file using -ovc lavc:

./mencoder tv:// -tv
driver=v4l2:noaudio:device=/dev/video0:width=640:height=480:fps=60 -o
/tmp/outtest.avi -ovc lavc

I've tested with MPlayer-1.3.0 and with last night's build
(2017-05-24). In both versions I get a memory leak of over
100K/minute.

mplayer playing from the same camera:
./mplayer tv:// -tv
driver=v4l2:noaudio:device=/dev/video0:width=640:height=480:fps=60
Does not have a memory leak.


The video camera is an e-con See3CAM, but we see the same problem with
an older Philips camera as well.

I can provide mencoder output for a short run if that's helpful (I
couldn't attach logs due to message size limits).


Any clues as to ways around this, or how to go about debugging further
would be much appreciated. We're trying to update from a MUCH older
version of MPlayer to something newer (to support new cameras), and
this leak kills us after a day or two of running.





CentOS 6.6
Linux KohneC22 2.6.32-504.8.1.el6.i686 #1 SMP Wed Jan 28 18:25:26 UTC
2015 i686 i686 i386 GNU/Linux


I'm running on a system slightly different than the one I build on.
The target system is the one where I'm RUNNING, the build system is
where I built the latest version of MPlayer.

This is libc from the target system.
[***@KohneC22 tmp]# ls -l /lib/libc*
-rwxr-xr-x. 1 root root 1906308 Jan 27 2015 /lib/libc-2.12.so
lrwxrwxrwx. 1 root root 18 Apr 11 13:07 /lib/libcap-ng.so.0 ->
libcap-ng.so.0.0.0
-rwxr-xr-x. 1 root root 17880 Jun 24 2011 /lib/libcap-ng.so.0.0.0
lrwxrwxrwx. 1 root root 14 Apr 11 13:07 /lib/libcap.so.2 -> libcap.so.2.16
-rwxr-xr-x. 1 root root 14328 Dec 7 2011 /lib/libcap.so.2.16
-rwxr-xr-x. 1 root root 190992 Jan 27 2015 /lib/libcidn-2.12.so
lrwxrwxrwx. 1 root root 15 Apr 11 13:29 /lib/libcidn.so.1 ->
libcidn-2.12.so
lrwxrwxrwx. 1 root root 17 Apr 11 13:07 /lib/libcom_err.so.2 ->
libcom_err.so.2.1
-rwxr-xr-x. 1 root root 15496 Oct 15 2014 /lib/libcom_err.so.2.1
-rwxr-xr-x. 1 root root 40296 Jan 27 2015 /lib/libcrypt-2.12.so
lrwxrwxrwx. 1 root root 22 Apr 11 13:09 /lib/libcryptsetup.so.1
-> libcryptsetup.so.1.1.0
-rwxr-xr-x. 1 root root 102508 Oct 15 2014 /lib/libcryptsetup.so.1.1.0
lrwxrwxrwx. 1 root root 16 Apr 11 13:29 /lib/libcrypt.so.1 ->
libcrypt-2.12.so
lrwxrwxrwx. 1 root root 12 Apr 11 13:29 /lib/libc.so.6 -> libc-2.12.so


gcc and ld from the build system:
[***@KohneL06Dev ~/sandbox/MPlayer_snap/mplayer-checkout-2017-05-24]# ld -v
GNU ld version 2.24
[***@KohneL06Dev ~/sandbox/MPlayer_snap/mplayer-checkout-2017-05-24]# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-3/root/usr/libexec/gcc/i686-redhat-linux/4.9.1/lto-wrapper
Target: i686-redhat-linux
Configured with: ../configure --prefix=/opt/rh/devtoolset-3/root/usr
--mandir=/opt/rh/devtoolset-3/root/usr/share/man
--infodir=/opt/rh/devtoolset-3/root/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap
--enable-shared --enable-threads=posix --enable-checking=release
--enable-multilib --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-gnu-unique-object
--enable-linker-build-id --enable-languages=c,c++,fortran,lto
--enable-plugin --with-linker-hash-style=gnu --enable-initfini-array
--disable-libgcj
--with-isl=/builddir/build/BUILD/gcc-4.9.1-20140922/obj-i686-redhat-linux/isl-install
--with-cloog=/builddir/build/BUILD/gcc-4.9.1-20140922/obj-i686-redhat-linux/cloog-install
--with-mpc=/builddir/build/BUILD/gcc-4.9.1-20140922/obj-i686-redhat-linux/mpc-install
--with-tune=generic --with-arch=i686 --build=i686-redhat-linux
Thread model: posix
gcc version 4.9.1 20140922 (Red Hat 4.9.1-10) (GCC)


as from the build system:
[***@KohneL06Dev ~/sandbox/MPlayer_snap/mplayer-checkout-2017-05-24]#
as --version
GNU assembler version 2.24
Copyright 2013 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `i686-redhat-linux'.


[***@KohneC22 tmp]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i3-4130T CPU @ 2.90GHz
stepping : 3
microcode : 29
cpu MHz : 800.000
cache size : 3072 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts xtopology
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2
ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt
tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb
xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase
bmi1 avx2 smep bmi2 erms invpcid
bogomips : 5786.17
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i3-4130T CPU @ 2.90GHz
stepping : 3
microcode : 29
cpu MHz : 800.000
cache size : 3072 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 2
apicid : 2
initial apicid : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts xtopology
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2
ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt
tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb
xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase
bmi1 avx2 smep bmi2 erms invpcid
bogomips : 5786.17
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i3-4130T CPU @ 2.90GHz
stepping : 3
microcode : 29
cpu MHz : 800.000
cache size : 3072 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 1
initial apicid : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts xtopology
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2
ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt
tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb
xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase
bmi1 avx2 smep bmi2 erms invpcid
bogomips : 5786.17
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Core(TM) i3-4130T CPU @ 2.90GHz
stepping : 3
microcode : 29
cpu MHz : 800.000
cache size : 3072 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 2
apicid : 3
initial apicid : 3
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts xtopology
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2
ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt
tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb
xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase
bmi1 avx2 smep bmi2 erms invpcid
bogomips : 5786.17
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:
--
Michael Kohne
Senior Software Engineer
Office: 215.283.0860 x208
***@moberg.com



Transforming Neurocritical Care

Moberg Research, Inc.
224 S Maple Street, Ambler, PA 19002
24/7 Customer Support: 1-888-662-7246
www.moberg.com

[Reimar Döffinger]

Post by Michael Kohne
I'm seeing a memory leak with mencoder when capturing data from a USB
./mencoder tv:// -tv
driver=v4l2:noaudio:device=/dev/video0:width=640:height=480:fps=60 -o
/tmp/outtest.avi -ovc lavc
I've tested with MPlayer-1.3.0 and with last night's build
(2017-05-24). In both versions I get a memory leak of over
100K/minute.
./mplayer tv:// -tv
driver=v4l2:noaudio:device=/dev/video0:width=640:height=480:fps=60
Does not have a memory leak.
The video camera is an e-con See3CAM, but we see the same problem with
an older Philips camera as well.
I can provide mencoder output for a short run if that's helpful (I
couldn't attach logs due to message size limits).

valgrind is the most useful tool to debug the leaks it should (after e.g. aborting the command) tell quite clearly where all the memory was allocated.
It is however fairly likely it is not a leak.
Typical things that look like a leak:
- index data for AVI which must be kept in memory and written at the end. It can get really large due to issues with "variable" frame rate or some audio codecs in AVI
- audio and video are getting out of sync, requiring one to be buffered a lot. Playing the file while recording might show it by one or the other ending earlier. -noskip -mc 0 usually avoids the leak, but not other issues.

[Michael Kohne]

Post by Reimar Döffinger
valgrind is the most useful tool to debug the leaks it should (after e.g.
aborting the command) tell quite clearly where all the memory was allocated.
It is however fairly likely it is not a leak.
- index data for AVI which must be kept in memory and written at the end.
It can get really large due to issues with "variable" frame rate or some
audio codecs in AVI
- audio and video are getting out of sync, requiring one to be buffered a
lot. Playing the file while recording might show it by one or the other
ending earlier. -noskip -mc 0 usually avoids the leak, but not other issues.

Well, I've solved the problem:
- Valgrind turned out to be useless for this. It choked on some
instructions in mencoder (I suspect some of the hand-written stuff). This
is a far from unknown problem with valgrind - it even tells you to how to
send a bug report with the offending instructions. I added my bug report to
their tracker just like many others, but since I'm working in 32 bit mode,
it's really not worth their fixing.

- I ended up using 'HeapUsage' (https://github.com/d99kris/heapusage),
which was much more amenable to working with mencoder.

- HeapUsage gave me a nice stack trace for the leak - an AVPacket in
encode_frame (libmpcodecs/ve_lavc.c) wasn't being cleaned up correctly (the
side_data was leaking).

- I checked the latest version of mencoder and this issue is clearly fixed
(the av_packet_unref at the end of the function), indicating that my
initial tests with the 2017-05-24 code were wrong in some way.

- It turns out that getting HeapUsage to deal with mencoder requires a
particular set of of compile/link option changes:
./configure --disable-gnutls --enable-debug --disable-relocatable
Modify the output config.mak file:
Remove -O options from CFLAGS & HOSTCFLAGS
Remove -Wl,--version-script,binary.ver from EXTRALIBS
Add -fno-inline to CFLAGS and CXXFLAGS

- If you don't get all the -O flags, and the version-script, then HeapUsage
still shows the leaked blocks, but the addresses it gives for where the
blocks were allocated are wrong and useless.


Thanks!
--
Michael Kohne
Senior Software Engineer
Office: 215.283.0860 x208
***@moberg.com


Transforming Neurocritical Care

Moberg Research, Inc.
224 S Maple Street, Ambler, PA 19002
24/7 Customer Support: 1-888-662-7246
www.moberg.com

[Reimar Döffinger]

Post by Michael Kohne

Post by Reimar Döffinger
valgrind is the most useful tool to debug the leaks it should (after e.g.
aborting the command) tell quite clearly where all the memory was allocated.
It is however fairly likely it is not a leak.
- index data for AVI which must be kept in memory and written at the end.
It can get really large due to issues with "variable" frame rate or some
audio codecs in AVI
- audio and video are getting out of sync, requiring one to be buffered a
lot. Playing the file while recording might show it by one or the other
ending earlier. -noskip -mc 0 usually avoids the leak, but not other issues.

- Valgrind turned out to be useless for this. It choked on some
instructions in mencoder (I suspect some of the hand-written stuff). This
is a far from unknown problem with valgrind - it even tells you to how to
send a bug report with the offending instructions. I added my bug report to
their tracker just like many others, but since I'm working in 32 bit mode,
it's really not worth their fixing.

Yeah, I guess you have your reasons, but 32 bit x86 is really unsuitable for multimedia (easily 10% slower on average) so it's not tested much.
But you should be able to use something like --target=generic configure option to disable all architecture-specific optimizations.
Just in case you might need to use valgrind after all at some point.
And yes, there are more lightweight alternatives that avoid such issues and are faster. google perftools is another option that might be more polished (your description sounds like your tool has problems reading advanced debug info). But they are a bit harder to use, so I usually recommend valgrind as first option.

[Michael Kohne]

Post by Reimar Döffinger
Yeah, I guess you have your reasons, but 32 bit x86 is really unsuitable
for multimedia (easily 10% slower on average) so it's not tested much.

Not at all surprising or unreasonable. In this case we're using a modified
version of mencoder to capture data from a video camera and feed it into
our medical monitor. The whole thing goes back over 10 years, and upgrading
things in the medical field is...problematic. On the next hardware rev I'll
be able to force through the switch to a 64 bit OS, but until them I'm
stuck at 32 bit. I'm just glad I was able to upgrade to a later
mplayer/mencoder - newer cameras forced our hand on that one, or I'd still
be using 1.0pre8.

But you should be able to use something like --target=generic configure

Post by Reimar Döffinger
option to disable all architecture-specific optimizations.

Just in case you might need to use valgrind after all at some point.
Oh, I hadn't considered that. I'm far from familiar with valgrind, so that
never occurred to me. Thank you.

Post by Reimar Döffinger
And yes, there are more lightweight alternatives that avoid such issues
and are faster. google perftools is another option that might be more
polished (your description sounds like your tool has problems reading
advanced debug info). But they are a bit harder to use, so I usually
recommend valgrind as first option.

I think for most folks valgrind is very much the right option. It's just
that I am (in so very many things) the outlier. I'll have to check out
perftools in the future. For the moment, now that I know what it needs,
HeapUsage did the job.

Thanks!
--
Michael Kohne
Senior Software Engineer
Office: 215.283.0860 x208 <(215)%20283-0860>
***@moberg.com


Transforming Neurocritical Care

Moberg Research, Inc.
224 S Maple Street, Ambler, PA 19002
24/7 Customer Support: 1-888-662-7246 <(888)%20662-7246>
www.moberg.com