Possibly encountering a memory leak on my patches?


#1

Hi all,
I’ve been developing some multi-track loopers from scratch on my Organelle and while all my patches work independently if they are the first thing I load after startup, if I load several of them in a row (or reload the same one multiple times) performance degrades - I start getting some pops and clicks in the audio - and sometimes the patch won’t even finish loading and will hang during the load process.

As I mentioned, no single patch has an issue if it is the first one loaded after startup, so there must be some retention of state patch-to-patch, which doesn’t seem like it should happen. My only guess is that PD is not releasing all the memory I allocate to arrays for my loopers in between patches, but I’m not sure how to improve (or even begin to investigate) this issue. Any thoughts?

Thanks!


#2

Without running your patch it is hard to say what is happening. When you choose a new patch with the Organelle encoder, Pd shuts down and then restarts with your new selection. This is true even if you select the same patch that was running before.

If you have a monitor, mouse, and keyboard connected to the Organelle you can use the utility called top. It will show CPU and memory usage in ‘real time.’ Type top into the command prompt and hit return (sorry if obvious). It looks something like this (thanks to @thetechnobear for this image):

Tip: To sort by memory usage type: top -o %MEM
More tricks here, though not all will work on the Organelle…

***You can also run this command from the terminal if you are already in GUI mode (from typing ‘startx’ at prompt), but Pd will reload on top of the terminal window and you won’t be able to see the change right away.


#3

It doesn’t appear that a memory leak is the issue - I kept top running during a session and saw my memory/CPU usage stay basically unchanged during normal, poppy, and crash behavior. Memory is in the 50-60% range and CPU is around 40% for my patches in question.

The bad behavior definitely depends on sequencing though, as mentioned - loading patches straight from boot never crashes, and they tend to get worse over time (edit for clarity - I mean worse in that the more patch loads I perform after a boot, the more the odds of a hang go up). Often they hang during load - I can tell because the previous patch unloads (I’m running in GUI mode so I can see the windows close) and my patches have some initialization messages on the OLED screen so I can track where they are during the process. I can see that when the hang occurs, it’s often partway through the load process. Sometimes, if I wait 30-60 seconds, the load completes, but this is rare. If it does complete, the subsequent behavior is often full of clicks and pops in the audio so a reboot is still necessary.

It’s worth mentioning I’m not using the Organelle’s internal audio device - I’m running a Scarlett 18i20, which hasn’t been giving me any problems as far as I know. This popping/crashing behavior is relatively recent and wasn’t associated with when I added the Scarlett to my setup.


#4

How are you powering the organelle?


#5

Even though you say it’s not related to your audio interface, have you observed the same behavior with the built in audio?

FYI: I’d say if your using an external interface over usb, you are probably going to need to increase buffer sizes a bit.

It’d also help to see/ be able to run the patch.

As has already been stated the previous process has been killed, so there’s not really anything to ‘hang over’ to a restart.

So think we’d need to see it to debug it.
(That again would need to be using the built in audio)

It’s odd that first time always works, otherwise I’d have thought it was perhaps related to indeterminate initialisation order in the patch having some effect.

Is he particularly interested what ‘it hangs during startup’ means in practice - I’d suspect that might the way to track down the underlying issue.

Anyway if you post patch ( and any required resources) perhaps we can take a look.


#7

Wannop, I’m using the included adapter


#8

I’ve posted two of the more frequent offender patches in https://www.dropbox.com/sh/0ona5xgmb6nekxg/AABu35EFTNQsgKtQf4CG57sua?dl=0

The Diagonal Lines patch is the most likely to hang, and the BF2 one sometimes hangs. The Include and Wavs folders are used by both those patches.

Technobear, I haven’t observed this exact behavior with the built-in interface, but I did have trouble with the patches loading at first back when I was using the built-in. I posted about that issue in this thread: My patch loads & runs fine in graphical mode - but not otherwise. Any ideas?

I fixed the issue by delaying and staggering the initialization of my looper arrays. I’m not sure if this is the same thing but it feels similar. As mentioned, the presence of the new interface alone does not cause the failure. Even if I can’t rule the interface out as a factor completely, I can say that there is more to it than just the presence of the interface, especially as some of my patches show this issue often and others never do. They’re all structurally very similar, just with different looper arrays and .wavs and sequencing in the part-behavior and gear-behavior subpatches.


#9

Also, sometimes MIDI messages that should get sent by the loadbang do not make it out on these patches. I have it set to configure my pedalboard and synthesizer via MIDI on load and the synth only loads the correct preset some of the time. Specifically, these are the jdxi MIDI messages in gear-behavior that should get sent based on the init_gear bang.


#10

ok, had a quick look…

for me, I could see when it failed, it was being killed… but unfortunately i couldn’t see what was killing it.
(unfortunately, we cant install gdb on the current OS,so I cant see the stack trace to get more details)
it could either be a segv fault, or it could be pd watcher is killing it due to not responding.

but honestly, I suspect the underlying issue is exceeding processor/hardware limitations, and its just not failing elegantly (with decent error messages)


your allocating 10 x 1 mins of audio buffer, but Im pretty sure @oweno has said in practice they got 5-6 minutes(*)?
so I think its going to run out of memory, which i think along with the indeterminate initialisation you have, is probably why its fairly randomly crashing.
PDs error checking is not great, so Id suspect one of the mallocs is failing ‘quietly’ and then something is trying to right in the memory region regardless.

not sure , why it sometimes it appears to work… id have said perhaps swapping, but afaik there is no swap configured (inappropriate for audio use), or perhaps this is related to a pd mem issue (see below)

(*) ok, I will say here, I don’t know why the available ram is so low, 10 minutes, even if using 32bit floats (as pd does i believe) is only 1.68mb (EDIT: see below, misread this) , the organelle is reporting 216mb free … so its a drop in the ocean really.
but to take this further, Id have to do some experimentation to see if PD is ‘exploding’ the size (so a PD issue) , of a more general linux/pd issue where the process is restricted in ram usage. unfortunately thats a bit time consuming to do properly - which i dont have at the moment.

EDIT: im a bit confused by your tables, although they have a size parameter (100), the size reported in the properties is still 2470129… so give we have 40 of them, and they are 4 bytes each … thats 376Mb which is too much.
so id suggest perhaps checking that size is really used from startup, i.e. PD doesn’t do something silly like try to allocate the sizes, and then later free them to reduce in size…
perhaps set them all to 0 in properties, and then use a variable passed in as arguments, so its easy for you to change without loads of typing .


I think your usb audio setup is probably overly optimistic,
i doubt your going to get 8 channels input and 10 channels of output with a buffer of 6 msec, (~270 samples) - I doubt the organelle has the throughput on usb nor the processing power.
(lets bare in mind the organelle was designed for 2 ch I/O at 4ms (176 samples) on an internal bus)

I assume you are also sending midi over this usb, so thats also extra data on top of your usb for audio.


sorry, as i say without gdb, not a lot I can see about exactly where its crashing, so all I can say is I think its too cpu and memory heavy, and PD is just coping with this poorly. (crashing)
Ive seen this with Orac, once you go over a certain cpu load, PD/arch linux start having really issues - this is partly because we run PD with realtime priority i.e. its CPU usage starts affecting system services.

as its hardware specific, Im sure it works on other hardware, so ive no opportunity to test on something else.
I dont have time to install it, on my upgraded linux image for organelle. (which is on the back burner at the moment) which would give me gdb.


what I would suggest is start hacking the patch back, less loopers (so less memory) and less IO channels, and higher buffer sizes - perhaps cut it back to the minimum, and then build it back up slowly.
this will hopefully let you find the ‘practical’ rather than theoretical limit of the Organelle.

of course, all just my opinion, about how Id tacked it.
if you really believe these are not the issues, your other cause of action is still similar, cut back the patch to find the ‘offending’ code that is blowing up.


#11

Thanks for taking the time to look at it! Everything you say tracks except for the fact that all of these patches work just fine from startup, and even the more problematic ones often present no issues. If I’m really running out of memory, CPU power, or USB bandwidth, I don’t see how I could run through songs with these with no glitches. When the Organelle closes and opens patches, is there a shell script or something executing? I’m wondering if I gave it more time to shut down and then re-initialize with the new patch if that would help.

It really seems like the Organelle has the resources to run what I want it to, it just has issues deallocating/reallocating those resources within some constraints that I can’t see. Of course, that’s just a semi-educated hunch from running these patches a lot, I could be totally wrong.


#12

Yeah I understand what you mean, but the organelle is using arch Linux, and Pd is just a Linux process, once’s its killed it’s resources are freed - that’s real basic Linux stuff (= userspace processes)

I also ran Pd manually from the command line, and saw the same/similar issues- basically it seemed really unstable.

All that said perhaps something deeper is going on , but we don’t really have any evidence of this - or really an easy way to get more info.

Also bare in mind whilst this patch is behaving erratically lots of others are fine


#13

Further experimentation shows that I am much more likely to crash if I am re-loading the same patch that is already open, whereas loading all of the patches I’ve made sequentially almost never causes a crash. I also almost never crash unless I’m running in GUI mode, which makes some sense.

I have a question about the resources expended by using additional audio IOs - I have 8 ins and 10 outs on my external interface, but in my program I’m only using 6 ins and 6 outs. If I change my pd-opts.txt so that instead of 8 and 10 I’m using 6 and 6, will that give me some overhead back? Or am I already not using those channels since I haven’t created adc~/dac~ objects for them?

Thanks!