So after upgrading to the latest version of the Mozilla code last week I started to get an assertion when I was using GStreamer to decode video on the N810. Debugging is usual much nice to do on the PC, so I set up building Fennec for Linux, I had to repeat most of the steps that I had done in scratchbox to make it compile with gstreamer, but since it's all Linux it wasn't much of an issue to just repeat the steps.
Sure enough the same error appeared on the laptop as I had seen in the device (lucky me). I didn't have much luck in attaching the debugger to Fennec after it had asserted, but when I redid the test starting Fennec from the debugger it was easy to see the call stack.
It seemed to be related to a delete of the buffer given to nsMediaDecoder::SetRGBData - diffing the old version of this function (where the code was working) with the new version, showed something interesting.
Originally the buffer given in SetRGBData had been copied to an internal buffer, meaning the caller of SetRGBData had ownership of the buffer. In the new version there is no longer a copy, the pointer to the buffer is stored in a nsAutoArrayPtr, meaning the next time this pointer is assigned something the previous content is deleted - ahh... we start to see a reason for the fault here, don't we?
Previously it was the responsibility of the caller to delete the buffer, this is now handed over to the called function, but since the GStreamer code hasn't been updated to reflect this there is a conflict of ownership.
Realising what the problem is, is the first step, now I need to fix it ;)