MuPDF is indeed a splendid piece of software - it’s also produced by Artifex!
MuPDF is a different beast than SmartOffice in that it only has to handle a smaller number of formats (PDF, XPS and some image types). It doesn’t have to do any of the layout or editing tasks that SmartOffice does. As such the requirements/restrictions on the internal handling of the formats are very different.
If you can share a problematic file with us, I’ll have a look at it to see if I can spot anything that’s taking much more memory than it should, but I can’t promise any magic bullets in the short term.