Many of these plugins come with proprietary closed source code that could potentially expose users to security vulnerabilities. PDF viewing plugins also come with extra code to do many things that Firefox already does well with no proprietary code, such as drawing images and text.
That's actually not the mudslinging that you might think. Adobe recently issued a warning about several current versions of Adobe Reader and Adobe Acrobat. The apps currently have a vulnerability that some hackers identified and have been exploiting that will let malware take over your computer.
And that's not even the first security bug to be found in closed apps like Adobe Reader, which is the reason why Mozilla released their own PDF Viewer. It's based on the open source PDF.js project. Thanks to the possibility that anyone can contribute bug fixes or point out issues, in the long run it's bound to become more secure than competing plugins like Adobe Reader.