I wrestled with the same question, but after the 2nd or 3rd time I almost changed lanes into someone because they were in my blind-spot, the decision was made! And I've never regretted it!
FWIW, Panteras sold through Lincoln-Mercury dealers in the US from 71-74 did not come from the factory with side-view mirrors. The driver's door side-view mirror was added by the dealerships. Adding a matching mirror to the passenger door is just finishes the job, improves the appearance, and makes your car a LOT safer to drive.
As was already said, better to have 2 tiny holes to mount your mirror than a wrecked "original" Pantera!