Yes, the fresh air does enter the system on the pressure side of the blower fan AND the downstream air flow from the evap/heater core. It does manage to pull in some fresh air, but it is marginal at best.
If you don't close the fresh air vent when using the heater or A/C, you will be mixing unconditioned (outside) air with the conditioned air from your evap/heater core (which Jerry pointed out).
John