I know this because I see the packets bounce off my internal firewall. When this happens, I recheck the proxy settings, and even confirm that they work by firing up Software Update and Safari and see if they can connect. They always do. Even going to the command line and using "scutil --proxy" works as expected.
One of the main culprits is VersionTracker Pro. Launching it starts off a disk scan to find applications that might need updating. It does this by submitting the versions on your machine to be compared with a master list located on the VT web site. This always works. You then get a list of applications that can be updated, and if you click on any of them a preview pane gives you some further info on the application and the update available.
Well, it's supposed to. Most of the time this doesn't work for me, as it tries to make the request for this info directly to the Internet instead of going through the proxy. Sometimes however it does, and up pops the details as expected. This could have been put down to flakiness within the VT application itself (and it certainly might be contributing in some way to the problem), but I've also noticed other flakey behaviour with a couple of other applications on my network; online TV guide updates within various apps spring to mind. Oh, and trying to register tools like Mac Pilot.
Anyway, I've lived with this annoyance for quite a while but recently I decided to do something about it once and for all. I tried poking around the lower level workings of Mac OS X to see if something was up there, but as mentioned above, command line utilities like scutil gave me the results I was expecting. I even tried posting to the online Apple forums, but so far, no joy there.
I did start to notice that perhaps there was some differentiation with the requests due to whether HTTP or HTTP/S was used for the request. In the VT example the first request (that works) is HTTP/S and the second request (that fails) is HTTP.
But there seemed to be no difference in my HTTP settings compared to my HTTP/S settings, so eventually I got sick of trying to nut this OS X flakiness out. And I certainly did not want to just blindly pick holes in my firewall just to let some rogue application and/or OS X flakiness have its way. Then it struck me that I should just transparently proxy these requests and be done with it. I already had a FreeBSD-based firewall running Squid for all web access.
Which worked a treat. I still get to enforce only specific rogue application/sites that are allowed to be transparently proxied, and everything is now working as expected. The rogue applications think they are directly communicating with their sites in question, but I still get to log them through the proxy and leave my firewall intact. It's fairly low maintenance to add a new rule if I ever need to. I just need to add forwarding rules to my firewall such as:
# ipfw add <ruleno> fwd <proxy-ip>,<proxy-port> tcp from <client-ip> to <rogue-site-ip> dst-port 80 setup keep-state
Configuring squid to work as a transparent proxy is a doddle nowadays as well. I found a good description at Leslie Viljoen's wiki called Eclectica, which also happens to run on a Mac Mini (a good sign I thought.) His instructions are for Squid 3 (in beta), but as pointed out by a visitor, they also happily work with the later 2.6.x versions as well (I'm running 2.6.16 myself). In my existing config pretty much all I needed to do was add the "transparent" option to the http_port directive and that was it!
So simple at the end of the day, even though initially I didn't think of it because it offended my sensibilities somewhat that things weren't working as they should. I wanted the proxies to work as advertised! But I guess, like a lot of things in life, sometimes you just have to forget about looking at the trees if you want to enjoy the forest...
