I've wondered this for a while now but never gave it any thought. I randomly decided to search for an answer and surprisingly there was a nice anandtech article on the subject:
http://www.anandtech.com/weblog/showpost.aspx?i=571
Why the hell are nvidia drivers so big?
Re: Why the hell are nvidia drivers so big?
Article's taking an aeon to load so I'm gonna go ahead and guess: because they're unified, and because they include app-specific tweaks.
Re: Why the hell are nvidia drivers so big?
PhysX is a big chunk, CUDA also plays a part, but yeah, those too.