There are a few games that currently uses vector textures as an addition to normal bitmap textures, notably for Valve's Source engine for Team Fortress 2, but as far as I know no one has implemented anything on a large scale (Valve used vectors on some decals).
Normal textures that we use for gaming are compressed into a bitmap "raster" image of the surface comprised of pixels, each pixel representing a colour dot. Vectors are mathematical equations that represents the points, lines, curves, polygons of the surface and also includes colour information for each point rather than each pixel.
With bitmap images, once you scale something up, the pixels become relatively large and you lose detail and definition. Scaling down, you start to run into mipmap issues where details become averaged out until they are non-existant. With transparent surfaces, you have to do alpha-testing which can result in either jagged or blurred out edges. Bitmap images do a fairly good job on small to medium size surfaces that tile, but once you need a unique pattern over a large surface, you have to rely on larger images which requires larger file sizes, larger bandwidth, more video memory.
The advantages of vector images for game textures are that they are infinitely scalable. Since they are just mathematical equations, you can just multiply or divide its values to scale them up and down without loss in definition. You don't have to do any alpha-testing, you can specify exact transparency levels and create smooth gradients between and even modulate the values independently for each point (you could do that with colour as well, making some sort of trippy, seizure-inducing, changing rainbow texture). For file sizes, vector graphics will usually have smaller file sizes compared to bitmaps, though this heavily depends on the complexity of the image, but because they are infinitely scalable, the physical size of the texture won't impact file size.
On the down side, no hardware is optimized to handle vector images since we've been using bitmaps for years and that's the way things have developed. AFAIK, you have to processes the vectors though shaders which probably brings up all sorts of programming and performance issues which will need to be ironed out, but I supposed if Valve can do it...
Also, it's hard to define small pixelated details using vectors, its really much stronger for smooth gradient textures with sharp, well defined edges. That worked out well for TF2 since everything was cel-animation styled anyway.I'm not really working with a vector engine so at the time, textures are still being rasterized to bitmap for final use, but even from an artist's point of view there are a few advantages. Its hard working on a large detailed texture at 1024x1024 only to have texture budget constraints put on you and you have to resize things down to 256x256. Resizing bitmaps also causes you to lose most of the fine-lined details painted into the texture. Creating textures in vectors, you can choose your final output size when you need them exported to bitmap form without having to resample and losing details.
Vectors are also easier to work with when you are trying to create smooth curves. I tried painting the above texture at first in Photoshop, but even with a drawing tablet, it's hard to get the smooth curves while also maintaining that razor sharp edge. The pen tool in Photoshop stinks. So I imported a hand-drawn bitmap into Illustrator using the "place" command (which really tightens up integration between Photoshop and Illustrator since, I think CS1) traced it manually with the pen tool and then "placed" back in Photoshop for some final touches and exported to TGA. The place command is awesome because you can make changes to the source file and it will automatically be updated in the working file, so if I make changes to the Illustrator file, it'll automatically update back in Photoshop, so I alt-tab between programs to flip-flop between vector and bitmap versions.
As far as vectors being used in games, I think it has a strength as an alternative to alpha-testing. I'm sure it can be implemented so that you use vectors to define the edge of a texture rather than an alpha-channel to get a sharp edge without jaggies. It'll also be significant for HUD's (font rendering) and GUI's (like Doom3 style in-game menu systems) and for some special effects, particle systems perhaps. It'll probably see a lot more use in games like Paper-Mario style games, but a little more limited on your traditional FPS game where fine texture details are more of the norm.
Here's a SIGGRAPH 2007 Paper by Chris Green at Valve (PDF):
http://www.valvesoftware.com/publicatio ... cation.pdf
