Dedicated to Q3Map2

[VolumetricSteve]

Anyone that's been bouncing around has probably encountered some of my many threads about accelerating/forcing q3map2 to do horrible things. After exploring many different routes with little to no success, I finally tried running q3map2 in linux as opposed to windows 7. The difference in compile time was staggering. It's so much faster in linux, I immediately retooled everything I could to help linux go as fast as I could get it to go. I started ripping apart openSuse 11.4 and the only thing it still has that makes it opensuse is the package manager/configuration tool. I'm running my first real linux command-line only q3map2 compile and it's already visibly faster than openSuse/KDE was, which was insanely fast in its own right. I'm hoping to develop this further (probably via suse studio) so that this "OS" could be booted from a flash drive, and could be configured with minimal effort to give everyone blazing fast compile times.

Right now the additional-applications on top of the text-only interface are:

telnet
full samba suite
gcc
make
development libraries
netradiant/q3map2
nano

I'm hoping I can shrink that further into:

q3map2
nano
telnet
samba suite

and redistribute it, but I don't know how flexible the various license agreements are.

If I can't do it that way, I can at least write up a quick-install guide that would make it incredibly easy to make any modern computer compile quake 3 maps much, much faster. But...to give some scope of how much faster it is...

Compiling my latest map in Windows 7 crashed from memory problems, but at the rate it was going, I projected it was going to take 2 to 3 days to fully compile.
Compiling the same map in openSuse 11.4/KDE took 18 hours and 20 minutes
I'm recompiling now in text-only mode and it's only been going for 4 hours and it's already 40% done.


I just wanted to announce this as a project

[Eraser]

Why Samba? Isnt that Linux's idea of a directory service like Active Directory or NDS? Or is that used for local user management as well?

[VolumetricSteve]

samba is the set of protocols that "windows networks" tend to use, so if I map anything on my laptop, I just share that directory and then I connect to it from my compiler so I'm not mounting and unmounting flash drives all the time. It'd be really easy to use ftp, nfs, or some other popular ones instead if you wanted.

And the latest compile just finished... the same compile that crashed in windows and took 18 hours with KDE just took 7 hours in text-only mode. I kinda wanna rescript that q3map2 benchmark to work in linux...

[VolumetricSteve]

I've been running compiles with this system and it's working really well. I'm working on the documentation simple enough that someone who has little to no experience with linux or q3map2 can set this same configuration up and enjoy ludicrously fast compile times no matter what options they have enabled. The problem is, I originally wanted to just list highly detailed steps, but the packages needed seem to change enough that listing steps that in-depth wouldn't be correct after like..a month...so I'm just having sections that kinda teach the user how to develop and work with linux enough to make it do what they want as opposed to "here's what I did verbatim."

I'm gounna run some tests later to see how well it works with decent, god fearing maps that are actually optimized and not meant to push quake 3 to the edge of madness. Right now, I'm able to crank out CoreFault tests with:

-light -bounce 8 -bouncegrid -samples 4 @ 6 hours and 30 minutes

I should also note that the key advantage to this acceleration is to really let q3map2 handle all of your map's lighting via surface lights. If you manually light your maps with point lights, compile with -fast, and don't do any bounces, your maps will compile so fast your skin might actually start falling off. The goal here is to let the computer do more, and do it faster (and I think it looks much better as long as you got your shaders programmed correctly)

[VolumetricSteve]

The Documentation for this is on-going

I've built a new test system designed from the ground up to be a more efficient linux-based single application machine.

To move forward with my project, I was wonder if anyone knew what Q3map2 really likes the most in a system. I know it has a smaller memory footprint by today's standards. My biggest map caused Q3map2 to call 1.27GB of ram under Win7-64. But...does it do a lot of ram I/O? Would it benefit drastically from really high-speed memory (DDR3 1600, or higher) or would it work just as well if the ram were DDR3 1333? I know having a beefy CPU is what it needs the most, but I wondered how much the ram speed is a factor in the kinds of calculations Q3map2 spends a lot of time working on.

Any ideas?

[^misantropia^]

q3map2 is mostly constrained by CPU and memory bandwidth. I think that if you run it through gprof and cachegrind / callgrind[1], you'll find that at least 50% of CPU time is spent in only a handful of functions. Optimize the bejesus out of them and you'll see dramatic performance improvements.

As a show case of the performance gains you can get out of profiling: https://github.com/tanoku/upskirt/pull/7

[1] http://valgrind.org/info/tools.html#cachegrind - and also check out kcachegrind, the visualizer.

[VolumetricSteve]

I've got a nerdy weekend ahead of me, I've got investigating to do. I'm lookin' forward to this.

Also, I'm not sure if this is the right place for it, but my system's memory can run in 1x128 (ganged) mode, or 2x64 (unganged) mode. Would q3map2 see a benefit to running in 1x128 mode? I'd experiment and try but I'm still at work.

[^misantropia^]

Unganged is faster in theory (less memory bus contention). In practice? I'm on the fence - but sceptical.

[VolumetricSteve]

I'm running into some issues with segmentation faults now...and I actually got "Illegal Instruction" the other day but I think that one was entirely my fault. For starters, if I can run Memtest for 8 hours at a time or more, my segmentation faults shouldn't be hardware issues, right? I'm experimenting with overclocking and I'm getting strange results and I'm having a hard time telling if the issue is my map or my overclock settings. I was getting segfaults before I started overclocking, and was running the system at stock speeds.

What I've been seeing is that BSP will run fine, over and over, then I'll run VIS, and it'll segfault at the exact same place each time, but I'll run light with my crazy settings, and it'll get really far, but it will also segfault at the same place repeatedly. So..it appears that different functions are responding differently to various OC settings. I'd expect segfaults to happen because of an OC at seemingly random intervals, not consistently like I'm seeing here. The other weird thing is Linux runs perfectly fine otherwise, q3map2 seems to be the only thing that's acting weird. Obviously, I'll try going back to stock speeds and starting over, but isn't it strange that a system-wide change would only mess with the operation of some parts of one program?

I should also note that I tried turning off -samples to avoid further segfaulting, but the fact that it's known to throw segfaults at all leads me to think the -samples function wasn't implemented correctly.

[^misantropia^]

If you're already getting segfaults, overclocking can only make it worse. Send over the map (+assets, shaders, etc.) and I'll have a look, repeatable test cases are a good thing. Couple of questions:

* What's the q3map2 version and where did you get it from?
* If you compiled it from source, what compiler (`gcc -v`) and compilers flags were used?
* What architecture are you running it on?
* What switches are you using?

[VolumetricSteve]

If you're already getting segfaults, overclocking can only make it worse. Send over the map (+assets, shaders, etc.) and I'll have a look, repeatable test cases are a good thing. Couple of questions:

Agreed. I didn't think they'd get as bad as they did, in such weird ways too.

* What's the q3map2 version and where did you get it from?

https://github.com/bnoordhuis/netradiant.git

* If you compiled it from source, what compiler (`gcc -v`) and compilers flags were used?

GCC and none that I know of, I just ran make.

* What architecture are you running it on?

X86 (AMD T1100 Phenom 2)

* What switches are you using?

BSP ran fine with -threads 12 -meta -patchmeta
I tried vis with 12 threads, and 6 threads, saveprt and without saveprt
light used to run fine with -light -bounce 8 -samples 4 -filter
then segfaults started to crop up, and I tried removing -samples and even switched to a new machine (the T1100 I'm using now)
I'm getting segfaults now with -threads 6 -light -bounce 2 -filter

[VolumetricSteve]

I reset the system to stock clocks. I'm re-running a memory test, and then I'll find something to stress the cpu with that isn't q3map2. I hear Mprime is the linux version of the popular Prime95 software, so I'll give that a shot.

Edit:

Memory test is looking good for the quick run, I couldn't get Prime95 but I'm going ahead with cpufreq-bench which should push it pretty far I'd think.

Re-edit:

ok this is weird, I was originally getting segfaults at stock clocks, kinda shrugged at it, I actually figured it was some horrible thing my map was doing to q3map2, I didn't worry about it too much, and I wanted to investigate overclocking possibilities, so I did, naturally, segfaults increase, I restore to stock clocks last night, mem test is fine, cpubench is fine, q3map2 ran -threads 12 -light -filter -bounce 12 with no issues...

did I blow something out of the system? what ...does that?

the other really weird thing is, I have two of these systems, the first one compiled my map in about 27,500 seconds, the new system, running at the same speed, compiled it in 19,800 seconds.