Quake3World.com Forums
     Level Editing & Modeling
        Quake 3 Map Compile Benchmark
Post new topicReply to topic
Login | Profile | | FAQ | Search | IRC




Print view Previous topic | Next topic 
Topic Starter Topic: Quake 3 Map Compile Benchmark
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-10-2005 05:08 PM           Profile Send private message  E-mail  Edit post Reply with quote


This is my attempt at a standardized benchmark to see which systems compile quake maps the best.

It's been a year or so since I last posted this, but a thread in GD made me think that perhaps some of you have upgraded your systems since then. I'm still curious to see how an Althon64 compares to a P4 system. It looks like a standard P4 is a better system than the Althon XP, but these processors are starting to get old (nevermind how old quake 3 is itself!) with the Althon64 out and dual processors around the corner.

Webpage: http://www.ciole.net/quake_bench/

From the webpage you can download the benchmark tool and post your results. :)




Last edited by mjrpes on 09-13-2011 09:33 PM, edited 1 time in total.
Top
                 
obsidian
I'm the dude!
I'm the dude!
Joined: 04 Feb 2002
Posts: 12498
PostPosted: 05-10-2005 06:05 PM           Profile Send private message  E-mail  Edit post Reply with quote


Maybe it's because my 3.0 GHz processor and 1GB of RAM is too fast for it, but I can't seem to get the benchmark to run correctly (or at least output the results).

I have it installed in C:\Games\Quake III Arena\benchmark\

It gets as far as displaying "Now Benchmarking Map Compile . . . Please Wait . . ." then the Command Prompt window closes. It seems as if the BSP and PRT file are generated, but I doubt that it's getting as far as the light compile, since it only runs for a few seconds.

In the results.txt file is this:
QUAKE3 MAP BENCHMARK 1.1 - RESULTS
==================================

Any ideas?



_________________
GtkRadiant | Q3Map2 | Shader Manual
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-10-2005 06:22 PM           Profile Send private message  E-mail  Edit post Reply with quote


Interesting.

The process goes from map compile, to vis, to bspc, then to lighting.

At the stage you mention, it is definitely getting caught up in the map compile, since no output for the map compile is made into the result.txt file. The actual command it runs at that point is:

"q3map2 -fs_basepath .. -meta -v q3dm1sample.map"

I wonder if something could be wrong here. I ran the batch file where I commented out the actual command, so as to emulate a super fast system that could compile the map instantly, and no problems were encountered.

I don't know what else to think here.
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-10-2005 11:10 PM           Profile Send private message  E-mail  Edit post Reply with quote


obsidian, try downloading this new version:

http://ciole.net/quake_bench/quake3_map ... rk_1.2.zip

This new version has all necessary files and doesn't rely on quake3 anymore; you can run it from anywhere. Perhaps this will solve the problem you were having.
Top
                 
[acid]
Commander
Commander
Joined: 18 Jul 2004
Posts: 128
PostPosted: 05-11-2005 11:41 AM           Profile Send private message  E-mail  Edit post Reply with quote


i need an upgrade :D
Top
                 
TTK-Bandit
Insane Quaker
Insane Quaker
Joined: 06 Sep 2002
Posts: 399
PostPosted: 05-11-2005 01:10 PM           Profile Send private message  E-mail  Edit post Reply with quote


the problem seems to be that obsidian has no english version of windows.
the date and time variables are diffent on different windows language versions.
so.. I just wrote a little test app, that can do it:
http://mitglied.lycos.de/Q3Coderz/files/

Just copy the 2 files into the "files" directory and run bench.exe
bench.cfg format is pretty simple:
1. line is benchmark version
all lines that follow are benchmark tests in this format:
name|commandline

But I've seen that you build your homepage data out of the output,
so you'd better tell me what output you wanna have, so I can rewrite some lines.
(I can't run your batch files either and I don't wanna fix'em :shrug: )
Have Fun!
Bandit.
Top
                 
bork[e]
The Borked One
The Borked One
Joined: 23 Mar 2004
Posts: 4708
PostPosted: 05-11-2005 01:35 PM           Profile Send private message  E-mail  Edit post Reply with quote


I didn't use the new version when I just posted my results. Didn't notice that print until I was comparing my results with others.
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-11-2005 03:22 PM           Profile Send private message  E-mail  Edit post Reply with quote


TTK-Bandit wrote:
the problem seems to be that obsidian has no english version of windows.
the date and time variables are diffent on different windows language versions.
so.. I just wrote a little test app, that can do it:
http://mitglied.lycos.de/Q3Coderz/files/

Just copy the 2 files into the "files" directory and run bench.exe
bench.cfg format is pretty simple:
1. line is benchmark version
all lines that follow are benchmark tests in this format:
name|commandline

But I've seen that you build your homepage data out of the output,
so you'd better tell me what output you wanna have, so I can rewrite some lines.
(I can't run your batch files either and I don't wanna fix'em :shrug: )
Have Fun!
Bandit.



Wow, very cool. I'm not that surprised the batch file was the culprit, but I never would have though about different languages being the issue. I don't like batch files anyway, so this is a good reason to get away from it.

I'm at work now but I'll get you the info on the formatting when I get home.
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-11-2005 05:54 PM           Profile Send private message  E-mail  Edit post Reply with quote


Alright here's the format:

Quote:
QUAKE3 MAP BENCHMARK 1.2 - RESULTS
==================================
Map Compile = 00:05
Vis = 01:05
Bspc = 00:22
Lighting = 02:12
Total = 03:44


Line 1 should be exactly as above. I parse to make sure the 24th character is equal to 2, so that results are submitted using the correct version (1.2) of the test.

Line 2 is ignored.

For the next five lines, all that matter is that there is an equal sign somewhere and that there is a time value of the format "mm:ss" to the right of the equal sign. No need to worry about whitespace or text to the left of the equal sign. These lines should be in the order of Map Compile, Vis, Bspc, Lighting, and Total, like above.

There shouldn't be any additional stuff besides these seven lines.
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-11-2005 08:25 PM           Profile Send private message  E-mail  Edit post Reply with quote


bork[e] wrote:
I didn't use the new version when I just posted my results. Didn't notice that print until I was comparing my results with others.


if you post a new score up i'll delete the old one
Top
                 
obsidian
I'm the dude!
I'm the dude!
Joined: 04 Feb 2002
Posts: 12498
PostPosted: 05-11-2005 09:39 PM           Profile Send private message  E-mail  Edit post Reply with quote


If I just compile the map with those compile switches using my own batch file, shouldn't I be able to achieve reliable results and just post the times from the compile log?

And I'm using an English version of WinXP Pro.



_________________
GtkRadiant | Q3Map2 | Shader Manual
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-11-2005 10:05 PM           Profile Send private message  E-mail  Edit post Reply with quote


I'd imagine so. But for consistancy sake I'd run it with the new benchmark tool I created, since there's less chance of your time being off with the rest because the files you have might be different. Then you could post your results here and I'd add them to the webpage.

Or you could wait until TTK-Bandit modifies his progam to output everything correctly, and I'll create a new benchmark tool to download and you can use that.
Top
                 
TTK-Bandit
Insane Quaker
Insane Quaker
Joined: 06 Sep 2002
Posts: 399
PostPosted: 05-12-2005 09:25 AM           Profile Send private message  E-mail  Edit post Reply with quote


go try it :)
Top
                 
Survivor
XXXG-00W0
XXXG-00W0
Joined: 27 Dec 2002
Posts: 2891
PostPosted: 05-12-2005 10:05 AM           Profile Send private message  E-mail  Edit post Reply with quote


There ya got my magnificent results. Maybe i'll retry when i do a format, my system is a bit clogged.
Top
                 
TTK-Bandit
Insane Quaker
Insane Quaker
Joined: 06 Sep 2002
Posts: 399
PostPosted: 05-12-2005 10:29 AM           Profile Send private message  E-mail  Edit post Reply with quote


optimized a few things (will not change results ;)) and updated the package
I'm not looking very often into this forum, so I added the sourcecode to the package if there's need to change something.
Top
                 
Survivor
XXXG-00W0
XXXG-00W0
Joined: 27 Dec 2002
Posts: 2891
PostPosted: 05-12-2005 10:33 AM           Profile Send private message  E-mail  Edit post Reply with quote


Maybe just an idea for you to look into but some programs check on processor, ram and os version themselves. Maybe try to integrate that into the results for generalized standard.
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-12-2005 11:32 AM           Profile Send private message  E-mail  Edit post Reply with quote


Survivor wrote:
Maybe just an idea for you to look into but some programs check on processor, ram and os version themselves. Maybe try to integrate that into the results for generalized standard.


ya know... that's a really good idea. mebbe when i get some time after school ends in a couple of weeks i'll experiment with that.
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-12-2005 11:39 AM           Profile Send private message  E-mail  Edit post Reply with quote


TTK-Bandit wrote:
optimized a few things (will not change results ;)) and updated the package
I'm not looking very often into this forum, so I added the sourcecode to the package if there's need to change something.


checking now. thanks for your help, i'll add in you in for credit when i update the site with the new benchmark... hopefully sometime tonight.

i'm also glad you including source. i'm thinking of playing around with an auto-submit to website feature, so people don't have to do the copy and paste thing. also the idea of automatically getting cpu/ram info sounds neat, whenever i might get a moment to look into that!
Top
                 
TTK-Bandit
Insane Quaker
Insane Quaker
Joined: 06 Sep 2002
Posts: 399
PostPosted: 05-12-2005 12:31 PM           Profile Send private message  E-mail  Edit post Reply with quote


writing system info is a good idea, but to submit automatically not!
I don't like programms that access internet without permission.
if you add it, ask the user at least if he wants to..
Top
                 
TTK-Bandit
Insane Quaker
Insane Quaker
Joined: 06 Sep 2002
Posts: 399
PostPosted: 05-12-2005 01:38 PM           Profile Send private message  E-mail  Edit post Reply with quote


whenever you might get a moment to look into that: ;)

edit: (removed code snippet for os, cpu, ram detection)
never mind.. I added it to the bench.exe and uploaded it on a seperate file to my webspace.
Sry dunno why I'm this generously today :dork:


ah nearly forgot:
you just need to adapt your web script to read the new format:

QUAKE3 MAP BENCHMARK 1.2 - RESULTS
==================================
OS = WinXP
CPU = Intel w/ SSE2 2670 MHz
RAM = 511 MByte
==================================
Map Compile = 00:05
...
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-13-2005 12:45 AM           Profile Send private message  E-mail  Edit post Reply with quote


Thanks for you generousity TTK-Bandit, even if you don't know why :)

I updated the tool with your program, so hopefully the wierd issue that obsidian had will go away, even if it wasn't related to the date thing. Give it another go, obsidian :)

I'm waiting a bit to impliment the bench2 program that you made, giving you system info, as I want to see if it's possible to get more specific cpu info. I notice when I go to system properties in windows I'm told I'm running an Athlon XP processor, but the cpu detail in the benchmark says only its an AMD with 3DNow. I found an article here,

http://www.codeguru.com/Cpp/W-P/system/ ... /c9087__2/

that offer another way to get cpu info, using windows system registry. I wonder if there would be any downsides to doing it this way. I'll have to delay looking at this for a couple of weeks, as I've procrastinated enough as it is with school work :(
Top
                 
TTK-Bandit
Insane Quaker
Insane Quaker
Joined: 06 Sep 2002
Posts: 399
PostPosted: 05-13-2005 03:16 AM           Profile Send private message  E-mail  Edit post Reply with quote


I've seen that too, but I'm not sure if all operating systems support these values..
winxp does I think win2k will too, but what with nt or 9x ?
well fortunately I have win98se on my laptop so I can tell you, it has only these values:
Identifier
Update Status
VendorIdentifier
and as you may have seen, they are not the values you are looking for.
you need to find another way...
Top
                 
TTK-Bandit
Insane Quaker
Insane Quaker
Joined: 06 Sep 2002
Posts: 399
PostPosted: 05-13-2005 05:02 AM           Profile Send private message  E-mail  Edit post Reply with quote


ok this time I know why I'm so generously today..
I've learned something new :D
got something for ya:

Output:
Quote:
Type: GenuineIntel
Name: Intel(R) Pentium(R) 4 CPU 2.66GHz
Features: FPU, MMX, SSE, SSE2, CLFLUSH, RDTSC, CMPXCHG8B, CMOV
Extended Features: VME, DE, PSE, MSR, PAE, MCE, APIC, SEP, MTRR, PGE, MCA, PAT,
PSE36, FXSR, DS, SS, TM

just tell me, what how you want them in the results.txt
Type and Extended Features may be irrelevant, so tell me if you want them or not.
(I'll leave the source inside, if you at some time want those, but I'll comment'em out)

edit: maybe this way?:
CPU = Intel(R) Pentium(R) 4 CPU 2.66GHz | FPU, MMX, SSE, SSE2, CLFLUSH, RDTSC, CMPXCHG8B, CMOV
Top
                 
obsidian
I'm the dude!
I'm the dude!
Joined: 04 Feb 2002
Posts: 12498
PostPosted: 05-13-2005 06:47 AM           Profile Send private message  E-mail  Edit post Reply with quote


BTW, the Q3Map2 version you have in the zip is 2.5.11. You may want to update that with 2.5.16 since there are some additional compiler optimizations since then.

The Bench.exe works great. Thanks for looking into this. I'll post some more accurate benchmark results later, since I tried it now with a bunch of other stuff running.



_________________
GtkRadiant | Q3Map2 | Shader Manual
Top
                 
obsidian
I'm the dude!
I'm the dude!
Joined: 04 Feb 2002
Posts: 12498
PostPosted: 05-13-2005 08:38 AM           Profile Send private message  E-mail  Edit post Reply with quote


Actually, looks like 2.5.11 is slightly faster overall:

2.5.11
Map Compile= 30125ms
Vis= 46984ms
Bspc= 35062ms
Lightning= 164438ms
Total Time= 276609ms

2.5.16
Map Compile= 29891ms
Vis= 47516ms
Bspc= 47797ms
Lightning= 174000ms
Total Time= 299204ms



_________________
GtkRadiant | Q3Map2 | Shader Manual
Top
                 
Fjoggs
Old Skool'
Old Skool'
Joined: 02 May 2002
Posts: 5230
PostPosted: 05-13-2005 09:04 AM           Profile Send private message  E-mail  Edit post Reply with quote


Why's bspc different when it's the same compiler used for that specific progress (?)
Top
                 
obsidian
I'm the dude!
I'm the dude!
Joined: 04 Feb 2002
Posts: 12498
PostPosted: 05-13-2005 09:12 AM           Profile Send private message  E-mail  Edit post Reply with quote


Not sure, but looking through it again, it's safe to say that something was wrong with the above results... 30 seconds for BSP? I just noticed that after posting the results.

Anyway, recompiled with 2.5.11 after restarting my computer and these look like better results. I'll try with 2.5.16 again later:

2.5.11
Map Compile= 4203ms
Vis= 43703ms
Bspc= 24609ms
Lightning= 115641ms

mjrpes: Please remove the slower results that I posted on the website. Thanks.



_________________
GtkRadiant | Q3Map2 | Shader Manual
Top
                 
obsidian
I'm the dude!
I'm the dude!
Joined: 04 Feb 2002
Posts: 12498
PostPosted: 05-13-2005 09:34 AM           Profile Send private message  E-mail  Edit post Reply with quote


Q3Map2 2.5.16 Results:

Map Compile= 4594ms
Vis= 43234ms
Bspc= 24672ms
Lightning= 112078ms
Total Time= 184578ms

Q3Map2 2.5.16 is faster overall by about 4 seconds. BSPC is the same.



_________________
GtkRadiant | Q3Map2 | Shader Manual
Top
                 
Fjoggs
Old Skool'
Old Skool'
Joined: 02 May 2002
Posts: 5230
PostPosted: 05-13-2005 10:04 AM           Profile Send private message  E-mail  Edit post Reply with quote


Yeah that made more sense. :)
Top
                 
obsidian
I'm the dude!
I'm the dude!
Joined: 04 Feb 2002
Posts: 12498
PostPosted: 05-13-2005 10:07 AM           Profile Send private message  E-mail  Edit post Reply with quote


Noticed in the results.txt it says "lightning" instead of "lighting".



_________________
GtkRadiant | Q3Map2 | Shader Manual
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-13-2005 10:50 AM           Profile Send private message  E-mail  Edit post Reply with quote


TTK-Bandit wrote:
ok this time I know why I'm so generously today..
I've learned something new :D
got something for ya:

Output:
Quote:
Type: GenuineIntel
Name: Intel(R) Pentium(R) 4 CPU 2.66GHz
Features: FPU, MMX, SSE, SSE2, CLFLUSH, RDTSC, CMPXCHG8B, CMOV
Extended Features: VME, DE, PSE, MSR, PAE, MCE, APIC, SEP, MTRR, PGE, MCA, PAT,
PSE36, FXSR, DS, SS, TM

just tell me, what how you want them in the results.txt
Type and Extended Features may be irrelevant, so tell me if you want them or not.
(I'll leave the source inside, if you at some time want those, but I'll comment'em out)

edit: maybe this way?:
CPU = Intel(R) Pentium(R) 4 CPU 2.66GHz | FPU, MMX, SSE, SSE2, CLFLUSH, RDTSC, CMPXCHG8B, CMOV



I think you can leave out the stuff to the right of the pipe symbol. What I'll do is let the user submit what's in the text file, and then I'll allow them to edit the processor info before they finally submit it, so they can include additional info like cache size, multi processors, and Hyper Threading. The stuff like FPU, MMX, SSE, etc, you can already deduce that it exists, based on what the processor is (Pentium 4, AMD Athlon, etc), so I don't think it's necessary. That's my take on it.

The RAM and OS info are also important. So the three additional fields in the Results.txt file should be,

CPU, RAM, OS

You can put it in any order you want, and you can put in a blank/dividing line in the results file to make everything look clearer if you want to. I'll just take what you have done and make my php script match it accordingly.

Thanks again :)
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-13-2005 10:56 AM           Profile Send private message  E-mail  Edit post Reply with quote


obsidian wrote:
BTW, the Q3Map2 version you have in the zip is 2.5.11. You may want to update that with 2.5.16 since there are some additional compiler optimizations since then.



I should have updated q3map2 to the most recent version when I redid the benchmark to version 1.2 and reset all the results. I don't know if I want to do it again, because then everyone would have to redo the test if we want the results to be consistent.

But I'm glad it finally worked for you. :)
Top
                 
TTK-Bandit
Insane Quaker
Insane Quaker
Joined: 06 Sep 2002
Posts: 399
PostPosted: 05-13-2005 12:26 PM           Profile Send private message  E-mail  Edit post Reply with quote


OK did it.. You can now download the (hopefully) last version: 1.2
(again with sourcecode)
Top
                 
mjrpes
Elite
Elite
Joined: 28 Nov 2000
Posts: 9847
PostPosted: 05-14-2005 03:35 PM           Profile Send private message  E-mail  Edit post Reply with quote


updated with system info included in results :up:



_________________
yea i've too been kind of thinking about maybe a new sig but sort of haven't come to quite a decision yet
Top
                 
Hr.O
brushmangler
brushmangler
Joined: 11 Jul 2000
Posts: 3760
PostPosted: 05-24-2005 10:11 AM           Profile Send private message  E-mail  Edit post Reply with quote


Silly benchmarking :)

Just a warning for you guys, do not rely too much on these benchmarks. Since you are all running it on your "working" systems, this benchmark is bound to be pretty subjective.

More often then not it's the background software that dictates the speed of your system, and that makes it dangerous to compare this benchmark with anything else then previous results from your own system. To make this long story a bit shorter, the only objective thing you can conclude is "What background programs/services make my computer run slower/faster then before"

It's a pitty we lost a great deal of forum stuff in the past, because this a recurring discussion. We've had it a few times before.
digging into memory I remember once in the time you could still time the BSP fase of q3dm7sample.map in minutes :icon26: think it was round 2k2.

Hr.O
Top
                 
Quake3World.com | Forum Index | Level Editing & Modeling


Post new topic Reply to topic


cron
Quake3World.com
© ZeniMax. Zenimax, QUAKE III ARENA, Id Software and associated trademarks are trademarks of the ZeniMax group of companies. All rights reserved.
This is an unofficial fan website without any affiliation with or endorsement by ZeniMax.
All views and opinions expressed are those of the author.