 |
BorlandTalk.com
Borland discussion newsgroups
|
| View previous topic :: View next topic |
| Author |
Message |
Pierre le Riche
Guest
|
 Posted: Wed May 25, 2005 7:20 am Post subject: Re: Some thoughts re the MM benchmarks |
 |
|
Hi Anders,
| Quote: | in absolute numbers in the result columns. Let the reader apply the
weighing...
|
The weighting is there for uninformed potential users to be able to pick the
best MM for their application. Without the weighting the casual observer
will have no idea which benchmarks are trivial and not really relevant and
which benchmarks are more "real world".
As it is now you can graph specific categories (e.g. only single-threaded
benchmarks or only multi-threaded benchmarks, etc.), so I think all the info
anyone would need is there.
Regards,
Pierre
|
|
| Back to top |
|
 |
Hannes Danzl[NDD]
Guest
|
| Posted: Wed May 25, 2005 10:14 am Post subject: Some thoughts re the MM benchmarks |
|
|
I've had the time today to look a bit into the current benchmarks and
memory managers. Please don't understand the points I raise as critic
on the benchmark or on the people involved.
First thing I saw is the inclusion of a lot of single threaded
benchmarks. That itself is imo not problem of course, except that I
think it's not really real-world like to add them in a way to the
benchmark application so that locking MM's can use the IsMultiThreaded
to completely avoid locking. Most real-world applications these days
will not have IsMultiThreaded set to true anymore especially if they
use databases, internet access (e.g. auto updates), ... I think that
you should either run the tests twice with each setting for
IsMultiThreaded or (a bit drastic i know) not allow the use of it at
all. I think it's a bit misleading for MOST users of the alternative
memory managers.
Secondly, I would like to see massively threaded tests that are run on
fast multi cpu machines like dual/quad AMD Opterons with full 2gig ram
available. These tests should be designed to provoke heavy contention.
After all these are the machines that are used for applications that
struggle with memory/cpu load often rooting in problems with the stock
MM. It would be very interesting to see how each of the entrants deals
with that.
Lastly i think it would make sense to create two sub scores for the
results table, one for single threaded tests and one for multithreaded
one (again please make sure to be as close to real world). E.g. for
selecting the correct MM for a heavy threaded network service or
database server the scores of single threaded performance play a much
lower role than the singlethreaded ones.
I've no idea where NexusMM will stand in this, i've not even run the
tests here, that's just stuff that came to mind when looking through
some sources
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at http://www.tamaracka.com/search.htm
|
|
| Back to top |
|
|
Dennis
Guest
|
| Posted: Wed May 25, 2005 10:44 am Post subject: Re: Some thoughts re the MM benchmarks |
|
|
Hi Hannes
I generally agree with you.
A few short comments.
The weigth of benchmarks are not based on the number of benchmarks in a
category. We use an explicit setting.
Do any MM's use the IsMultiThreaded variable?
To all: Please post benchmark and validation results from all the PC's you
have. Multiprocessor runs are expecially welcome.
Regards
Dennis
|
|
| Back to top |
|
|
Robert Houdart
Guest
|
| Posted: Wed May 25, 2005 10:51 am Post subject: Re: Some thoughts re the MM benchmarks |
|
|
Hello Hannes,
Thanks for your comments.
1) The benchmark system is based on a 40 % single-thread, 60 % multi-thread
weighting. This is quite reasonable (and see also point 4 below).
2) It's perfectly smart to take advantage of the IsMultiThread variable. If
you know you're running a single-threaded application, you do not need to
lock the memory.
3) In the multi-threaded benchmarks we have *a lot* more contention than you
would find in any real-world situation.
4) Currently the fastest MM in single-thread are also the fastest in
multi-thread.
5) Of course you know where NexusMM stands in this, it's way behind the
current leaders, both in the single-threaded and the multi-threaded
benchmarks.
Take a loook at some of the results Dan Downs posted yesterday on a bi-CPU
machine. Speed-wise NexusMM is at 55.9% of the top performer, memory-wise at
76.7%, combined at 67.5%. That's massive.
Kind regards,
Robert
|
|
| Back to top |
|
|
Hannes Danzl[NDD]
Guest
|
| Posted: Wed May 25, 2005 11:22 am Post subject: Re: Some thoughts re the MM benchmarks |
|
|
| Quote: | 1) The benchmark system is based on a 40 % single-thread, 60 %
multi-thread weighting. This is quite reasonable (and see also point
4 below).
2) It's perfectly smart to take advantage of the IsMultiThread
variable. If you know you're running a single-threaded application,
you do not need to lock the memory.
|
as mentioned, it's fine for "benchmarks for benchmark reasons", but i
think the bench application should have the *option* to disallow it.
"Single threaded" speed IS important but to assume that because a
worker routine is single threaded that there are no other threads
created/active in the application is not real-world like and thus
misleading people.
| Quote: | 3) In the multi-threaded benchmarks we have *a lot* more contention
than you would find in any real-world situation.
|
i don't agree, sorry. try to run a highly threaded database server with
50+ concurrent users doing queries, batch updates, inserts, updates in
fully transactional environments on a quad cpu machine and you see what
i mean. but of course that's a very specific need. Also look at a
massive size clustered bitmap calculations etc, ...
| Quote: | 4) Currently the fastest MM in single-thread are also the fastest in
multi-thread.
|
with a good design that's logical isn't it?
| Quote: | 5) Of course you know where NexusMM stands in this, it's way behind
the current leaders, both in the single-threaded and the
multi-threaded benchmarks.
|
no i don't know where nexusdb WILL be standing with such changes. I've
not said i don't know where we ARE standing, of course i know.
| Quote: | Take a loook at some of the results Dan Downs posted yesterday on a
bi-CPU machine. Speed-wise NexusMM is at 55.9% of the top performer,
memory-wise at 76.7%, combined at 67.5%. That's massive.
|
It is, but that was not the point of my post, if you have read the top
few sentences. I don't think you achieve anything really with not
making the benchmark as close to real world as possible. And let's face
it: the small developer running a small app that uses 120 megs ram and
shut down after a day on a single cpu is not really the target audience
is it?
what you are looking at are heavy threaded server application, high
memory usage number crunchers, highly parallel worker threads, etc.
At least the test should have some weighing schemas like: Desktop
application, HighThread server, HighMemoryLoad and so on which the
interested user can apply. I also strongly vote for a forced
IsMultiThreaded setting option.
Again, that has NOTHING whatsoever to do where we stand. I don't think
it will change our position, but it might point out some problems with
each entrant for a certain scenario.
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at http://www.tamaracka.com/search.htm
|
|
| Back to top |
|
|
Hannes Danzl[NDD]
Guest
|
| Posted: Wed May 25, 2005 11:29 am Post subject: Re: Some thoughts re the MM benchmarks |
|
|
| Quote: | I generally agree with you.
|
:)
| Quote: | The weigth of benchmarks are not based on the number of benchmarks in
a category. We use an explicit setting.
|
sure, i've seen. that's of course very important.
| Quote: | Do any MM's use the IsMultiThreaded variable?
|
yes. I've no idea how much influence it has and how they use it, but in
theory it's possible to simply skip any locking in that mode. We had
that in an internal version to skip some code (we don't use locks but
still can skip/replace costly code) and the improvements were for
certain cases quite big.
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at http://www.tamaracka.com/search.htm
|
|
| Back to top |
|
|
Robert Houdart
Guest
|
| Posted: Wed May 25, 2005 11:54 am Post subject: Re: Some thoughts re the MM benchmarks |
|
|
Hello Hannes,
| Quote: | "Single threaded" speed IS important but to assume that because a
worker routine is single threaded that there are no other threads
created/active in the application is not real-world like and thus
misleading people.
|
As said, single-thread gets 40% of the over-all weight, multi-thread 60 %.
That's a fair balance and does not mislead any potential users.
| Quote: | i don't agree, sorry. try to run a highly threaded database server with
50+ concurrent users doing queries, batch updates, inserts, updates in
fully transactional environments on a quad cpu machine and you see what
i mean. but of course that's a very specific need. Also look at a
massive size clustered bitmap calculations etc, ...
|
Our multi-threaded benchmarks execute *only* memory management operations.
In a normal application the MM would only consume around 10% of the CPU, so
we effectively have a *10 times higher* contention rate than you would get
in a real application.
The eLink benchmarks runs with 16 threads doing nothing else than allocating
mostly blocks smaller than 32 bytes. That's what I call "contention".
| Quote: | what you are looking at are heavy threaded server application, high
memory usage number crunchers, highly parallel worker threads, etc.
|
The Link Server benchmark is a replay log from a heavy threaded database
server application. In this benchmark NexusMM is at about 50% of the
performance of the leaders.
| Quote: | At least the test should have some weighing schemas like: Desktop
application, HighThread server, HighMemoryLoad and so on which the
interested user can apply. I also strongly vote for a forced
IsMultiThreaded setting option.
|
Your comments suggest that you think you would get different results or
winners by splitting the benchmarks results in these categories.
You're wrong.
Please prove the contrary by entering *any* benchmark on which NexusMM would
perform significantly better than the current benchmark results suggest.
That's a challenge!
Regards,
Robert
|
|
| Back to top |
|
|
Hannes Danzl[NDD]
Guest
|
| Posted: Wed May 25, 2005 12:00 pm Post subject: Re: Some thoughts re the MM benchmarks |
|
|
| Quote: | Your comments suggest that you think you would get different results
or winners by splitting the benchmarks results in these categories.
You're wrong.
|
No. But maybe one or the other might perform better than an all in one
solution for certain tasks?
| Quote: | Please prove the contrary by entering any benchmark on which NexusMM
would perform significantly better than the current benchmark results
suggest. That's a challenge!
|
already commented on the purpose of this post, that must do.
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at http://www.tamaracka.com/search.htm
|
|
| Back to top |
|
|
Robert Houdart
Guest
|
| Posted: Wed May 25, 2005 12:10 pm Post subject: Re: Some thoughts re the MM benchmarks |
|
|
Hello Dennis,
| Quote: | Do any MM's use the IsMultiThreaded variable?
|
Yes, most do. RTL MM, BucketMM, FastMM, RecyclerMM, WinMem... all use this
feature.
This makes a lot of sense: if you know you're running a single-threaded
application there's no point in having the performance penalty of protecting
the MM against multi-threaded access.
Regards,
Robert
|
|
| Back to top |
|
|
Eric Grange
Guest
|
| Posted: Wed May 25, 2005 12:22 pm Post subject: Re: Some thoughts re the MM benchmarks |
|
|
| Quote: | [...] Most real-world applications these days
will not have IsMultiThreaded set to true anymore
|
If they don't have it set, they'll experience crashes in the RTL
as Borland relies on it for their default allocator, therefore,
not setting IsMultiThread can be considered a bug (at worst, to test
its influence, it's easy to force it to True at the start of the
becnhmark).
| Quote: | Secondly, I would like to see massively threaded tests that are run on
fast multi cpu machines like dual/quad AMD Opterons with full 2gig ram
available.
|
There are already massively multithreaded tests, though I would also
like to see more runs from machines with more than 2 cores, there seems
to be an availability issue. The tests on dual CPU and dual CPU+HT
didn't seem to indicate bad behaviour for BucketMem or FastMM though.
| Quote: | These tests should be designed to provoke heavy contention.
|
Most are already irrealistically stressful for the MM IMO,
with practically only allocs/reallocs/frees and little access
to the allocated memory.
It might be interesting to have some less deterministic memory
usages (by that I don't mean more random, but with more loops
and code in between, so that the MM's code isn't fully in L1 cache,
or the jump predictions and indirections aren't so predictible
for the CPU).
| Quote: | Lastly i think it would make sense to create two sub scores for the
results table, one for single threaded tests and one for multithreaded
one (again please make sure to be as close to real world).
|
Yep.
Eric
|
|
| Back to top |
|
|
Eric Grange
Guest
|
| Posted: Wed May 25, 2005 12:26 pm Post subject: Re: Some thoughts re the MM benchmarks |
|
|
| Quote: | This makes a lot of sense: if you know you're running a single-threaded
application there's no point in having the performance penalty of protecting
the MM against multi-threaded access.
|
Yes, the 'lock' prefix can have a relatively huge cost, wether or not
there is actually a conflict or not.
The instruction count for most calls to the fast MMs is so low that just
about anything will have a measurable impact. Testing IsMultiThread
means extra instructions, but since it's well predicted, the penalty
is rather puny.
Eric
|
|
| Back to top |
|
|
Dennis
Guest
|
| Posted: Wed May 25, 2005 12:46 pm Post subject: Re: Some thoughts re the MM benchmarks |
|
|
Hi Both of you especially Robert
This is not a discission about Nexus versus BucketMM ! Hannes emphazised
this.
I think Hannes comments are making sense, we just have considered them quite
well. The B&V is quite realworld and is a very tough test. We just need to
collect a lot of benchmark results on 2 and 4 processor machines as soon as
we reach release 1.0.
We also need to add a few more replays. We have two pending.
It is very important we take these discussions now and are willing to change
the B&V according to the outcome.
I will kindly ask more people to throw in their opinion.
Regards
Dennis
|
|
| Back to top |
|
|
Michael Moreno
Guest
|
| Posted: Wed May 25, 2005 12:50 pm Post subject: Re: Some thoughts re the MM benchmarks |
|
|
Hi Dennis,
I can run from time to time if needed some tests on a Quad CPU machine.
Would that help?
--
----
http://michael.moreno.free.fr/
|
|
| Back to top |
|
|
Dennis
Guest
|
| Posted: Wed May 25, 2005 12:53 pm Post subject: Re: Some thoughts re the MM benchmarks |
|
|
Hi Michael
Oh yes a lot.
Report your resultsin this NG and I will put them in the spreadsheet.
Do it whenever you will.
Regards
Dennis
|
|
| Back to top |
|
|
Michael Moreno
Guest
|
| Posted: Wed May 25, 2005 12:55 pm Post subject: Re: Some thoughts re the MM benchmarks |
|
|
| Quote: | It is, but that was not the point of my post, if you have read the top
few sentences. I don't think you achieve anything really with not
making the benchmark as close to real world as possible. And let's face
it: the small developer running a small app that uses 120 megs ram and
shut down after a day on a single cpu is not really the target audience
is it?
|
Well actually this is the case for many Delphi developpers !
Changing the MM is actually one of the easiest way to :
- reduce memory usage of your app
- speed it up
So there is really no reasons to actually not change it even for a
simple app.
--
----
http://michael.moreno.free.fr/
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
