 |
BorlandTalk.com
Borland discussion newsgroups
|
| View previous topic :: View next topic |
| Author |
Message |
Trevor Thomson
Guest
|
 Posted: Thu Feb 02, 2006 10:50 pm Post subject: STL set memory problems |
 |
|
Hi,
I'm using BCB5 update1 WinXP.
I have a problem with using the set container where if I fill it up with
items (ints in the test) and then clear it, it doesn't seem to give back the
memory it used. Also, it seems to be using far too much memory as well.
Have a look at this simple example:
Create a simple application with a form. Put a button on the form. Add the
typdef for the set and the private variable "mySet".
In the button OnClick event add the listed code. It inserts 3 million ints
into the set. Then it clears the set.
=================== test.h ============================
//snip
#include <set>
typedef std::set< int, std::less< int > > intSet;
//---------------------------------------------------------------------------
class TForm1 : public TForm
{
__published: // IDE-managed Components
TButton *Button1;
void __fastcall Button1Click(TObject *Sender);
private: // User declarations
intSet mySet;
public: // User declarations
__fastcall TForm1(TComponent* Owner);
};
//snip
=================== test.cpp ==========================
void __fastcall TForm1::Button1Click(TObject *Sender)
{
for(int i=0;i<3000000;i++)
{
mySet.insert(i);
}
mySet.clear();
}
===================================================
The problem I observe is when I watch the memory usage in task manager after
pressing the button. It increases as the ints are inserted but doesn't go
down when clear is called.
Before button press: memory is ~8MB, VM is ~4MB
After button press: memory is ~67MB, VM is ~62MB
Problem 1: So the set doesn't give back the memory until it is destroyed,
quite a problem as I see it. It does seem to re-use the memory if you click
the button again. A second press of the button doesn't cause the memory to
go up again.
Problem 2: It seems to use far too much memory. By my calculation
3,000,000 ints (4 bytes) is 12,000,000 bytes or approx 12MB. The memory
usage goes up by 117MB (59MB+58MB)
Is this a problem with the Rogue Wave STL? Is it a BCB5 problem? Can
anyone verify this behaviour or offer a solution?
Thanks,
Trevor.
(sorry for the long post, I tried to be as specific as possible)
|
|
| Back to top |
|
 |
Rob
Guest
|
| Posted: Thu Feb 02, 2006 11:09 pm Post subject: Re: STL set memory problems |
|
|
Trevor Thomson wrote:
| Quote: | Hi,
I'm using BCB5 update1 WinXP.
I have a problem with using the set container where if I fill it up
with items (ints in the test) and then clear it, it doesn't seem to
give back the memory it used. Also, it seems to be using far too
much memory as well. Have a look at this simple example:
Create a simple application with a form. Put a button on the form.
Add the typdef for the set and the private variable "mySet".
In the button OnClick event add the listed code. It inserts 3
million ints into the set. Then it clears the set.
=================== test.h ============================
//snip
#include <set
typedef std::set< int, std::less< int > > intSet;
//--------------------------------------------------------------------
------- class TForm1 : public TForm
{
__published: // IDE-managed Components
TButton *Button1;
void __fastcall Button1Click(TObject *Sender);
private: // User declarations
intSet mySet;
public: // User declarations
__fastcall TForm1(TComponent* Owner);
};
//snip
=================== test.cpp ==========================
void __fastcall TForm1::Button1Click(TObject *Sender)
{
for(int i=0;i<3000000;i++)
{
mySet.insert(i);
}
mySet.clear();
}
===================================================
The problem I observe is when I watch the memory usage in task
manager after pressing the button. It increases as the ints are
inserted but doesn't go down when clear is called.
Before button press: memory is ~8MB, VM is ~4MB
After button press: memory is ~67MB, VM is ~62MB
Problem 1: So the set doesn't give back the memory until it is
destroyed, quite a problem as I see it. It does seem to re-use the
memory if you click the button again. A second press of the button
doesn't cause the memory to go up again.
Problem 2: It seems to use far too much memory. By my calculation
3,000,000 ints (4 bytes) is 12,000,000 bytes or approx 12MB. The
memory usage goes up by 117MB (59MB+58MB)
Is this a problem with the Rogue Wave STL? Is it a BCB5 problem?
Can anyone verify this behaviour or offer a solution?
|
No, it's not a problem with the Rogue Wave STL or with BCB5. A lot of
programs allocate and release memory quite a few times during their
execution time, and having to go back to the operating system every
time that happens can introduce a significant performance hit --- and,
for most programs, run time performance usually has more effect on the
user than peak memory usage. As a performance optimisation (and to
reduce average number of complaints from users to programmers, and
therefore reduce number of complaints by programmers to
compiler/library vendors), the working of dynamic memory allocations
schemes (malloc/free in C, operators new and delete in C++) usually
involves keeping track of memory that has been accessed. This means
that releasing the memory does NOT return it to the operating system
but a subsequent call to (for example) operator new can obtain memory
without having to explicitly ask the operating system for it. A side
effect of that scheme is that, even when your program releases memory,
the memory usage reported by task manager will not necessarily go down.
The memory is not actually lost (or leaked) and will either be reused
when the program uses dynamic memory allocation again, or be recovered
when the program exits. Some advanced schemes might actually detect
when the system is low on memory, and be a little more frugal on peak
memory usage, but I'm not sure if BCB5 or RogueWave try to do this.
As to why the memory used appears more than you asked for: when memory
is dynamically allocated by the program, more is actually requested
from the OS to allow for possible additional demands of the program.
Again, the reasoning is minimising the number of times the program must
go back to the OS, and achieving a performance optimisation. In the
specific case of set, the internal representation of a set may be a
linked list or some other data structure, which means that there is a
memory overhead of using the set, over and above the memory needed to
store the values in it.
As to a solution: I wouldn't worry about it. As long as you haven't
introduced a leak, the impact of having dynamically allocated memory
not being immediately returned to the OS is relatively minor: if it
causes any real problems, there are likely to be other problems of much
greater concern.
|
|
| Back to top |
|
|
Gillmer J. Derge [TeamB]
Guest
|
| Posted: Fri Feb 03, 2006 12:50 am Post subject: Re: STL set memory problems |
|
|
Rob wrote:
| Quote: | As to why the memory used appears more than you asked for: when memory
is dynamically allocated by the program, more is actually requested
from the OS to allow for possible additional demands of the program.
|
He also overlooked quite a bit of memory that will be used by the set.
Each node in a std::set<int> is quite a bit larger than just an int. In
a typical implementation it's the int, a pointer to the parent node, a
pointer to the "left" child, a pointer to the "right" child, and a flag
(some sort of Boolean or integral type) indicating whether the node is
"red" or "black" in a red/black tree. With 32 bit pointers and
integers, that's somewhere in the range of 17 to 20 bytes per node,
depending on the size of the red/black flag.
--
Gillmer J. Derge [TeamB]
|
|
| Back to top |
|
|
Trevor Thomson
Guest
|
| Posted: Fri Feb 03, 2006 12:55 am Post subject: Re: STL set memory problems |
|
|
Hi Rob,
Thanks for the reply and explanation, I did have my suspicions that the
memory wasn't actually "lost" and could be reused.
Do you think it is the set or the Borland Memory Manager that is keeping the
extra memory for reuse? My guess is it is the set because if the set is
destroyed then the memory is released. Also, changing the OnClick event to
the following:
============================
for(int i=0;i<3e6;i++)
{
mySet.insert(i);
}
mySet.clear();
for(int i=0;i<3e6;i++)
{
mySet2.insert(i);
}
mySet2.clear();
===========================
uses twice the memory. If it was the BMM keeping the memory then it would
get reused for the second set. It is documented that vector capacity and
size are different, perhaps the same is so for the set.
I still haven't got my head around if this is going to be a problem for us
or not. Our program uses a lot of memory because we analyse large amounts
of data. It's not unusual for us to have 800MB tied up in sets that are
keeping it for later. I supose that as long as we want to use the memory
again in the set it should be ok. It could be a problem if we wanted that
memory for something else. Another issue is that we have 1GB of RAM in our
machines, having that much memory tied up means we are more likely to be
using the hard drive for the next analysis, which is a performance hit in
itself. Or is windows smart enough to move the less used memory onto the
HD?
"Rob"
No, it's not a problem with the Rogue Wave STL or with BCB5. A lot of
programs allocate and release memory quite a few times during their
execution time, and having to go back to the operating system every
time that happens can introduce a significant performance hit --- and,
for most programs, run time performance usually has more effect on the
user than peak memory usage.
......
As to a solution: I wouldn't worry about it. As long as you haven't
introduced a leak, the impact of having dynamically allocated memory
not being immediately returned to the OS is relatively minor: if it
causes any real problems, there are likely to be other problems of much
greater concern.
|
|
| Back to top |
|
|
Rob
Guest
|
| Posted: Fri Feb 03, 2006 4:09 am Post subject: Re: STL set memory problems |
|
|
Gillmer J. Derge [TeamB] wrote:
| Quote: | Rob wrote:
As to why the memory used appears more than you asked for: when
memory is dynamically allocated by the program, more is actually
requested from the OS to allow for possible additional demands of
the program.
He also overlooked quite a bit of memory that will be used by the
set. Each node in a std::set<int> is quite a bit larger than just an
int. In a typical implementation it's the int, a pointer to the
parent node, a pointer to the "left" child, a pointer to the "right"
child, and a flag (some sort of Boolean or integral type) indicating
whether the node is "red" or "black" in a red/black tree. With 32
bit pointers and integers, that's somewhere in the range of 17 to 20
bytes per node, depending on the size of the red/black flag.
|
Yep. The bit you quoted from me was the first half of a paragraph in
my previous post. The bit of that paragraph you snipped actually
mentioned it may be an artefact of set's implementation.
|
|
| Back to top |
|
|
Rob
Guest
|
| Posted: Fri Feb 03, 2006 4:21 am Post subject: Re: STL set memory problems |
|
|
Trevor Thomson wrote:
| Quote: | Hi Rob,
Thanks for the reply and explanation, I did have my suspicions that
the memory wasn't actually "lost" and could be reused.
Do you think it is the set or the Borland Memory Manager that is
keeping the extra memory for reuse? My guess is it is the set
because if the set is destroyed then the memory is released. Also,
changing the OnClick event to the following:
============================
for(int i=0;i<3e6;i++)
{
mySet.insert(i);
}
mySet.clear();
for(int i=0;i<3e6;i++)
{
mySet2.insert(i);
}
mySet2.clear();
===========================
uses twice the memory. If it was the BMM keeping the memory then it
would get reused for the second set. It is documented that vector
capacity and size are different, perhaps the same is so for the set.
|
Most of the STL containers try to optimise how they use memory which,
in practice, can mean they grab more memory than immediately needed
just in case it eventually will be.
| Quote: |
I still haven't got my head around if this is going to be a problem
for us or not. Our program uses a lot of memory because we analyse
large amounts of data. It's not unusual for us to have 800MB tied up
in sets that are keeping it for later. I supose that as long as we
want to use the memory again in the set it should be ok. It could be
a problem if we wanted that memory for something else. Another issue
is that we have 1GB of RAM in our machines, having that much memory
tied up means we are more likely to be using the hard drive for the
next analysis, which is a performance hit in itself. Or is windows
smart enough to move the less used memory onto the HD?
|
Windows will tend to swap the memory used by inactive processes from
memory to (typically) swap space on hard disk, so that programs which
are active have access to more RAM when they need it. There are
limits on that. For example, if the memory your program uses exceeds
available RAM, parts of your program will have to be swapped in and
out, and system performance will slow. The effect on performance
depends on how you move through the set (eg do you traverse all
elements in the set frequently, or do you usually only access a few at
a time?).
The only real way to know if this gives a performance hit for your
program is to test it.
Your philosophy of having 800MB of data "tied up in sets that are
keeping it for later" is something you may want to look more closely
at. In practice, it is VERY rare that such a set will be kept in
memory because you need to access ALL of it later --- it will be more
usual to need a only small part of it later. If that's true, a bit of
judicious design can allow you to only keep the small part you need in
memory for later. You may also find that, for example, retrieving
data from a well designed database may be a more efficient means of
getting the actual parts of the data you need. In other words, using
efficient means of storing your data on disk, designed to allow for
quick retrieval of particular parts of your data, may actually be more
efficient than keeping large amounts of data in memory at once.
|
|
| Back to top |
|
|
Oliver Rutsch
Guest
|
| Posted: Fri Feb 03, 2006 8:12 am Post subject: Re: STL set memory problems |
|
|
Hi Trevor,
| Quote: | memory for something else. Another issue is that we have 1GB of RAM in our
machines, having that much memory tied up means we are more likely to be
using the hard drive for the next analysis, which is a performance hit in
itself. Or is windows smart enough to move the less used memory onto the
HD?
|
Windows is smart enough, at least Win XP. If you're working on such
large data sets then I would recommend to use the FastMM memory manager
instead of the old BorlandMM. There's a replacement borldnmm.dll which
works very nice with BCB5. But don't forget to link with dynamic RTL then.
With FastMM I saw a speed enhancement of about 10% in one of our
applications and I only need to replace one dll! And when it comes to
multithreaded calculations then FastMM is a must, because the BorlandMM
behaves very very slow in MT applications.
It was really a good idea to use the FastMM in BCB2006!
Bye,
--
Dipl. Ing. Oliver Rutsch
EMail: [email]orutsch (AT) sympatec (DOT) com[/email]
Sympatec GmbH
|
|
| Back to top |
|
|
Andrue Cope [TeamB]
Guest
|
| Posted: Fri Feb 03, 2006 9:23 am Post subject: Re: STL set memory problems |
|
|
Rob wrote:
| Quote: | A lot of
programs allocate and release memory quite a few times during their
execution time, and having to go back to the operating system every
time that happens can introduce a significant performance hit
|
Most OSes also don't have the capacity to track a large number of
allocations. Win32 for instance can only track 65,535 allocations per
process. That's because the OS 'heap manager' is designed to be
minimalist. It has no idea what kind of demands will be made from it so
rather than try and write the perfect manager for all processes the OS
implementor does the least possible and lets the process provide its
own specialised manager.
--
Andrue Cope [TeamB]
[Bicester, Uk]
http://info.borland.com/newsgroups/guide.html
|
|
| Back to top |
|
|
Andrue Cope [TeamB]
Guest
|
| Posted: Fri Feb 03, 2006 9:26 am Post subject: Re: STL set memory problems |
|
|
Trevor Thomson wrote:
| Quote: | Hi Rob,
Thanks for the reply and explanation, I did have my suspicions that
the memory wasn't actually "lost" and could be reused.
|
It 'might' also be the infamous heap fragmentation bug. This is
something that bites some programs from time to time. It is usually
triggered by applications allocating a large number of mixed size
blocks. Apparently FastMM doesn't suffer from this and you can install
that for BCB6.
Us BDS users have it straight out of the box which is a definite
advantage :)
--
Andrue Cope [TeamB]
[Bicester, Uk]
http://info.borland.com/newsgroups/guide.html
|
|
| Back to top |
|
|
Oliver Rutsch
Guest
|
| Posted: Fri Feb 03, 2006 10:06 am Post subject: Re: STL set memory problems |
|
|
Hi Andrue,
| Quote: | It 'might' also be the infamous heap fragmentation bug. This is
something that bites some programs from time to time. It is usually
triggered by applications allocating a large number of mixed size
blocks. Apparently FastMM doesn't suffer from this and you can install
that for BCB6.
|
Some time ago we had serious problems with heap fragmentation. We had
hundredthousands of different size allocations/deallocations and
wondered why we're running out of memory although there was enough free
memory in the system. But fortunately there was FastMM and the problems
disappeared. Thanks to Pierre le Riche for this great stuff!
BTW: In Win XP there's a Low-fragmentation Heap (look for LFH in the
MSDN) you can use instead of the normal heap. But it should be useful
only if you're using VirtualAlloc etc. directly from your program.
Bye,
--
Dipl. Ing. Oliver Rutsch
EMail: [email]orutsch (AT) sympatec (DOT) com[/email]
Sympatec GmbH
|
|
| Back to top |
|
|
Chris Uzdavinis (TeamB)
Guest
|
| Posted: Fri Feb 03, 2006 2:33 pm Post subject: Re: STL set memory problems |
|
|
"Trevor Thomson" <trevor.thomson (AT) Laseranalysis (DOT) com> writes:
| Quote: | Do you think it is the set or the Borland Memory Manager that is keeping the
extra memory for reuse? My guess is it is the set because if the set is
destroyed then the memory is released.
|
This is correct. Actually both cache memory, but the set is more
likely the problem you're seeing. STL containers tend to cache all of
their memory. If you want to clear the underlying memory of an
object, the idiomatic way to do that is to swap it with a temporary.
(That way, the temporary is destroyed and takes the original memory
with it, while the existing object now holds the relatively small
footprint of a default-constructed, empty temporary object.)
// release the memory from mySet
std::set<int>().swap(mySet);
| Quote: | I still haven't got my head around if this is going to be a problem for us
or not. Our program uses a lot of memory because we analyse large amounts
of data. It's not unusual for us to have 800MB tied up in sets that are
keeping it for later. I supose that as long as we want to use the memory
again in the set it should be ok. It could be a problem if we wanted that
memory for something else. Another issue is that we have 1GB of RAM in our
machines, having that much memory tied up means we are more likely to be
using the hard drive for the next analysis, which is a performance hit in
itself. Or is windows smart enough to move the less used memory onto the
HD?
|
As Rob already said, and it is a very good suggestion, you should
consider an alternate way of storing and accessing your data. It is
usually a bad design to store so much data in memory that it causes
you to worry about running out. Inevitably, your data will grow and
then you will use too much. Don't use a design that is marching into
a dead-end.
--
Chris (TeamB);
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
