BorlandTalk.com Borland discussion newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  Memory Allocation Goto page 1, 2  Next          BorlandTalk.com Forum Index -> C++ Builder (Non-Technical) View previous topic :: View next topic   Author Message Arnie Mauer Guest Posted: Thu Jul 29, 2004 11:52 pm    Post subject: Memory Allocation Sorry if this is the wrong group. Please redirect it if required. We have to allocate large blocks of contigous memory. Typically a vector, but using a dequeue in a pinch. I have to go to the Intel/HP 64 bit porting workshop next week. I have little experience with MS VS and their C++. (Using MS VS 2005 beta). I decided to write a memory 'intensive' program. See the source later. I compiled it with both BCB v5 and VC++ 8. Very interesting. BCB could allocate up to 1.1 GB and then die. It did this in an 'eye blink'. VC++ took time to allocate everything, but just kept going. See the BCB output below: Allocated 100000000 bytes. Allocated 200000000 bytes. Allocated 300000000 bytes. Allocated 400000000 bytes. Allocated 500000000 bytes. Allocated 600000000 bytes. Allocated 700000000 bytes. Allocated 800000000 bytes. Allocated 900000000 bytes. Allocated 1000000000 bytes. Allocated 1100000000 bytes. bad alloc exception thrown This happens in about a second or so. I think someone is lying. The same code under VS will get up to about 1.4 GB before I lose patience. My laptop has 1GB RAM. I have run both of these on 2GB and 4GB servers. BCB does the same, exactly. The VC version gets up to about 1.8 GB before I lose patience (on the 4GB server, which had a 3GB working set before I started). What the dickens is wrong with BCB herre? - Arnie // Borland version // Alloc32.cpp : Defines the entry point for the console application. // #include <iostream> using namespace std; int _tmain(int argc, _TCHAR* argv[]) { std::size_t i; unsigned char * mem; for ( i = 1; i <= 20; ++i ) { try { mem = new unsigned char[i * 100000000]; cout << "Allocated " << i * 100000000 << " bytes." << endl; delete [] mem; } catch( std::exception & e ) { cout << e.what() << endl; return 16; } } return 0; } Back to top Andy Walker Guest Posted: Fri Jul 30, 2004 10:33 am    Post subject: Re: Memory Allocation What a reall interesting problem. After looking at the code for a few minutes and changing a few things, I have come to the conclusion that the problem has something to do with the 'new' operator. If you replace 'new' with 'malloc' BCB does the allocation upto 2GB no problem at all, and in the blink of an eye. #include <iostream> using namespace std; int _tmain(int argc, _TCHAR* argv[]) { std::size_t i; unsigned char * mem; for ( i = 1; i <= 20; ++i ) { try { //mem = new unsigned char[i * 100000000]; mem = (unsigned char*) malloc(i*100000000); cout << "Allocated " << i * 100000000 << " bytes." << endl; //free (mem); delete [] mem; } catch( std::exception & e ) { cout << e.what() << endl; return 16; } } return 0; } Back to top Andy Walker Guest Posted: Fri Jul 30, 2004 10:37 am    Post subject: Re: Memory Allocation ...oops, my mistake. It isn't actually allocating beyond 1.1GB at all, it is just that malloc doesn't throw an exception. DOH! Still interesting though! Andy Back to top Yahia El-Qasem Guest Posted: Fri Jul 30, 2004 11:06 am    Post subject: Re: Memory Allocation there is an option for max heap size on the linker tab of project options... perhaps this has something to do with what you in BCB... Yahia Back to top Arnie Mauer Guest Posted: Fri Jul 30, 2004 3:33 pm    Post subject: Re: Memory Allocation "Yahia El-Qasem" <Yahia.El-Qasem (AT) mgh (DOT) metro-ag.de> wrote Quote: there is an option for max heap size on the linker tab of project options... perhaps this has something to do with what you in BCB... Yahia I doubt it. The default max heap size is 1MB. - Arnie Back to top Arnie Mauer Guest Posted: Fri Jul 30, 2004 3:34 pm    Post subject: Re: Memory Allocation "Andy Walker" <enquiries (AT) NOiolaSPAM (DOT) co.uk> wrote Quote: ..oops, my mistake. It isn't actually allocating beyond 1.1GB at all, it is just that malloc doesn't throw an exception. DOH! Still interesting though! Andy Whew! You had me worried there for a moment ;-) - Arnie Back to top Tamas Demjen Guest Posted: Fri Jul 30, 2004 6:04 pm    Post subject: Re: Memory Allocation Quote: mem = new unsigned char[i * 100000000]; You may want to try to use VirtualAlloc directly: VirtualAlloc(NULL, 100000000, MEM_COMMIT, PAGE_READWRITE); You can not go lower level than that. Use VirtualFree(p, 0, MEM_RELEASE); to delete the allocated pointer. I use VirtualAlloc in some cases to avoid the disastrous fragmentation bug in Borland's memory manager. Tom Back to top Andre Kaufmann Guest Posted: Fri Jul 30, 2004 6:09 pm    Post subject: Re: Memory Allocation VC uses the Windows heap manager to allocate memory (on Win2K and XP), while BCB uses its own heap manager, at least for small memory blocks. Don´t know it for sure for large memory blocks. But since the heap manager sources are distributed with BCB you may have a look at them and find it out. For VC the behavior may be changed by using _set_sb_treshold. Perhaps the borland heap manager may be reconfigured, too ? Also i think you compiled a VC debug version, which fills all the allocated memory with a special byte sequence. Cheers Andre "Arnie Mauer" <xxx (AT) nowhere (DOT) net> wrote Quote: Sorry if this is the wrong group. Please redirect it if required. We have to allocate large blocks of contigous memory. Typically a vector, but using a dequeue in a pinch. I have to go to the Intel/HP 64 bit porting workshop next week. I have little experience with MS VS and their C++. (Using MS VS 2005 beta). I decided to write a memory 'intensive' program. See the Back to top Arnie Mauer Guest Posted: Fri Jul 30, 2004 6:50 pm    Post subject: Re: Memory Allocation "Tamas Demjen" <tdemjen (AT) yahoo (DOT) com> wrote Quote: mem = new unsigned char[i * 100000000]; You may want to try to use VirtualAlloc directly: VirtualAlloc(NULL, 100000000, MEM_COMMIT, PAGE_READWRITE); You can not go lower level than that. Use VirtualFree(p, 0, MEM_RELEASE); to delete the allocated pointer. I use VirtualAlloc in some cases to avoid the disastrous fragmentation bug in Borland's memory manager. Tom Thanks Tom. I presented a pretty simple-minded test program. Our problems usually occur with, eg, vector::push_back(), when the vector doubles in size. I suppose we could provide our own allocator and give it a shot. - Arnie Back to top Arnie Mauer Guest Posted: Fri Jul 30, 2004 6:54 pm    Post subject: Re: Memory Allocation "Andre Kaufmann" <##andre_no_spam_.kaufmann_ (AT) t-online (DOT) de##> wrote in message news:410a8d49 (AT) newsgroups (DOT) borland.com... Quote: VC uses the Windows heap manager to allocate memory (on Win2K and XP), while BCB uses its own heap manager, at least for small memory blocks. Don´t know it for sure for large memory blocks. But since the heap manager sources are distributed with BCB you may have a look at them and find it out. I usually forget about the source. Yes, I'll have a look at it. Quote: For VC the behavior may be changed by using _set_sb_treshold. Perhaps the borland heap manager may be reconfigured, too ? I'm not very familiar with VC. I didn't know that. As above, I'll look at the BCB source. Quote: Also i think you compiled a VC debug version, which fills all the allocated memory with a special byte sequence. I didn't know that either. I'll try it under release and see what happens. Thanks for your input. - Arnie Quote: Cheers Andre Back to top Arnie Mauer Guest Posted: Fri Jul 30, 2004 7:15 pm    Post subject: Re: Memory Allocation "Andre Kaufmann" <##andre_no_spam_.kaufmann_ (AT) t-online (DOT) de##> wrote in message news:410a8d49 (AT) newsgroups (DOT) borland.com... Quote: VC uses the Windows heap manager to allocate memory (on Win2K and XP), while BCB uses its own heap manager, at least for small memory blocks. Don´t know it for sure for large memory blocks. But since the heap manager sources are distributed with BCB you may have a look at them and find it out. For VC the behavior may be changed by using _set_sb_treshold. Perhaps the borland heap manager may be reconfigured, too ? Also i think you compiled a VC debug version, which fills all the allocated memory with a special byte sequence. Cheers Andre Well, you're absolutely correct. I rebuilt the VC++ version in release mode. The program allocated a contigous 2GB in a flash. Now, if only I could get rid of all those pesky AnsiStrings, TDateTimes and TADO* components, I could port to VC++ :-( Thanks again for your response, - Arnie Back to top Tamas Demjen Guest Posted: Sat Jul 31, 2004 12:31 am    Post subject: Re: Memory Allocation Quote: Thanks Tom. I presented a pretty simple-minded test program. Our problems usually occur with, eg, vector::push_back(), when the vector doubles in size. I suppose we could provide our own allocator and give it a shot. - Arnie It's not always applicable, but if you can use a deque, it allocates several smaller chunks, as opposed to a huge contiguous one. Reading from a deque is almost as fast as a vector, and pushing data to it is even faster, because it doesn't perform a copy of all items when the container has to grow. I have a different kind of problem, the memory fragmentation: void Test() { std::vector<unsigned char*> Guard; int Count = 1000; while(Count--) { //void* b1 = VirtualAlloc(NULL, 16000000, MEM_COMMIT, // PAGE_READWRITE); unsigned char* b1 = new unsigned char[16000000]; int cnt = 3; while(cnt--) { unsigned char* b3 = new unsigned char[150]; Guard.push_back(b3); } //VirtualFree(b1, 0, MEM_RELEASE); delete [] b1; } for(std::vector<unsigned char*>::iterator it = Guard.begin(); it != Guard.end(); ++it) delete [] *it; } This code has nothing wrong, it should work. It doesn't allocate a lot of memory, it has no leak, and its memory usage is orders of magnitude below the available RAM, and it still crashes. With VC++ it doesn't. My temporary (immediate) solution is to use VirtualAlloc, instead of new[]. You may say this loop does not reflect a real-world situation, but in some cases, it does. It actually happened to me, and it took me 2 days to figure it out. It typically happens when writing out big multipage TIFF or PDF files with large images. There's only one image at a time in the memory, which is deallocated right after saving, so the total memory consumption is minimal. However, the small and the large blocks are interleaved. In a real-world application, b1 is a bitmap image or TMemoryStream kind of thing, and b3 is a string, or vector, or other bookkeeping object. Exactly this interleaving of small and large blocks causes the fragmentation, at least I believe that's the cause of the problem, because I have over 600MB RAM when it crashes, and other compilers don't have this problem, only BCB. A 3rd party memory manager that replaces Borland's would probably help. A few alternative ideas to avoid the fragmentation problem: 1. Allocate enough space for all images before the loop, and reuse the allocated buffer repeatedly. It even increases the performance of the application. 2. If using contiguous memory is not absolutely essential, break it up into smaller segments (like a vector of smaller buffers, or a deque<char>). You can't do it with TMemoryStream, but we can write a much better stream implementation than that. Some 3rd party imaging libraries are written in C, and they require that you use a contigous memory (including the Win32 API itself). Tom Back to top pleasedont Guest Posted: Sat Jul 31, 2004 2:33 am    Post subject: Re: Memory Allocation "Arnie Mauer" <xxx (AT) nowhere (DOT) net> wrote: Quote: I compiled it with both BCB v5 and VC++ 8. Very interesting. BCB could allocate up to 1.1 GB and then die. It did this in an 'eye blink'. VC++ took time to allocate everything, but just kept going. See the BCB output below: Allocated 1100000000 bytes. bad alloc exception thrown Compiled from the command line with BCB6 your code stops at 1.9GB for me. Turn off Dynimic RTL in the linker options and it will stop at 1.9GB built in the IDE also. One of the RTL DLLS is loaded at a fixed base address which limits the address space available for heap. Something like that anyway. Back to top pleasedont Guest Posted: Sat Jul 31, 2004 2:58 am    Post subject: Re: Memory Allocation pleasedont <nospam (AT) please (DOT) invalid> wrote: Quote: One of the RTL DLLS is loaded at a fixed base address which limits the address space available for heap. Something like that anyway. I checked with the debugger cc3260.dll the main RTL loads at 0x32600000 which in the 2GB user space leaves about 0.8GB below it and 1.1GB above it. Back to top Micha Nelissen Guest Posted: Sat Jul 31, 2004 10:36 am    Post subject: Re: Memory Allocation Arnie Mauer wrote: Quote: Well, you're absolutely correct. I rebuilt the VC++ version in release mode. The program allocated a contigous 2GB in a flash. Now, Sorry to drop in, but how is continuous 2GB possible? For user space there is less than 2GB total available (0x10000 - 0x7FFFFFFF), the executable is loaded at 0x40000, so how can there be a 2GB block in there? Micha Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         BorlandTalk.com Forum Index -> C++ Builder (Non-Technical) All times are GMT Goto page 1, 2  Next Page 1 of 2   Jump to: Select a forum C++ Builder Newsgroups----------------C++ BuilderC++ Builder (ActiveX)C++ Builder (Announcements)C++ Builder (Command Line Tools)C++ Builder DatabasesC++ Builder Databases (ADO)C++ Builder Databases (Desktop)C++ Builder Databases (InterBase Express)C++ Builder Databases (Multi-Tier)C++ Builder Databases (SQL Servers)C++ Builder (Graphics)C++ Builder (IDE)C++ Builder (Internet)C++ Builder (Internet Socket)C++ Builder (Internet Web)C++ Builder (Jobs)C++ Builder (Language)C++ Builder (Language C++)C++ Builder (Compatibility with MFC)C++ Builder (Native API)C++ Builder (Non-Technical)C++ Builder (Students)C++ Builder (Third Party Tools)C++ Builder (Upgrade)C++ Builder (VCL)C++ Builder (VCL Components Usage)C++ Builder (VCL Components Development)C++ Builder (VCL Usage)C++ Builder (WinAPI)C++ Builder Installation Delphi Newsgroups----------------Delphi (General)Delphi ActiveX Controls UsageDelphi ActiveX Controls DevelopmentDelphi As400Delphi DatabasesDelphi Databases (ADO)Delphi Databases (dbExpress)Delphi Databases (Desktop)Delphi Databases (InterBase Express)Delphi Databases (Multi-Tier)Delphi Databases (SQL Servers)Delphi DeploymentDelphi GraphicsDelphi IDEDelphi InternationalizationDelphi InternetDelphi Internet Isapi-WebBrokerDelphi Internet WebSnapDelphi Internet WinsockDelphi JobsDelphi Language BASMDelphi Language ObjectPascalDelphi Model Driven Architecture (General)Delphi Model Driven Architecture (Third Party)Delphi MultimediaDelphi Native APIDelphi Non-TechnicalDelphi ObjectPascalDelphi OLE AutomationDelphi OO designDelphi OpenToolsAPIDelphi Reporting-ChartingDelphi RTLDelphi StudentsDelphi TeamSourceDelphi Thirdparty ToolsDelphi Thirdparty Tools (General)Delphi Thirdparty Tools (IntraWeb)Delphi Thirdparty Tools (RaveReports)Delphi UpgradeDelphi VCLDelphi VCL ComponentsDelphi VCL Components UsingDelphi VCL Components WritingDelphi WebServices SOAPDelphi WinAPIDelphi XMLDelphi Installationcomp.lang.pascal.delphi.components.misccomp.lang.pascal.delphi.components.usagecomp.lang.pascal.delphi.components.writingcomp.lang.pascal.delphi.databasescomp.lang.pascal.delphi.misccomp.sources.delphi JBuilder Newsgroups----------------JBuilderJBuilder AnnouncementsJBuilder Applet IssuesJBuilder As400JBuilder CompilerJBuilder CORBA/RMIJBuilder DatabasesJBuilder DB SwingJBuilder DebuggerJBuilder DeploymentJBuilder DocumentationJBuilder EnterpriseJBuilder HandheldJBuilder IDEJBuilder JavaJBuilder JavaAPIJBuilder Java LanguageJBuilder Java SwingJBuilder Java BeansJBuilder Java Beans UsageJBuilder Java Beans DevelopmentJBuilder JBCLJBuilder JDataStoreJBuilder JobsJBuilder LinuxJBuilder MacintoshJBuilder & MobileSetJBuilder MultiLingual AppsJBuilder Non-TechnicalJBuilder OpenToolsJBuilder Servlets (JSP)JBuilder StudentsJBuilder Team DevelopmentJBuilder Thirdparty ToolsJBuilder UpgradeJBuilder Installation InterBase Newsgroups----------------InterBaseInterBase Bug DiscussionInterBase GeneralInterBase IBConsoleInterBase OpenSourceInterBase ReplicationInterBase SQLInterBase ThirdpartyInterBase Installation Miscellaneous Borland Newsgroups----------------AppCenterAppServerEnteraVisiBroker  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.