BorlandTalk.com Borland discussion newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  db performance on insert          BorlandTalk.com Forum Index -> Delphi Databases (SQL Servers) View previous topic :: View next topic   Author Message Peter Hartlén Guest Posted: Fri Mar 17, 2006 10:03 am    Post subject: db performance on insert Hi! I've noticed some major perfomance issues with importing large quantaties of data from a textfile. Simple bulk-insert is not possible as the relation between different datatables (both existing and imported) must be checked during the import. We use simple adoqueries with prepared parameterized statements. The import file is located on the client machine. Would it be a great performance benefit to use stored proceedures with the import file located on the server? Thanks, Peter Back to top Viatcheslav V. Vassiliev Guest Posted: Fri Mar 17, 2006 11:03 am    Post subject: Re: db performance on insert Which DBMS are you using? //------------------------------------------ Regards, Vassiliev V. V. http://www.managed-vcl.com - using .Net objects in Delphi for Win32 + ADO.Net http://www.oledbdirect.com - The fastest way to access MS SQL Server, MS Jet (Access) and Interbase (through OLEDB) "Peter Hartlén" <ph.no (AT) sib (DOT) spam.se> ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ: news:441a7ee2 (AT) newsgroups (DOT) borland.com... Quote: Hi! I've noticed some major perfomance issues with importing large quantaties of data from a textfile. Simple bulk-insert is not possible as the relation between different datatables (both existing and imported) must be checked during the import. We use simple adoqueries with prepared parameterized statements. The import file is located on the client machine. Would it be a great performance benefit to use stored proceedures with the import file located on the server? Thanks, Peter Back to top Peter Hartlén Guest Posted: Fri Mar 17, 2006 6:03 pm    Post subject: Re: db performance on insert Microsoft SQL Server 2000, I have also noticed that enabling write cache on the disks greatly improves the import speed. What I have noticed is that without the write cache enabled, the server seems to almost idle while doing the import. Processor usage is low (<10%), RAM usage is constant, listening to the computer I can here the disks work hard when cache is used, while they are silent when cache is disabled. With Cache Without Cache CPU ~50% <10% RAM Stable Stable Disk Busy Silent / Peter "Viatcheslav V. Vassiliev" <support (AT) oledbdirect (DOT) com> skrev i meddelandet news:441a8753$1 (AT) newsgroups (DOT) borland.com... Quote: Which DBMS are you using? //------------------------------------------ Regards, Vassiliev V. V. http://www.managed-vcl.com - using .Net objects in Delphi for Win32 + ADO.Net http://www.oledbdirect.com - The fastest way to access MS SQL Server, MS Jet (Access) and Interbase (through OLEDB) "Peter Hartlén" <ph.no (AT) sib (DOT) spam.se> ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ: news:441a7ee2 (AT) newsgroups (DOT) borland.com... Hi! I've noticed some major perfomance issues with importing large quantaties of data from a textfile. Simple bulk-insert is not possible as the relation between different datatables (both existing and imported) must be checked during the import. We use simple adoqueries with prepared parameterized statements. The import file is located on the client machine. Would it be a great performance benefit to use stored proceedures with the import file located on the server? Thanks, Peter Back to top Oliver Townshend Guest Posted: Sat Mar 18, 2006 12:03 am    Post subject: Re: db performance on insert Quote: Microsoft SQL Server 2000, I have also noticed that enabling write cache on the disks greatly improves the import speed. The problem being that with write caching if you have a crash you may lose parts of your transactions. Oliver Townshend Back to top Viatcheslav V. Vassiliev Guest Posted: Sat Mar 18, 2006 10:03 am    Post subject: Re: db performance on insert MS SQL Server 2000 has interface IRowsetFastLoad that is used in bulk inserts and can be used from program. It is implemented in OLEDB provider, but ADO can not use this interface. You may try OLEDB Direct (http://www.oledbdirect.com) and example SQL Server\1000000 rows for example that inserts 1M rows into simple table in 10 sec. //------------------------------------------ Regards, Vassiliev V. V. http://www.managed-vcl.com - using .Net objects in Delphi for Win32 + ADO.Net http://www.oledbdirect.com - The fastest way to access MS SQL Server, MS Jet (Access) and Interbase (through OLEDB) "Peter Hartlén" <ph.no (AT) sib (DOT) spam.se> ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ: news:441ae951 (AT) newsgroups (DOT) borland.com... Quote: Microsoft SQL Server 2000, I have also noticed that enabling write cache on the disks greatly improves the import speed. What I have noticed is that without the write cache enabled, the server seems to almost idle while doing the import. Processor usage is low (<10%), RAM usage is constant, listening to the computer I can here the disks work hard when cache is used, while they are silent when cache is disabled. With Cache Without Cache CPU ~50% <10% RAM Stable Stable Disk Busy Silent / Peter "Viatcheslav V. Vassiliev" <support (AT) oledbdirect (DOT) com> skrev i meddelandet news:441a8753$1 (AT) newsgroups (DOT) borland.com... Which DBMS are you using? //------------------------------------------ Regards, Vassiliev V. V. http://www.managed-vcl.com - using .Net objects in Delphi for Win32 + ADO.Net http://www.oledbdirect.com - The fastest way to access MS SQL Server, MS Jet (Access) and Interbase (through OLEDB) "Peter Hartlén" <ph.no (AT) sib (DOT) spam.se> ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ ÓÌÅÄÕÀÝÅÅ: news:441a7ee2 (AT) newsgroups (DOT) borland.com... Hi! I've noticed some major perfomance issues with importing large quantaties of data from a textfile. Simple bulk-insert is not possible as the relation between different datatables (both existing and imported) must be checked during the import. We use simple adoqueries with prepared parameterized statements. The import file is located on the client machine. Would it be a great performance benefit to use stored proceedures with the import file located on the server? Thanks, Peter Back to top Arthur Hoornweg Guest Posted: Sat Mar 18, 2006 5:03 pm    Post subject: Re: db performance on insert Peter Hartlén wrote: Quote: Microsoft SQL Server 2000, I have also noticed that enabling write cache on the disks greatly improves the import speed. I suspect you're not wrapping your routine inside a large transaction; so every single statement is committed individually. Hence the hard disk activity. It is considered bad practice because an error will leave you with incomplete data in the database. Try something like this: MyAdoConnection.Begintrans; TRY CallMyImportProcedure; // No error? Then commit all changes at once. MyAdoConnection.CommitTrans; Except // an error happened? Then undo all changes. MyAdoConnection.RollbackTrans; Showmessage('something went wrong - transaction cancelled'); End; -- Arthur Hoornweg (In order to reply per e-mail, please just remove the ".net" from my e-mail address. Leave the rest of the address intact including the "antispam" part. I had to take this measure to counteract unsollicited mail.) Back to top Peter Hartlén Guest Posted: Mon Mar 20, 2006 1:03 pm    Post subject: Re: db performance on insert "Arthur Hoornweg" <antispam.hoornweg (AT) casema (DOT) nl.net> skrev i meddelandet news:441c2e7f$1 (AT) newsgroups (DOT) borland.com... Quote: I suspect you're not wrapping your routine inside a large transaction; so every single statement is committed individually. Hence the hard disk activity. It is considered bad practice because an error will leave you with incomplete data in the database. Hi Arthur! This is exactly what I am doing, and I don't think that would have any significant impact on performance either. Regards, Peter Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         BorlandTalk.com Forum Index -> Delphi Databases (SQL Servers) All times are GMT Page 1 of 1   Jump to: Select a forum C++ Builder Newsgroups----------------C++ BuilderC++ Builder (ActiveX)C++ Builder (Announcements)C++ Builder (Command Line Tools)C++ Builder DatabasesC++ Builder Databases (ADO)C++ Builder Databases (Desktop)C++ Builder Databases (InterBase Express)C++ Builder Databases (Multi-Tier)C++ Builder Databases (SQL Servers)C++ Builder (Graphics)C++ Builder (IDE)C++ Builder (Internet)C++ Builder (Internet Socket)C++ Builder (Internet Web)C++ Builder (Jobs)C++ Builder (Language)C++ Builder (Language C++)C++ Builder (Compatibility with MFC)C++ Builder (Native API)C++ Builder (Non-Technical)C++ Builder (Students)C++ Builder (Third Party Tools)C++ Builder (Upgrade)C++ Builder (VCL)C++ Builder (VCL Components Usage)C++ Builder (VCL Components Development)C++ Builder (VCL Usage)C++ Builder (WinAPI)C++ Builder Installation Delphi Newsgroups----------------Delphi (General)Delphi ActiveX Controls UsageDelphi ActiveX Controls DevelopmentDelphi As400Delphi DatabasesDelphi Databases (ADO)Delphi Databases (dbExpress)Delphi Databases (Desktop)Delphi Databases (InterBase Express)Delphi Databases (Multi-Tier)Delphi Databases (SQL Servers)Delphi DeploymentDelphi GraphicsDelphi IDEDelphi InternationalizationDelphi InternetDelphi Internet Isapi-WebBrokerDelphi Internet WebSnapDelphi Internet WinsockDelphi JobsDelphi Language BASMDelphi Language ObjectPascalDelphi Model Driven Architecture (General)Delphi Model Driven Architecture (Third Party)Delphi MultimediaDelphi Native APIDelphi Non-TechnicalDelphi ObjectPascalDelphi OLE AutomationDelphi OO designDelphi OpenToolsAPIDelphi Reporting-ChartingDelphi RTLDelphi StudentsDelphi TeamSourceDelphi Thirdparty ToolsDelphi Thirdparty Tools (General)Delphi Thirdparty Tools (IntraWeb)Delphi Thirdparty Tools (RaveReports)Delphi UpgradeDelphi VCLDelphi VCL ComponentsDelphi VCL Components UsingDelphi VCL Components WritingDelphi WebServices SOAPDelphi WinAPIDelphi XMLDelphi Installationcomp.lang.pascal.delphi.components.misccomp.lang.pascal.delphi.components.usagecomp.lang.pascal.delphi.components.writingcomp.lang.pascal.delphi.databasescomp.lang.pascal.delphi.misccomp.sources.delphi JBuilder Newsgroups----------------JBuilderJBuilder AnnouncementsJBuilder Applet IssuesJBuilder As400JBuilder CompilerJBuilder CORBA/RMIJBuilder DatabasesJBuilder DB SwingJBuilder DebuggerJBuilder DeploymentJBuilder DocumentationJBuilder EnterpriseJBuilder HandheldJBuilder IDEJBuilder JavaJBuilder JavaAPIJBuilder Java LanguageJBuilder Java SwingJBuilder Java BeansJBuilder Java Beans UsageJBuilder Java Beans DevelopmentJBuilder JBCLJBuilder JDataStoreJBuilder JobsJBuilder LinuxJBuilder MacintoshJBuilder & MobileSetJBuilder MultiLingual AppsJBuilder Non-TechnicalJBuilder OpenToolsJBuilder Servlets (JSP)JBuilder StudentsJBuilder Team DevelopmentJBuilder Thirdparty ToolsJBuilder UpgradeJBuilder Installation InterBase Newsgroups----------------InterBaseInterBase Bug DiscussionInterBase GeneralInterBase IBConsoleInterBase OpenSourceInterBase ReplicationInterBase SQLInterBase ThirdpartyInterBase Installation Miscellaneous Borland Newsgroups----------------AppCenterAppServerEnteraVisiBroker  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.