BorlandTalk.com Borland discussion newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  HTTP Page Request Preview Utility          BorlandTalk.com Forum Index -> Delphi Internet Winsock View previous topic :: View next topic   Author Message Johnnie Norsworthy Guest Posted: Mon Mar 21, 2005 8:12 pm    Post subject: HTTP Page Request Preview Utility I have automated IE to login and download pages requested from a web site. As is was very important that I got the process working again quickly, I had to go the browser-based route, due to some strict and as yet unknown authentication the web site performs expecting a browser to be used. Now that I have IE working (actually TEmbeddedWB and TIEDownload), I would like to convert my page requests to Indy, but I still cannot seem to determine all of the criteria the web site uses for authentication. I have copied all of the headers and cookies from IE to an IdHTTP and Cookie Manager, and even the UserAgent, but the requests still do not work at all with TIdHTTP. I don't know everything that is sent by IE or any addtional header that may need to be set, so I was hoping there was some utility program I could install that would intercept Internet Explorer when it requested a page and would show me absolutely everything that was being sent during the request in the exact format sent so I could again try to use TIdHTTP for my page requests. Thanks for any suggestions. -Johnnie Back to top Remy Lebeau (TeamB) Guest Posted: Mon Mar 21, 2005 8:20 pm    Post subject: Re: HTTP Page Request Preview Utility "Johnnie Norsworthy" <jln206 (AT) verizon (DOT) net> wrote Quote: Now that I have IE working (actually TEmbeddedWB and TIEDownload), I would like to convert my page requests to Indy, but I still cannot seem to determine all of the criteria the web site uses for authentication. Use a packet sniffer to view everything that the browser receives and transmits. Then you can mimic that with Indy as needed. Quote: I have copied all of the headers and cookies from IE to an IdHTTP and Cookie Manager, and even the UserAgent, but the requests still do not work at all with TIdHTTP. Did you set up any actual authentication settings in TIdHTTP? Quote: I don't know everything that is sent by IE, or any addtional header that may need to be set You will if you use a packet sniffer. Quote: so I was hoping there was some utility program I could install that would intercept Internet Explorer when it requested a page and would show me absolutely everything that was being sent during the request in the exact format sent so I could again try to use TIdHTTP for my page requests. Yes - a packet sniffer. Gambit Back to top Johnnie Norsworthy Guest Posted: Mon Mar 21, 2005 9:05 pm    Post subject: Re: HTTP Page Request Preview Utility "Remy Lebeau (TeamB)" <no.spam (AT) no (DOT) spam.com> wrote Quote: "Johnnie Norsworthy" <jln206 (AT) verizon (DOT) net> wrote in message news:423f2aab$1 (AT) newsgroups (DOT) borland.com... Now that I have IE working (actually TEmbeddedWB and TIEDownload), I would like to convert my page requests to Indy, but I still cannot seem to determine all of the criteria the web site uses for authentication. Use a packet sniffer to view everything that the browser receives and transmits. Then you can mimic that with Indy as needed. Any recommendations for a packet sniffer? I could Google, but I value your (Remy's) recommendation as you seem to be the main person that helps me in this group. (Thanks!) Quote: I have copied all of the headers and cookies from IE to an IdHTTP and Cookie Manager, and even the UserAgent, but the requests still do not work at all with TIdHTTP. Did you set up any actual authentication settings in TIdHTTP? No, I turned off basic authentication when the site started using ASP session cookies for authentication. Thank you, as always, -Johnnie Back to top eshipman Guest Posted: Mon Mar 21, 2005 10:19 pm    Post subject: Re: HTTP Page Request Preview Utility In article <423f3724$1 (AT) newsgroups (DOT) borland.com>, [email]jln206 (AT) verizon (DOT) net[/email] says... Quote: "Remy Lebeau (TeamB)" <no.spam (AT) no (DOT) spam.com> wrote in message news:423f2c99$1 (AT) newsgroups (DOT) borland.com... "Johnnie Norsworthy" <jln206 (AT) verizon (DOT) net> wrote in message news:423f2aab$1 (AT) newsgroups (DOT) borland.com... Now that I have IE working (actually TEmbeddedWB and TIEDownload), I would like to convert my page requests to Indy, but I still cannot seem to determine all of the criteria the web site uses for authentication. Use a packet sniffer to view everything that the browser receives and transmits. Then you can mimic that with Indy as needed. Any recommendations for a packet sniffer? I could Google, but I value your (Remy's) recommendation as you seem to be the main person that helps me in this group. (Thanks!) Ethereal : http://www.ethereal.com/ Quote: I have copied all of the headers and cookies from IE to an IdHTTP and Cookie Manager, and even the UserAgent, but the requests still do not work at all with TIdHTTP. Did you set up any actual authentication settings in TIdHTTP? No, I turned off basic authentication when the site started using ASP session cookies for authentication. Well...there could be your problem. I have never been able to get ASP sessions working correctly using Indy. Back to top Remy Lebeau (TeamB) Guest Posted: Mon Mar 21, 2005 10:22 pm    Post subject: Re: HTTP Page Request Preview Utility "Johnnie Norsworthy" <jln206 (AT) verizon (DOT) net> wrote Quote: Any recommendations for a packet sniffer? Ethereal (http://www.ethereal.com) Gambit Back to top Remy Lebeau (TeamB) Guest Posted: Mon Mar 21, 2005 10:23 pm    Post subject: Re: HTTP Page Request Preview Utility "eshipman" <mr_delphi_developer@yahoo!!!.com> wrote Quote: Well...there could be your problem. I have never been able to get ASP sessions working correctly using Indy. Why not? Please be more specific. Gambit Back to top Johnnie Norsworthy Guest Posted: Mon Mar 21, 2005 10:30 pm    Post subject: Re: HTTP Page Request Preview Utility "Remy Lebeau (TeamB)" <no.spam (AT) no (DOT) spam.com> wrote Quote: "Johnnie Norsworthy" <jln206 (AT) verizon (DOT) net> wrote in message news:423f3724$1 (AT) newsgroups (DOT) borland.com... Any recommendations for a packet sniffer? Ethereal (http://www.ethereal.com) Already downloaded and installed. It works great and I have logged all of the requests from TIEDownload precisely. I also see a bunch more traffic I am going to have to check into. I am going to play around with substituting all of the header information I found from TIEDownload and see if I can get TIDHTTP to work again. I get the feeling I will. Then maybe I can stop asking so many questions here. Thanks for all the invaluable help. -Johnnie Back to top Johnnie Norsworthy Guest Posted: Tue Mar 22, 2005 7:32 am    Post subject: Re: HTTP Page Request Preview Utility "Remy Lebeau (TeamB)" <no.spam (AT) no (DOT) spam.com> wrote Quote: "eshipman" <mr_delphi_developer@yahoo!!!.com> wrote in message news:MPG.1ca9048acc7cb51b98993e (AT) forums (DOT) borland.com... Well...there could be your problem. I have never been able to get ASP sessions working correctly using Indy. Why not? Please be more specific. I got it working today. Perfectly. -Johnnie Back to top Remy Lebeau (TeamB) Guest Posted: Tue Mar 22, 2005 9:30 pm    Post subject: Re: HTTP Page Request Preview Utility "Johnnie Norsworthy" <jln206 (AT) verizon (DOT) net> wrote Quote: I got it working today. Perfectly. That is fine and all, but you never said what the problem was to begin with, or how you got it working. Other people might benefit from that information. Gambit Back to top Johnnie Norsworthy Guest Posted: Tue Mar 22, 2005 9:55 pm    Post subject: Re: HTTP Page Request Preview Utility "Remy Lebeau (TeamB)" <no.spam (AT) no (DOT) spam.com> wrote Quote: "Johnnie Norsworthy" <jln206 (AT) verizon (DOT) net> wrote in message news:423fca27$1 (AT) newsgroups (DOT) borland.com... I got it working today. Perfectly. That is fine and all, but you never said what the problem was to begin with, or how you got it working. Other people might benefit from that information. The problem was that the web site was partly basing their authentication on some other browser produced header that was specific to IE but was not in TIdHTTP. By capturing the actual headers and cookies sent by IE and duplicating them to TIdHTTP it works. The Ethereal packet sniffer exposed the exact header values for me to duplicate them. Thank you again for your help. -Johnnie Back to top eshipman Guest Posted: Tue Mar 22, 2005 10:09 pm    Post subject: Re: HTTP Page Request Preview Utility In article <42409442$1 (AT) newsgroups (DOT) borland.com>, [email]jln206 (AT) verizon (DOT) net[/email] says... Quote: "Remy Lebeau (TeamB)" <no.spam (AT) no (DOT) spam.com> wrote in message news:42408e6f$1 (AT) newsgroups (DOT) borland.com... "Johnnie Norsworthy" <jln206 (AT) verizon (DOT) net> wrote in message news:423fca27$1 (AT) newsgroups (DOT) borland.com... I got it working today. Perfectly. That is fine and all, but you never said what the problem was to begin with, or how you got it working. Other people might benefit from that information. The problem was that the web site was partly basing their authentication on some other browser produced header that was specific to IE but was not in TIdHTTP. By capturing the actual headers and cookies sent by IE and duplicating them to TIdHTTP it works. The Ethereal packet sniffer exposed the exact header values for me to duplicate them. Mind telling us what they were? Back to top Johnnie Norsworthy Guest Posted: Tue Mar 22, 2005 10:19 pm    Post subject: Re: HTTP Page Request Preview Utility "eshipman" <mr_delphi_developer@yahoo!!!.com> wrote Quote: Mind telling us what they were? Sorry, I didn't think the exact changes would interest anyone. I did not attempt to find the single header value that made a difference, instead I just changed them all to mimic IE. In TIdDHTTP, I set: Request.Accept=*/* Request.CacheControl=no-cache Request.Connection=Keep-Alive Request.ContentType=application/x-www-form-urlencodedAccept-Encoding: gzip, deflate Request.UserAgent=Mozilla/4.0 (compatible; MSIE 6.0; Win32) Request.Host=(web site name) -Johnnie Back to top eshipman Guest Posted: Wed Mar 23, 2005 3:19 pm    Post subject: Re: HTTP Page Request Preview Utility In article <424099d9$1 (AT) newsgroups (DOT) borland.com>, [email]jln206 (AT) verizon (DOT) net[/email] says... Quote: "eshipman" <mr_delphi_developer@yahoo!!!.com> wrote in message news:MPG.1caa53a442a977f398993f (AT) forums (DOT) borland.com... Mind telling us what they were? Sorry, I didn't think the exact changes would interest anyone. I did not attempt to find the single header value that made a difference, instead I just changed them all to mimic IE. In TIdDHTTP, I set: Request.Accept=*/* Request.CacheControl=no-cache Request.Connection=Keep-Alive Request.ContentType=application/x-www-form-urlencodedAccept-Encoding: gzip, deflate Request.UserAgent=Mozilla/4.0 (compatible; MSIE 6.0; Win32) Request.Host=(web site name) My bet is that it was the Request.Host. Most sites check the referrer as that is most readily avail but some use the request.host because most developers don't think to set it and it would be easy to overlook. Back to top Johnnie Norsworthy Guest Posted: Wed Mar 23, 2005 4:07 pm    Post subject: Re: HTTP Page Request Preview Utility "eshipman" <mr_delphi_developer@yahoo!!!.com> wrote Quote: My bet is that it was the Request.Host. Most sites check the referrer as that is most readily avail but some use the request.host because most developers don't think to set it and it would be easy to overlook. Not in this case. That was one I tried before setting the other headers. It didn't work. -Johnnie Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         BorlandTalk.com Forum Index -> Delphi Internet Winsock All times are GMT Page 1 of 1   Jump to: Select a forum C++ Builder Newsgroups----------------C++ BuilderC++ Builder (ActiveX)C++ Builder (Announcements)C++ Builder (Command Line Tools)C++ Builder DatabasesC++ Builder Databases (ADO)C++ Builder Databases (Desktop)C++ Builder Databases (InterBase Express)C++ Builder Databases (Multi-Tier)C++ Builder Databases (SQL Servers)C++ Builder (Graphics)C++ Builder (IDE)C++ Builder (Internet)C++ Builder (Internet Socket)C++ Builder (Internet Web)C++ Builder (Jobs)C++ Builder (Language)C++ Builder (Language C++)C++ Builder (Compatibility with MFC)C++ Builder (Native API)C++ Builder (Non-Technical)C++ Builder (Students)C++ Builder (Third Party Tools)C++ Builder (Upgrade)C++ Builder (VCL)C++ Builder (VCL Components Usage)C++ Builder (VCL Components Development)C++ Builder (VCL Usage)C++ Builder (WinAPI)C++ Builder Installation Delphi Newsgroups----------------Delphi (General)Delphi ActiveX Controls UsageDelphi ActiveX Controls DevelopmentDelphi As400Delphi DatabasesDelphi Databases (ADO)Delphi Databases (dbExpress)Delphi Databases (Desktop)Delphi Databases (InterBase Express)Delphi Databases (Multi-Tier)Delphi Databases (SQL Servers)Delphi DeploymentDelphi GraphicsDelphi IDEDelphi InternationalizationDelphi InternetDelphi Internet Isapi-WebBrokerDelphi Internet WebSnapDelphi Internet WinsockDelphi JobsDelphi Language BASMDelphi Language ObjectPascalDelphi Model Driven Architecture (General)Delphi Model Driven Architecture (Third Party)Delphi MultimediaDelphi Native APIDelphi Non-TechnicalDelphi ObjectPascalDelphi OLE AutomationDelphi OO designDelphi OpenToolsAPIDelphi Reporting-ChartingDelphi RTLDelphi StudentsDelphi TeamSourceDelphi Thirdparty ToolsDelphi Thirdparty Tools (General)Delphi Thirdparty Tools (IntraWeb)Delphi Thirdparty Tools (RaveReports)Delphi UpgradeDelphi VCLDelphi VCL ComponentsDelphi VCL Components UsingDelphi VCL Components WritingDelphi WebServices SOAPDelphi WinAPIDelphi XMLDelphi Installationcomp.lang.pascal.delphi.components.misccomp.lang.pascal.delphi.components.usagecomp.lang.pascal.delphi.components.writingcomp.lang.pascal.delphi.databasescomp.lang.pascal.delphi.misccomp.sources.delphi JBuilder Newsgroups----------------JBuilderJBuilder AnnouncementsJBuilder Applet IssuesJBuilder As400JBuilder CompilerJBuilder CORBA/RMIJBuilder DatabasesJBuilder DB SwingJBuilder DebuggerJBuilder DeploymentJBuilder DocumentationJBuilder EnterpriseJBuilder HandheldJBuilder IDEJBuilder JavaJBuilder JavaAPIJBuilder Java LanguageJBuilder Java SwingJBuilder Java BeansJBuilder Java Beans UsageJBuilder Java Beans DevelopmentJBuilder JBCLJBuilder JDataStoreJBuilder JobsJBuilder LinuxJBuilder MacintoshJBuilder & MobileSetJBuilder MultiLingual AppsJBuilder Non-TechnicalJBuilder OpenToolsJBuilder Servlets (JSP)JBuilder StudentsJBuilder Team DevelopmentJBuilder Thirdparty ToolsJBuilder UpgradeJBuilder Installation InterBase Newsgroups----------------InterBaseInterBase Bug DiscussionInterBase GeneralInterBase IBConsoleInterBase OpenSourceInterBase ReplicationInterBase SQLInterBase ThirdpartyInterBase Installation Miscellaneous Borland Newsgroups----------------AppCenterAppServerEnteraVisiBroker  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.