 |
BorlandTalk.com
Borland discussion newsgroups
|
| View previous topic :: View next topic |
| Author |
Message |
Mike Both
Guest
|
 Posted: Tue Feb 14, 2006 2:03 am Post subject: Socket count increasing with TIdFtp usage? |
 |
|
Hi all
I have a service written in Delphi 6 that uses the Indy TIdFtp control
that connects to a remote server (via our firewall), downloads a html
file, then parses it and builds and executes an insert query for an
Oracle server. The service runs 24/7 on one of our company servers, but
after 2-3 months fails to execute the ftp connect / get / disconnect
cycle. Using 'netstat -s' on the server (now running the 'broken'
service, I find an 'active opens' count of more than 200,000. I also
notice that running a ftp connect / disconnect cycle adds one to
the 'active opens' count. However, the 'active opens' count also
increases by one every minute or so even WITHOUT the service running
(when it is correctly making the ftp transfer).
So, some questions:
- does the large 'active opens' count indicated that the failure is
related to a lack of available sockets? Note - once the service stops
making ftp transfers correctly ftp doesn't work from the command line,
nor from a 3-rd party program. Rebooting the machine is necessary to get
it all working again (obviously not ideal on a server)
- should the 'active opens' count decreases again after a connect /
disconnect cycle with the TIdFtp component?
- what the hell does the 'active opens' figure given by netstat mean
anyway? :^)
As an aside - the service is stopped and restarted every 24 hours at
midnight; the previous developer found the service was leaking memory
somewhere but couldn't narrow down precisely where, so wrote another
service to stop and restart the original service as a (dodgy IMHO) 'fix'.
TIA for any help....
Mike
--- posted by geoForum on http://delphi.newswhat.com |
|
| Back to top |
|
 |
Remy Lebeau (TeamB)
Guest
|
| Posted: Tue Feb 14, 2006 8:03 am Post subject: Re: Socket count increasing with TIdFtp usage? |
|
|
"Mike Both" <both.michaelNOSPAM (AT) etsa (DOT) com.au> wrote in message
news:43f131da$1 (AT) newsgroups (DOT) borland.com...
| Quote: | I have a service written in Delphi 6 that uses the Indy TIdFtp control
|
Which version of Indy specifically?
| Quote: | However, the 'active opens' count also increases by one every minute
or so even WITHOUT the service running
|
"Active Opens" is the running total of overall sockets that have been opened
since the machine was booted. It is not the current number of sockets that
are actually running at that moment in time. To find out how many sockets
are atually running, use the "-a" command-line parameter for netstat, rather
than the "-s" parameter.
| Quote: | once the service stops making ftp transfers correctly ftp doesn't work
from the command line, nor from a 3-rd party program.
|
Sockets are a global resource. Once the system runs out of them, no program
will be able to use them until the existing ones are closed.
| Quote: | - should the 'active opens' count decreases again after a
connect / disconnect cycle with the TIdFtp component?
|
No.
| Quote: | - what the hell does the 'active opens' figure given by netstat mean
anyway?
|
Exactly what is says - the number of sockets that have been actively opened
on the machine.
Gambit |
|
| Back to top |
|
|
Mike Both
Guest
|
| Posted: Wed Feb 15, 2006 12:03 am Post subject: Re: Socket count increasing with TIdFtp usage? |
|
|
Thanks for the response!
| Quote: | I have a service written in Delphi 6 that uses the Indy TIdFtp control
Which version of Indy specifically?
|
10.0.52
| Quote: | Sockets are a global resource. Once the system runs out of them, no
program
will be able to use them until the existing ones are closed.
|
Ok - well I have confirmed the sockets end up closing (i.e. so they are
no longer active using netstat) when the service is running, however
there is still the issue of WHY ftp fails after a few months. I'm
assuming, then, it can have nothng to do with the 'active opens' count
being over 200,000? I'm not really familiar with any ftp/internet/socket
stuff, and am coming to a dead end with this problem. Hmmm ...
--- posted by geoForum on http://delphi.newswhat.com |
|
| Back to top |
|
|
Mike Both
Guest
|
| Posted: Wed Feb 15, 2006 1:03 am Post subject: Re: Socket count increasing with TIdFtp usage? |
|
|
| Quote: |
"Mike Both" <both.michaelNOSPAM (AT) etsa (DOT) com.au> wrote in message
news:43f26356$1 (AT) newsgroups (DOT) borland.com...
10.0.52
The latest snapshot build is 10.1.5. You should consider upgrading.
|
Will do, the 'problem' being the only way I can check the problem is
fixed is running the service until it 'dies' again.
| Quote: | What EXACTLY is happening? Just saying it fails is not very descriptive.
Are there any exceptions being thrown?
|
Yes - apologies for that, but the Previous Programmer didn't do a good
job with exception handler wrapping or logging, and as a result the first
exception that is thrown is when the file parser dies after finding there
is nothing to parse! :^( However, examination of what little log info
there is and tracing the code leads back to the ftp connection. Also, a
test ftp app I wrote (using same component, same ftp settings as used in
production) fails in the same way.
Here's the appropriate code from the test app:
procedure TfrmMainForm.btnConnectClick(Sender: TObject);
var
sMsg: string;
begin
mmoActivity.Clear;
// update proxy settings and report
ftpConnection.ProxySettings.Host := lblProxyName.Text;
ftpConnection.ProxySettings.Port := strtoint(lblProxyPort.Text);
ftpConnection.ProxySettings.UserName := lblProxyLoginUser.Text;
ftpConnection.ProxySettings.Password := lblProxyPassword.Text;
sMsg := 'Using proxy settings host=' + ftpConnection.ProxySettings.Host
+
', port=' + inttostr(ftpConnection.ProxySettings.Port) +
', user=' + ftpConnection.ProxySettings.UserName +
', password=' + ftpConnection.ProxySettings.Password;
mmoActivity.Lines.Add(sMsg);
// update ftp settings and report
ftpConnection.Host := lblFTPHostName.Text;
ftpConnection.Username := lblFTPUserName.Text;
ftpConnection.Password := lblFTPPassword.Text;
sMsg := 'Using ftp settings host=' + ftpConnection.Host +
', user=' + ftpConnection.Username +
', password=' + ftpConnection.Password;
mmoActivity.Lines.Add(sMsg);
// connect!
ftpConnection.Connect;
if (ftpConnection.Connected) then
mmoActivity.Lines.Add('connected!')
else
mmoActivity.Lines.Add('error connecting');
UpdateButtonStates;
end;
This is the series of messages when the 'connect' button is selected, and
the test app is running on the 'ftp broken' machine:
-----------------------
Using proxy settings host=proxygw, port=21, user=svc-app-fdiloader,
password=xxx
Using ftp settings host=134.178.63.xxx, user=anonymous,
password=blah (AT) blah (DOT) com
STATUS:Resolving hostname proxygw.
STATUS:Connecting to 159.13.99.xx.
STATUS:Connected.
STATUS:Disconnecting.
STATUS - disconnected
STATUS:Disconnected.
------------------------
I have status reporting functions like the following for each event:
procedure TfrmMainForm.ftpConnectionStatus(ASender: TObject;
const AStatus: TIdStatus; const AStatusText: String);
begin
mmoActivity.Lines.Add('STATUS:' + AStatusText)
end;
For a 'good' connect the status messages are:
--------------------------
Using proxy settings host=proxygw, port=21, user=svc-app-fdiloader,
password=xxxxxxxx
Using ftp settings host=134.178.63.xxx, user=anonymous,
password=blah (AT) blah (DOT) com
STATUS:Resolving hostname proxygw.
STATUS:Connecting to 159.13.99.xx.
STATUS:Connected.
STATUS:220 Blue Coat FTP Service
User anonymous (AT) 134 (DOT) 178.63.xxx svc-app-fdiloader
PASS xxxxxxxxxxxx
ACCT xxxxxxxxxxxx
STATUS:230 Login successful.
System is:
STATUS:Connection established
connected!
--------------------
Finally, I have a custom ftp connect procedure that was necessary to get
the app working through our firewall, but I don't think this is involved.
| Quote: | You did not say how often you are opening sockets in your service, or how
many sockets are running outside of your service. But 200,000 is not
uncommon for a machine that is running constantly for over a month. On
my
machine, my statistics show 4246 "active opens" just for today. Multiply
that by 47 days and you reach approx 200,000.
|
I'm unsure about the details of the server load, but I do know it's
running other programs (probably nothing that does ftp though). The
service runs every 10 minutes.
Thanks
Mike
--- posted by geoForum on http://delphi.newswhat.com |
|
| Back to top |
|
|
Remy Lebeau (TeamB)
Guest
|
| Posted: Wed Feb 15, 2006 1:03 am Post subject: Re: Socket count increasing with TIdFtp usage? |
|
|
"Mike Both" <both.michaelNOSPAM (AT) etsa (DOT) com.au> wrote in message
news:43f26356$1 (AT) newsgroups (DOT) borland.com...
The latest snapshot build is 10.1.5. You should consider upgrading.
| Quote: | however there is still the issue of WHY ftp fails after a few months.
|
What EXACTLY is happening? Just saying it fails is not very descriptive.
Are there any exceptions being thrown?
| Quote: | I'm assuming, then, it can have nothng to do with the 'active
opens' count being over 200,000?
|
You did not say how often you are opening sockets in your service, or how
many sockets are running outside of your service. But 200,000 is not
uncommon for a machine that is running constantly for over a month. On my
machine, my statistics show 4246 "active opens" just for today. Multiply
that by 47 days and you reach approx 200,000.
Gambit |
|
| Back to top |
|
|
Remy Lebeau (TeamB)
Guest
|
| Posted: Wed Feb 15, 2006 9:03 pm Post subject: Re: Socket count increasing with TIdFtp usage? |
|
|
"Mike Both" <both.michaelNOSPAM (AT) etsa (DOT) com.au> wrote in message
news:43f27b31$1 (AT) newsgroups (DOT) borland.com...
| Quote: | the first exception that is thrown is when the file parser dies
after finding there is nothing to parse! :^(
|
What is the EXACT exception type and message? TIdFTP's directory parsing
system is very extensive and well-tested. If there is a problem with it,
then please provide specific details, traffic logs, etc so it can be fixed.
| Quote: | However, examination of what little log info there is and tracing
the code leads back to the ftp connection.
|
Again, please provide SPECIFIC details. What exactly did you trace the
problem to?
| Quote: | Also, a test ftp app I wrote (using same component, same ftp
settings as used in production) fails in the same way.
|
Fails in WHAT way exactly?
| Quote: | ftpConnection.Connect;
if (ftpConnection.Connected) then
|
Connect() throws an exception if it fails to establish a connection:
try
ftpConnection.Connect;
mmoActivity.Lines.Add('connected!')
except
on E: Exception do mmoActivity.Lines.Add('error connecting: ' +
E.Message);
end;
Gambit |
|
| Back to top |
|
|
Mike Both
Guest
|
| Posted: Thu Feb 16, 2006 4:03 am Post subject: Re: Socket count increasing with TIdFtp usage? |
|
|
| Quote: | the first exception that is thrown is when the file parser dies
after finding there is nothing to parse! :^(
What is the EXACT exception type and message? TIdFTP's directory parsing
system is very extensive and well-tested. If there is a problem with it,
then please provide specific details, traffic logs, etc so it can be
fixed. |
Sorry for the lack of clarity - I'm not referring to TIdFTP, I'm
referring to the code I'm working with. Once the file is ftped, it is
then parsed by another procedure. An exception is thrown within this
procedure as there is no file to parse. The (poor) way the code is
written currently means there is no testing of whether the file has been
successfully ftped in or not. The assumption is that if the ftp control
connects, and doesn't throw and exception, then the file has been
retrieved.
| Quote: | However, examination of what little log info there is and tracing
the code leads back to the ftp connection.
Fails in WHAT way exactly?
|
Well - there is no exception thrown by TIdFTP, it just connects, then the
connection immediately terminates (as can be seen by the log in the
previous message, included here again):
--------------
Using proxy settings host=proxygw, port=21, user=svc-app-fdiloader,
password=xxx
Using ftp settings host=134.178.63.xxx, user=anonymous,
password=blah (AT) blah (DOT) com
STATUS:Resolving hostname proxygw.
STATUS:Connecting to 159.13.99.xx.
STATUS:Connected.
STATUS:Disconnecting.
STATUS - disconnected
STATUS:Disconnected.
---------------
It seems that the system can connect to the firewall, but then the
connection is terminated -> no exception raised by the component when
connecting.
I know this information is still incomplete, but without installing a
better logging version of the service and waiting months for it to fail
again, it's all I've got to work with. If I can't resolve the problem
with this info, then I'll just tell the powers that be here about it, do
the new debugging version and try again when it fails in the future. I
guess I was hoping someone might have experienced something similar using
TIdFTP in long-running services or apps.
Thanks for your help!
Mike
--- posted by geoForum on http://delphi.newswhat.com |
|
| Back to top |
|
|
Remy Lebeau (TeamB)
Guest
|
| Posted: Thu Feb 16, 2006 8:03 am Post subject: Re: Socket count increasing with TIdFtp usage? |
|
|
"Mike Both" <both.michaelNOSPAM (AT) etsa (DOT) com.au> wrote in message
news:43f3f1d7 (AT) newsgroups (DOT) borland.com...
| Quote: | Sorry for the lack of clarity - I'm not referring to TIdFTP, I'm
referring to the code I'm working with. Once the file is ftped, it
is then parsed by another procedure. An exception is thrown
within this procedure as there is no file to parse.
|
Then your procedure is buggy. Please show the actual code.
| Quote: | The (poor) way the code is written currently means there is no
testing of whether the file has been successfully ftped in or not.
|
TIdFTP throwns an exception if a transfer fails. If that happens, then you
should not be calling your procedure at all.
| Quote: | The assumption is that if the ftp control connects, and doesn't
throw and exception, then the file has been retrieved.
|
Connecting and transferring are separate operations. But in general, yes,
if no exception is thrown when performing a transfer then the transfer was
successful.
| Quote: | Well - there is no exception thrown by TIdFTP, it just connects,
then the connection immediately terminates (as can be seen by the
log in the previous message, included here again):
|
Your log is not very detailed, but it appears that you are not actually
performing any transfer at all.
| Quote: | It seems that the system can connect to the firewall, but then the
connection is terminated -> no exception raised by the component
when connecting.
|
What you describe should not be possible. TIdFTP.Connect() internally sends
several commands to the FTP server before returning control back to the
calling thread. Your log suggests that the socket is being disconnected
immediately, before any of those commands can be processed. In which case,
TIdFTP is guaranteed to throw an exception when it fails to send a command
over, or read a response from, the lost connection.
| Quote: | I know this information is still incomplete, but without installing a
better logging version of the service and waiting months for it to
fail again, it's all I've got to work with.
|
I am sorry, but you are not providing enough details to adequately diagnose
your problem.
Gambit |
|
| Back to top |
|
|
Mike Both
Guest
|
| Posted: Wed Feb 22, 2006 12:03 am Post subject: Re: Socket count increasing with TIdFtp usage? |
|
|
Ok, here's the source for the function that's supposed to ftp transfer
the file (specified by global FFileName):
-------------------------------
procedure TdmFDI.GetFile;
begin
if FileExists('.\'+FFileName) then
begin
DeleteFile('.\'+FFileName);
end;
ftpconnection.ProxySettings.Host := FProxyName;
ftpconnection.ProxySettings.Port := strtoint(FProxyPort);
ftpconnection.ProxySettings.UserName := FProxyPassword;
ftpconnection.ProxySettings.Password := FProxyUser;
LogMessage('proxy host:' + ftpconnection.ProxySettings.Host);
LogMessage('proxy port:' + inttostr(ftpconnection.ProxySettings.Port));
LogMessage('proxy username:' + ftpconnection.ProxySettings.UserName);
LogMessage('proxy password:' + ftpconnection.ProxySettings.Password);
LogMessage('proxy type:' + inttostr(ord
(ftpconnection.ProxySettings.ProxyType)));
LogMessage('trying proxy connection...');
try
ftpconnection.Connect;
LogMessage('proxy connection succeeded, trying file get...');
ftpconnection.ChangeDir(FDirName);
ftpconnection.Get(FFileName, '.\'+FFileName);
LogMessage('after file get');
except
on E: Exception do
LogMessage('FTP connect failed: ' + E.Message,
EVENTLOG_WARNING_TYPE, true);
end;
if ftpconnection.Connected then
begin
ftpconnection.Disconnect;
LogMessage('after proxy disconnect');
end;
if not FileExists('.\'+FFileName) then
raise Exception.Create('File was not downloaded successfully!');
end;
-----------------------
As noted before, there is a custom ftp proxy handler for our firewall:
------------------
procedure TdmFDI.ftpconnectionCustomFTPProxy(Sender: TObject);
begin
try
ftpconnection.SendCmd('User anonymous@' + FHostName + ' ' +
FProxyUser);
ftpconnection.SendCmd('PASS randompassword');
ftpconnection.SendCmd('ACCT ' + FProxyPassword);
except
on E:exception do
LogMessage('Custom ftp connect failed - ' + e.Message,
EVENTLOG_WARNING_TYPE);
end;
------------------
What happens is that none of the above exceptions is thrown, and instead
an exception is thrown inside the bit of code after GetFile; the actual
file isn't present as the ftp connection through the firewall failed.
Does anything like 'wrong'?
Thanks :^)
Mike
--- posted by geoForum on http://delphi.newswhat.com |
|
| Back to top |
|
|
Remy Lebeau (TeamB)
Guest
|
| Posted: Wed Feb 22, 2006 1:03 am Post subject: Re: Socket count increasing with TIdFtp usage? |
|
|
"Mike Both" <both.michaelNOSPAM (AT) etsa (DOT) com.au> wrote in message
news:43fb9b59 (AT) newsgroups (DOT) borland.com...
| Quote: | Ok, here's the source for the function that's supposed
to ftp transfer the file (specified by global FFileName):
|
You did not specify where exactly the exception is occuring.
| Quote: | if FileExists('.\'+FFileName) then
begin
DeleteFile('.\'+FFileName);
end;
|
In the latest snapshot, if you set the ACanOverwite parameter to True, and
the AResume parameter to False, then the file is automatically deleted if it
exists before the download begins.
| Quote: | ftpconnection.Get(FFileName, '.\'+FFileName);
|
I am assuming that FFileName only contains a filename and no path
information at all, correct?
| Quote: | if ftpconnection.Connected then
begin
ftpconnection.Disconnect;
LogMessage('after proxy disconnect');
end;
|
You should have put the Disconnect() into a try..finally block.
| Quote: | if not FileExists('.\'+FFileName) then
raise Exception.Create('File was not downloaded successfully!');
end;
|
You are not handling partially downloaded files. Is that intentional? Your
code checks for the existence of the file after the download, but it does
not check whether the download was actually complete or not. If an
exception occurs during the download, you should delete the partial file if
you can't use it. Even better would be if you also change your download
procedure to return a True/False value as well, so the caller knows if the
file is ok before trying to use it.
| Quote: | As noted before, there is a custom ftp proxy handler for our firewall:
|
Why are you trying to manage that manually? TIdFTP already has built-in
support for proxy servers. In this particular case, the format of the
commands you are trying to send is fully supported. Simply set up the
values before calling Connect() (or Login() or you have set the AutoLogin
property to False).
With that said, use this code instead:
function TdmFDI.GetFile: Boolean;
begin
Result := False;
ftpconnection.ProxySettings.ProxyType := FProxyType; // set to
fpcmUserHostFireWallID
ftpconnection.ProxySettings.Host := FProxyName;
ftpconnection.ProxySettings.Port := Sys.StrToInt(FProxyPort);
ftpconnection.ProxySettings.UserName := FProxyUser;
ftpconnection.ProxySettings.Password := FProxyPassword;
LogMessage('proxy type:' + IntToStr(Ord(FProxyType)));
LogMessage('proxy host:' + FProxyName);
LogMessage('proxy port:' + FProxyPort);
LogMessage('proxy username:' + FProxyUser);
LogMessage('proxy password:' + FProxyPassword);
ftpconnection.Host := FHostName;
ftpconnection.UserName := 'anonymous';
ftpconnection.Password := 'randompassword';
LogMessage('trying FTP connection...');
try
ftpconnection.Connect;
try
LogMessage('FTP connection succeeded, trying file get...');
ftpconnection.ChangeDir(FDirName);
try
ftpconnection.Get(FFileName, '.\'+FFileName, True,
False);
except
Sys.DeleteFile('.\'+FFileName);
raise;
end;
LogMessage('after file get');
finally
ftpconnection.Disconnect;
LogMessage('after FTP disconnect');
end;
Result := True;
except
on E: Exception do
LogMessage('FTP download failed - ' + e.Message,
EVENTLOG_WARNING_TYPE);
end;
end;
procedure TdmFDI.DoSomething;
begin
if GetFile then
// use the file as needed ...
else
// the file was not downloaded ...
end;
| Quote: | an exception is thrown inside the bit of code after GetFile
|
You did not show any of that code.
| Quote: | the actual file isn't present
|
The only way that can happen is if you are not connecting to the server at
all. If the connection is successful, and the navigation to the folder is
successful, then you are guaranteed to have a file created, even if it is 0
bytes in size due to a failed transfer.
Gambit |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
