 |
BorlandTalk.com
Borland discussion newsgroups
|
| View previous topic :: View next topic |
| Author |
Message |
Dennis
Guest
|
 Posted: Fri Dec 26, 2003 11:36 pm Post subject: Virtual listview and inserting data |
 |
|
I am trying to find out if there are other ways to insert items
into a virtual listview, instead of using a TList. I am
assuming when you use a TList, you are using a lot of overhead
then. This is what i have so far to read event logs, sometimes
the log count goes in the mega thousands.
struct TLogInfo
{
AnsiString aType;
AnsiString aDate;
AnsiString aTime;
AnsiString aSource;
AnsiString aCategory;
};
void __fastcall TForm1::ListView1Data(TObject *Sender, TListItem *Item)
{
TLogInfo* loginfo = (TLogInfo*)logList->Items[Item->Index];
// add items to the listview
}
I am not sure how fast this technique is when adding the items
to the TList, then adding them to the ListView. Plus the kind
of overhead that is required. But if there is a faster way, i
want to know what it is.
Thanks.
Dennis
|
|
| Back to top |
|
 |
Dennis
Guest
|
| Posted: Sat Dec 27, 2003 12:12 am Post subject: Re: Virtual listview and inserting data |
|
|
| Quote: | keep, say, 100 records in memory at any given time, just enough for the
actual display and a few records above the below the visual display for
scrolling purposes, and then use the OnDataHint event to tell you when you
should adjust your cache with new records as needed.
|
This sounds like a great idea, i was just reading the help on the OnDataHint, but i don't see how to really use it. With the following example of what i am doing. Can you show me how to use the OnDataHint within my code?
// Ietms added via the OnData property
void __fastcall TForm1::ListView1Data(TObject *Sender, TListItem *Item)
{
// items pulled from the TList() into the ListView;
}
ListView1->Items->Count = 20000;
ListView1->Invalidate();
I guess as i think about it, i am getting a bit more confused.
How would i go about in adding about 100 items to my list, then
add the rest when needed? Would that be something like a loop
when adding the items to the list in the first place.
Really confused now.
Dennis
|
|
| Back to top |
|
|
Remy Lebeau (TeamB)
Guest
|
| Posted: Sat Dec 27, 2003 12:54 am Post subject: Re: Virtual listview and inserting data |
|
|
"Dennis" <sorry (AT) nomail (DOT) com> wrote
| Quote: | I am trying to find out if there are other ways to insert
items into a virtual listview, instead of using a TList.
|
Of course. The whole purpose of a virtual ListView is that you can store
the data anyway you want, the ListView has no concept of the data at all.
It asks the application to provide necessary data when actually needed, and
then the application can decide where the data comes from and how it is
produced.
| Quote: | I am assuming when you use a TList, you are using a lot
of overhead then.
|
Not as much as you may think.
| Quote: | This is what i have so far to read event logs, sometimes
the log count goes in the mega thousands.
|
That is fine. Although with that many records, you may not want to store
them all in memory at one time, but instead cache a few at a time. The
ListView can only display so many records at a time to begin with. You can
keep, say, 100 records in memory at any given time, just enough for the
actual display and a few records above the below the visual display for
scrolling purposes, and then use the OnDataHint event to tell you when you
should adjust your cache with new records as needed.
| Quote: | I am not sure how fast this technique is when adding the
items to the TList, then adding them to the ListView.
|
It is very fast. Keep in mind that in virtual mode, you don't actually add
new items to the ListView itself at all. All you do is tell the ListView
how many items it is supposed to display, and then is requests the data when
needed. Whether you display 1 record or 1 million, the ListView's memory
usage and performance are constant. The only overhead involved is in your
actual data storage, whatever you decide to use.
Gambit
|
|
| Back to top |
|
|
Remy Lebeau (TeamB)
Guest
|
| Posted: Sat Dec 27, 2003 3:34 am Post subject: Re: Virtual listview and inserting data |
|
|
"Dennis" <sorry (AT) nomail (DOT) com> wrote
| Quote: | This sounds like a great idea, i was just reading the help on
the OnDataHint, but i don't see how to really use it.
|
Whenever the user scrolls the ListView, it knows which items will be needed
during/after the scrolling, so it triggers the OnDataHint event to tell you
what the starting and ending indexes are for the range of items that will be
needed. That helps you prepare the necessary data ahead of time before the
OnData event will be triggered later on.
| Quote: | I guess as i think about it, i am getting a bit more confused.
|
You are confusing yourself because you are using TList to begin with. Since
TList already contains everything in memory, OnDataHint is not necessary in
this case, you already have everything readily available. The OnDataHint
event becomes more useful when you are storing data in a database, or a file
on the harddrive, or the like. Places where performance and memory issues
make a bigger difference.
| Quote: | How would i go about in adding about 100 items to my list,
then add the rest when needed?
|
In your example, you would need to use a second TList. When OnDataHint is
triggered, you take the items specified in the range and copy them from the
first TList into the second TList, and then use the second TList for the
OnData event instead of the original. But that would be redundant given the
nature of TList's design.
Gambit
|
|
| Back to top |
|
|
Dennis
Guest
|
| Posted: Sat Dec 27, 2003 4:06 am Post subject: Re: Virtual listview and inserting data |
|
|
| Quote: | event becomes more useful when you are storing data in a database, or a file
on the harddrive, or the like. Places where performance and memory issues
make a bigger difference.
|
Actually i will be using a database to retrieve the
information from. I just used a TList because i was reading
an article from bridgespublishing.com, and it utilized a Tlist.
I don't know much about using a Virtual ListView, but after
reading the article, i became a little more familiar with it.
I am supposed to be retrieving all the information from the
eventlog, then i will be storing everything into a db for use
later on. So you can see my dilemma here, i am thinking i have
to use a TList, and you are showing me that i don't need a
Tlist if i access the info directly from a DB.
I guess whats confusing me more, is, the thought of loading
about 100 records into the listview. Then when i need more, i
just load another 100. To be totally honest with you, i would definitely have to see an example on doing something like that.
I searched mers.com, but i could not find anything on doing
that specific task. I guess i will have to keep looking,
because i just don't see how that will be done.
|
|
| Back to top |
|
|
Remy Lebeau (TeamB)
Guest
|
| Posted: Sat Dec 27, 2003 7:16 am Post subject: Re: Virtual listview and inserting data |
|
|
"Dennis" <sorry (AT) nomail (DOT) com> wrote
| Quote: | I am supposed to be retrieving all the information from the
eventlog, then i will be storing everything into a db for use
later on. So you can see my dilemma here, i am thinking i
have to use a TList, and you are showing me that i don't need
a Tlist if i access the info directly from a DB.
|
Yes, you can access the DB directly, if desired. As I mentioned earlier,
the whole point of using a ListView in virtual mode is that it doesn't care
where the data is actually coming from. As long as you can convert a
ListView index into its cooresponding storage element that you can then
extract data from, then you can use anything you want for the actual data
storage.
| Quote: | I guess whats confusing me more, is, the thought of
loading about 100 records into the listview. Then when
i need more, i just load another 100.
|
Ok, let me try to clearify it for you by using a real-world example that I
myself use for my day job (sorry, no source code - that's confidential).
We have a custom logging application that we developed for our commerical
applications, where all of our applications can report debug and tracing
messages and such to a central log file on the machine. We have a custom
binary format for our log files. So we had to create a custom log file
viewer to display the file data as well. Our log files can contain
thousands upon tens of thousands, even millions, of records very quickly or
over time, depending on the detail level of the log records that are being
produced. We also had to provide filtering and searching of records as
well. So having a fast, scrollable display that can handle that many
records at a time was important. We went with a virtual ListView, and it
works very well, albeit it is amongst the most complex code I have ever seen
(I did not write it, my predecessor did - ah the joys of maintaining ugly
legacy code).
What we implemented is way way beyond anything you would need at this point,
so lets ignore the ugly parts for now. Basically, all we have is an array
in memory that contains at most about 300 records at any given time. When
the viewer first opens a log file, the viewer scans through the file
records, 1) to determine how many records are going to be displayed in the
first place, and 2) to scan the actual record contents and test them against
the current filter criteria and such, placing matching records into the
memory array. After 300 records have been loaded into memory, it stops
loading records, and then keeps track of the file indexes of the first and
last records that were stored into the array, as well as the ListView
indexes that they coorespond to.
By tracking the indexes, when the OnDataHint event occurs, the viewer can
check whether the indexes provided by the event handler coorespond to the
file indexes of the records that are loaded into memory. If they match,
then nothing extra needs to happen yet as the requested records are already
in memory. When the ListView indexes provided by the OnDataHint event go
outside the range of the indexes for the current records loaded in the
memory array, such as during scrolling, pressing Ctrl+Home or Ctrl+End, page
up/down, etc., the stored indexes help the viewer determine which records in
memory can be discarded from the array, and where in the log file any new
records need to be read from to re-fill the array, always keeping the array
as full as possible, always keeping the tracking indexes up-to-date with the
range of items that the ListView wants to display.
For example, if the OnDataHint's StartIndex is 50 items before the index of
the first record in the array, then the view knows that it has to discard
the first items of the array, go back to the log file, and load 50 new
records from the previously known location in the file. Likewise, if the
OnDataHint's EndIndex is 10 items after the index of the last record in the
array, then the viewer can discard the last items in the array, go back to
the log file, and read the next 10 records from the file. If the user does
something like grabbing the scrollbar and dragging it halfway down the
screen quickly, then the StartIndex/EndIndex provided by the OnDataHint
event are going to be way outside the range of the current records in the
array, so the viewer has to just discard all of the current records in the
array, calculate where in the file it needs to seek to, and then read in 300
brand new records again.
I know, it sounds complicated, and it can be, especially the more elaborate
the buffering and tracking becomes. But doing it this way, whenever the
OnData event handler is triggered, it can do a simple calculation to convert
the provided ListView index into an index within the current memory array
contents, and display the data for the appropriate record. The whole
purpose of the OnDataHint event is for the application to have a chance to
pre-cache any necessary items that the OnData event is likely to ask for
soon afterwards. Depending on the screen resolution and window size and
such, a ListView can display anywhere from 10-50 items at a time, so the
application should have the data for those particular items that are
actually visible readily available as soon as possible. As the display
changes, so should the buffering as well to ensure the data for whatever
items are actually visible are always available when needed.
I'm sorry, but I can't make it much clearer then that without visuals. You
just have to think of the ListView in terms of visual pages, and perform
your buffering (if you do buffering at all) in terms of the pages of records
that are going to be displayed.
| Quote: | I searched mers.com, but i could not find anything on doing
that specific task.
|
That is because it is not always a simple task to accomplish, thus nodoby
ever delves into it when dealing with simple examples. It is an
intermediate level task that you just have to develop over time to suit your
particular needs for your project.
Gambit
|
|
| Back to top |
|
|
Dennis
Guest
|
| Posted: Sat Dec 27, 2003 10:12 pm Post subject: Re: Virtual listview and inserting data |
|
|
Ok, i understand pretty much everything you said here. I was
looking at the example in the bcb examples dir, and i played
with the virtual listview example. One of the things in the
code that i just cant figure out, is when they load the list
the second time.
The first time the list is loaded, it displays 13 items from
my computer. StartIndex = 0, and EndIndex = 13. Then when
i go into the windows dir, EndIndex = 51, but it loads only
19 items at a time.
So is it actually loading all the items into the TList(), then
when you scroll, it pulls more from the TList() as it needs it?
That is my only conclusion on that one, because it never adds
any more items into the TList() as far as i can see anywhere in
the code. As you click on a dir, it loads everything it needs
into the list, then as you scroll the listview, it pulls more
data into the listview as needed.
But that is still utilizing the TList to the fullest, i guess
it will speed up the display of items in the listview, because
it only displays a few items at a time. I still would like to
know how to access the data from the DB directly without using
a TList.
|
|
| Back to top |
|
|
Remy Lebeau (TeamB)
Guest
|
| Posted: Sun Dec 28, 2003 12:29 am Post subject: Re: Virtual listview and inserting data |
|
|
"Dennis" <sorry (AT) nomail (DOT) com> wrote
| Quote: | One of the things in the code that i just cant figure out,
is when they load the list the second time.
|
I have not looked at the example, but I would imagine that it only uses a
single list that has everything loaded in memory at one time. That is the
basic implementation for a virtual ListView. Extra buffering is not a
requirement, it is merely a performance helper, if actually needed.
| Quote: | The first time the list is loaded, it displays 13 items from
my computer. StartIndex = 0, and EndIndex = 13. Then
when i go into the windows dir, EndIndex = 51, but it loads
only 19 items at a time.
|
I cannot comment on that without seeing your actual code.
| Quote: | So is it actually loading all the items into the TList(),
then when you scroll, it pulls more from the TList()
as it needs it?
|
I doubt that is the case for just an example. It would make more sense for
the entire directory to be loaded into the list in full at one time instead.
Please keep in mind that what I described about using multiple lists and
such is only *one* way to implement a virtual ListView, and what I described
about the log file viewer is merely how *I* implement my use of a virtual
ListView. There are certainly other ways to do it as well. How *you*
decode to implement a virtual ListView is based on *your* actual needs for
your project.
| Quote: | That is my only conclusion on that one, because it never adds
any more items into the TList() as far as i can see anywhere in
the code. As you click on a dir, it loads everything it needs
into the list, then as you scroll the listview, it pulls more
data into the listview as needed.
|
If everything is loaded into the list at one time, there is no additional
loading is ever performed. It is not needed, the data is already available
when it is requested by the events.
| Quote: | But that is still utilizing the TList to the fullest, i guess
it will speed up the display of items in the listview,
because it only displays a few items at a time. I still
would like to know how to access the data from the
DB directly without using a TList.
|
To work with any records from any DB for any reason, you have to perform a
query first to set up the appropriate records and create a recordset for
them. Then, you set the ListView's Items->Count property to the number of
records in the recordset, and then just access the appropriate record from
the recordset in the OnData event handler based on the provided index.
I'm curious, why are you using a TListView instead of a TDBGrid to display
the record data?
Gambit
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
