Author Topic: What does "building file list" mean?  (Read 17727 times)

0 Members and 4 Guests are viewing this topic.

Mizkreant

  • Member
  • **
  • Posts: 11
    • View Profile
What does "building file list" mean?
« on: 2019-02-21 21:57:29 »
And why does Thumbsplus have to do it every time I switch folders? If all of the data and the thumbnails are in the database already, why is it doing this so often, and why does it take so long? I already allocated the maximum amount for the thumbnail cache (1000 apparently). I've tried storing the database on an SSD, but this didn't improve the speed or frequency of building the list. If this is unavoidable, perhaps future versions should have the option of storing more in RAM to avoid this.

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
Re: What does "building file list" mean?
« Reply #1 on: 2019-02-21 22:21:53 »
It means that ThumbsPlus is reading the thumbnail records for that folder from its database, matching it to the image files on disk, and building the list so it can display it.

Things that may slow down the process include:

- When the folders and files are on a slow (network or external) drive
- When you have very large numbers of files or subfolders in a folder (e.g. 10,000)
- When you use Color Management in the thumbnail listing (Options | Preferences | tab:Thumbnail view | Enable Color Mgt and Options | Viewing | Color Management | Thumbnail View)
- When you display one or more User Fields in the thumbnail listing (press light bulb button above listing to select fields)

For many more suggestions see the ThumbsPlus Performance Hints on Cerious' website.

Anyway, rest assured that the program should (be able to) work much faster than you seem to be experiencing today.
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

Mizkreant

  • Member
  • **
  • Posts: 11
    • View Profile
Re: What does "building file list" mean?
« Reply #2 on: 2019-02-21 22:39:54 »
The frequency with which it does this suggests that it doesn't cache very much in RAM. Also, the time it takes is inconsistent and I don't understand why. I have a folder which I work with regularly which has about 20,000 files. Most of the time building the file list only takes 5-10 seconds. Sometimes, however it has taken several minutes to display.

I've turned off Color Management for thumbnails. However, I still meet two of the criteria you listed for slowdowns. I have a large amount of files, and some of them are stored on an external drive. The database itself is stored on an internal HDD. As I said though, whether the database was placed on an SSD or HDD didn't seem to matter.

Is a SQLite3 database slower than MS Access? I've used Thumbplus 7 in the past and I seem to remember it being faster, though I could be mistaken.

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
Re: What does "building file list" mean?
« Reply #3 on: 2019-02-22 02:50:50 »
I have a folder which I work with regularly which has about 20,000 files.

That's almost certainly the cause of the delays. With so many files, the folder will be read very slowly due to limitations in Windows. However, once it has been read, it will be temporarily stored in Windows' internal disk cache and then it will load faster during that session, until you, or a background program, reads some differrent folders that will then be cached at the expense of your first folder.

I don't think there's much that you can do about this. Defragmenting that drive may help for a while. Also, I believe there are third party drive caching tools that cache your HDD to RAM or to SSD, but I have no experience with them.

As for SQLite3, it used to be slow in earlier versions of TP, but it should offer very good performance in the current version 10 sp2.
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

Mizkreant

  • Member
  • **
  • Posts: 11
    • View Profile
Re: What does "building file list" mean?
« Reply #4 on: 2019-02-22 07:26:41 »
been read, it will be temporarily stored in Windows' internal disk cache and then it will load faster during that session, until you, or a background program, reads some differrent folders that will then be cached at the expense of your first folder.
What setting in Windows controls this? Virtual Memory?

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
Re: What does "building file list" mean?
« Reply #5 on: 2019-02-22 11:14:20 »
I don't think Windows offers you any controls over its disk caching. Maybe through the registry.. try Google.
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

Mizkreant

  • Member
  • **
  • Posts: 11
    • View Profile
Re: What does "building file list" mean?
« Reply #6 on: 2019-02-22 14:59:40 »
I found an old piece of software from SysInternals called Cacheset which controls Windows' disk cache. Unfortunately it didn't help. The cache never seems to go above 700 MB no matter what its set to.

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #7 on: 2019-03-04 10:42:09 »
....    I've used Thumbplus 7 in the past and I seem to remember it being faster, though I could be mistaken.

YES - T7 is blazing fast compared to t10 .. I just posted my tale of disappointment and then I saw that you were onto the same thing.

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #8 on: 2019-03-04 10:54:12 »
I have a folder which I work with regularly which has about 20,000 files.
....  However, once it has been read, it will be temporarily stored in Windows' internal disk cache and then it will load faster during that session, until you, or a background program, reads some different folders that will then be cached at the expense of your first folder.


I'm not sure that this is entirely correct, but points to something that may be going on, not with windows but with T10 -  I suspect that it is NOT the entire file list of thumbs in the folder that is cached ( for display)  but only those that are going to be displayed in the current shown window.  I say this because if you wait for a spell,  thinking that OK ill give it some time to load all the thumbs into somewhere and then Ill start scrolling, you will be disappointed because as soon as you scroll "off" the current display, to say the next rows of images, T10 is going to lag as it seems to be rebuilding something and then displaying them... HOWEVER if you suffer thru the delayed display as you scroll a bit and wait and then scroll a bit more and wait, and scroll a bit more and wait ..... to the bottom of the folder of your images, THEN after that, the fast scroll back and forward and fast presentation of the thumbnails is as it was in T7.

But, then, if you click to another folder and then click back into the first folder you will have to suffer this delay and aggravation all over again.

Again -  T7 did not have this problem even on Win 10 machines.

« Last Edit: 2019-03-04 17:45:40 by Street Mann »

Mizkreant

  • Member
  • **
  • Posts: 11
    • View Profile
Re: What does "building file list" mean?
« Reply #9 on: 2019-03-06 04:47:27 »
YES - T7 is blazing fast compared to t10 .. I just posted my tale of disappointment and then I saw that you were onto the same thing.
It's good to know I wasn't imagining things. Thumbsplus 10 is very inefficient in terms of caching. I have 16 GB of RAM and TP only uses 700 MB with the max thumbnail cache size.

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
Re: What does "building file list" mean?
« Reply #10 on: 2019-03-06 15:48:40 »
YES - T7 is blazing fast compared to t10 ..

Again, the bad performance that you two are seeing is not representative of v10 and is not inherent to v10. Its speed should be comparable to that of v7, and on my system (and countless others) it is.
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #11 on: 2019-03-06 16:09:37 »
What "system" you working on .. PC? I am on a HP Spectre 360 SDD 16gb ram images on the disk and also have images on USB external hard drive.

Update .. It seems that the default database will do the pictures folder on the HD faster that it will do the display on the same folder using the database that is on the External.. Let me move the DB to the computer and see what happens ....

« Last Edit: 2019-03-06 16:34:01 by Street Mann »

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
Re: What does "building file list" mean?
« Reply #12 on: 2019-03-06 16:40:53 »
Let me move the DB to the computer and see what happens ....

;D

Put it on the SSD! It's going to be very slow on an external drive.
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #13 on: 2019-03-06 18:21:35 »
Moved the DB to SDD .. No Joy ... BUT .. it is getting real crazy. When I select the default database and then go to a folder having a lot of images on it , said folder located on my SDD too, the build of the thumbnails and the scrolling is zippy fast ... If I set  the database to the one that I converted from my td4 and then look into the same folder that is on the SDD that I looked at before, the images load and scroll slow ...

The image folder is the same and I am just setting a different database to use as the default ... This folder has never been indexed by either database ...  Let me scan my External  drive using the default database and see what happens ... my problem i foresee is how to get my keywords etc over to the new database.


 I am running a TP db repair right now .. ill see if that changes anything.

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
Re: What does "building file list" mean?
« Reply #14 on: 2019-03-06 23:48:50 »
my problem i foresee is how to get my keywords etc over to the new database.

If this only concerns JPG, TIF and/or PNG files, TP7 has an IPTC editor that lets you write your database keywords and annotations into the metadata headers of the actual image files (doesn't work for raw). When that's been done, you can create a new database in TP10, scan the image files, and whilst doing so have TP10 automatically copy the keywords and annotations into the respective database fields. There's a bit more to this procedure - holler if interested.

Anyway, it does sound like you have a problem in the converted TD4 database. If you want to look deeper into that (e.g. compare its structure to a new database), Alex Nolan's MDB Viewer Plus is a great (and free) tool for that purpose. Before doing all that, you could also try to simply re-scan your image folders (to rebuild the thumbnail records) and see if that helps.
« Last Edit: 2019-03-06 23:50:26 by Daan van Rooijen »
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #15 on: 2019-03-07 10:33:26 »
my problem i foresee is how to get my keywords etc over to the new database.

 - holler if interested.



Im hollering ...

I have figured out how to map Keywords and Annotations to the metadata fields that travels with the image. But is there a way to batch getting it FROM the metadata and into the image database field (the Info Section)??

I see that I can copy and paste from the Metadata into Info Section (for all except gallery information) but doing that for 10k images is not appealing.


 .. I note that if I add a keyword T10 does not add it into the metadata, but to the image Info section which I am surmising is recorded into the database and not into the metadata,.  [Keyword in the Info  tab is different field that Keyword in the Metadata tab.]

« Last Edit: 2019-03-07 12:19:00 by Street Mann »

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #16 on: 2019-03-07 11:07:36 »
What about the gallery assignments?

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
Re: What does "building file list" mean?
« Reply #17 on: 2019-03-07 13:05:35 »
I have figured out how to map Keywords and Annotations to the metadata fields that travels with the image.
Good. In TP7, the setting Options | Preferences | General | "Retain original file date/time when quick process turning or editing IPTC" will prevent that copying the keywords and annotations into the image files alters their time and date stamps in Windows.

(if that has already happened, TP7 has a command "Modify File Info" (under Image | Quick Process) that lets you set file dates to their original EXIF 'date created' date, in batches too).

Quote
But is there a way to batch getting it FROM the metadata and into the image database field (the Info Section)??
Yes, TP10 can read the IPTC keywords and Caption when it (re)creates thumbnails, and copy their contents to the Annotation and Keywords database fields (which are shown in the info section of the metadata panel). See Options | Preferences | Metadata and Options | Preferences | Keywords.

As for galleries, there's no simple mechanism to copy those from TP7 to TP10. What I've done in the past is give all images in a gallery a keyword like "GALL1". Then copy those keywords (along with any regular keywords)  into the metadata of the image files using the IPTC editor. Then, once they're cataloged in TP10, select all files that match GALL1 and put them in a new gallery (which will then hold the same files as the original gallery in TP7). This worked well but I hope you don't have hundredss of galleries :)
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #18 on: 2019-03-07 14:09:47 »
Not to worry  :o

I only have about 60,000 images and it only takes about 1 second er image to process in the batch mode so Ill be able to have a lot of coffee, do some push-ups, watch my grand-kids grow up, and then go back to see if the process has finished up!

i was thinking the same kind of workaround for the galleries.

Ill let you know how it turns out.

It appears now that doing things this way is going to speed up the scroll problem ... Fingers crossed



 
« Last Edit: 2019-03-07 14:35:13 by Street Mann »

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #19 on: 2019-03-08 13:12:59 »
Yes, TP10 can read the IPTC keywords and Caption when it (re)creates thumbnails, and copy their contents to the Annotation and Keywords database fields (which are shown in the info section of the metadata panel). See Options | Preferences | Metadata and Options | Preferences | Keywords.

LOL .. I was going crazy until i reread your post and remade the thumbs ... that annotation came in but not the keywords until I did that ...



Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #20 on: 2019-03-09 09:17:34 »
Well out of 118,000 images in my main drive .. I  only have about 90,000 more to index with the metadata utility ..... still running after 10 hours ... another 20 to go .....

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
Re: What does "building file list" mean?
« Reply #21 on: 2019-03-09 12:24:00 »
Well out of 118,000 images in my main drive .. I  only have about 90,000 more to index with the metadata utility ..... still running after 10 hours ... another 20 to go .....

I thought you were doing this from ThumbsPlus v7?

TP7, which had the keywords and annotations in its database, which has the IPTC-editor that can transfer those to the image files, and which didn't suffer any delays? Something doesn't add up here.
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #22 on: 2019-03-09 17:09:57 »
Hey Daan

Did not try to use the T7 xfer utility on all these files ..

I started using T10 because T7 was acting in unpredictable ways on my Win 10 machine .. even after repairing that database.

So, I converted a td4 T7 db to the new T10 format, and then, on using T10 saw that the difficulty was arising on the slow scrolling... I also saw that the T7 was a bit buggy an sometimes would not fire up under my Windows 10.  Because of the T7 on Win 10 seeming bugginess and umpredictiblality, I stuck with the T10 Win 10 version and the tbdb8 new database structure. I have so many images that I did not want to do all the editing/metadata writing with my TD4 database in T7 and totally mess it up, so I stuck with the T10 metadata editor and changed to using  Win 10 T10 on the conveted tpdb8 file..  It takes about 1 second per image to process the data out of the slow-scrolling T10 "myconvertedtfromTP7td4 database into a clean T10 Win 10 thumbs.tpdb8 database.

« Last Edit: 2019-03-09 17:13:13 by Street Mann »

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
Re: What does "building file list" mean?
« Reply #23 on: 2019-03-09 19:00:10 »
I see..! Well, best of luck with it. When the whole procedure is over, you should have a much more responsive database.   :-)
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #24 on: 2019-03-11 12:05:11 »
I see..! Well, best of luck with it. When the whole procedure is over, you should have a much more responsive database.   :-)

Finished .. lets see what was that     only about 3 days to batch process the meta data to the 118,000 files ... only 257 errors .. now to go make the 75 galleries again ...

A prelim check indicates that things are scrolling fast now...

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #25 on: 2019-03-19 20:16:31 »
 :) ;) :D ;D

Success .. i manually had to re-create my galleries ( T7 does not move over the galleries) and then populate them with the images according to the Metadata procedure of Daan in previous replies in this thread .. Scrolling is now fast thru the thumbs of images and all is well now...

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
Re: What does "building file list" mean?
« Reply #26 on: 2019-03-19 21:05:19 »
Thanks, that's great to hear!  :)

So, that's "one down, one to go" -- I hope Mizkreant will manage to solve the speed issue on his system, too.
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

Street Mann

  • Full Member
  • ***
  • Posts: 62
    • View Profile
Re: What does "building file list" mean?
« Reply #27 on: 2019-03-19 22:14:16 »
i wonder if his problem is that he needs to create a clean new database and do the same thing as I did.

I just called up a gallery with 5900 images and 27gb size in about 1 second.