Author Topic: TD4 Squeezer Tool  (Read 5564 times)

0 Members and 1 Guest are viewing this topic.

mschnell

  • Full Member
  • ***
  • Posts: 98
    • View Profile
TD4 Squeezer Tool
« on: 2017-04-21 02:34:41 »
Being hit by the Access database 2 GByte limit, I am doing a TD4 squeezer Tool. (Which I of course will share when it's ready and which supposedly also can be mofified to support tpdb8 files.)

The goal is to have the user collect would-be "old" thumbnails in a Gallery and then run the Squeezer that will reduce the space by applying an extremely strong JPEG compression on the selected thumbnail pictures. Same would be displayed a lot less nice in thumbnail view, but could be recreated by just hitting F3.

Maye this is not possible, then the tool will either remove the selected thumbnail pictures or replace them by some very small default picture.

My program already can access the database and find and load the data of the selected thumnail picture information.

Now I found that this information is not a full JPF file content, but supposedly just the pixel information without the JPEG header. This information supposedly is given in some other fields of the thumbnail record.

Does anybody know a definition of this information, so that I can manage it ?

Thanks,
-Michael

mschnell

  • Full Member
  • ***
  • Posts: 98
    • View Profile
TD4 Squeezer Tool
« Reply #1 on: 2017-04-21 13:50:30 »
I found that the information in the Thumbnail filed in the database in fact is a standard JPEG information, i.e. the same bytes as would be in a *.jpg file. This even seems to be an official database standard, as the softwre development environment I use, does have a "control" that simply can be assigned the database (BLOB) field name and then it will automatically display the picture without  me writing any code.

I also was able to reduce the picture JPEG quality to 2% with that shrinking an average thumbnaul picture (Quality 70 %) from 4K to 1K. This is exactly what  I was up to.

-Michael

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
TD4 Squeezer Tool
« Reply #2 on: 2017-04-21 18:14:31 »
> I found that the information in the Thumbnail filed in the database in fact is a standard JPEG information, i.e. the same bytes as would be in a *.jpg file.

Phillip once said "a JPEG thumbnail in the database is a complete JPEG data stream, including JFIF header". If you search the old newsgroup messages for 'stream', a bit more information may pop up. Anyway, it seems you got it to work! What development environment are you using? And did you find out if you still have to 'compact' the database afterwards?
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

mschnell

  • Full Member
  • ***
  • Posts: 98
    • View Profile
TD4 Squeezer Tool
« Reply #3 on: 2017-04-21 18:39:39 »
> Anyway, it seems you got it to work! What development environment are you using? And did you find out if you still have to 'compact' the database afterwards?

As a development environment I use "Lazarus" (free Pascal IDE) enhanced by "ZEOS" Database access library.

Unfortunately, using this development system,  I was not yet able to write the updated records back in the database. Right now, I get an error message from the Access (=jet) database engine. I suppose I need to do some appropriate settings before writing.

-Michael

mschnell

  • Full Member
  • ***
  • Posts: 98
    • View Profile
TD4 Squeezer Tool
« Reply #4 on: 2017-04-24 18:41:39 »
It works !!!

As I did suppose, "post"ing an updated (with th modified LOB containing the thumbnail JPEG) record in the database creates a new record and marks the old one as deleted. So the database size increases by using the tool and will only be reduced when compacting it.

-Michael

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
TD4 Squeezer Tool
« Reply #5 on: 2017-04-24 18:55:18 »
> It works !!!

Good news! :-)

> As I did suppose, "post"ing an updated (with th modified LOB containing the thumbnail JPEG) record in the database creates a new record and marks the old one as deleted. So the database size increases by using the tool and will only be reduced when compacting it.

I see.. how do you solve this? By posting new records until 2GB is hit and then compacting, and then repeating that cycle? Maybe you could create a new database instead and copy the records (with updated thumbnails where possible) to it?
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

mschnell

  • Full Member
  • ***
  • Posts: 98
    • View Profile
TD4 Squeezer Tool
« Reply #6 on: 2017-04-25 01:13:44 »
Yep.

The original idea was an "incremental" use:
 - have the user create a gallery that contains the "not that important thumbnails".
 - shrink all those that are in that gallery and not already smaller than the selected maximum size.

So the user can easily manage the result and add items to the "squeezed" gallery.

It might be a nice idea to add an option to move all file in a new database (which supposedly would have been created by TP before) instead of doing the change in-place.

-Michael

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
TD4 Squeezer Tool
« Reply #7 on: 2017-04-25 03:58:24 »
> a new database (which supposedly would have been created by TP before)

Or (if there is sufficient room on disk) make a copy of the present database, delete its records and compact it, and then it's ready to receive the new, size-reduced, records.
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

mschnell

  • Full Member
  • ***
  • Posts: 98
    • View Profile
TD4 Squeezer Tool
« Reply #8 on: 2017-04-25 15:07:57 »
I am don't know how to make sure this will retain all relations to Galleries, Keywords, Annotations, etc.

-Michael

mschnell

  • Full Member
  • ***
  • Posts: 98
    • View Profile
TD4 Squeezer Tool
« Reply #9 on: 2017-04-28 10:21:29 »
Herer the help text of the tool:

License:

This software is a personal tool and provided publicly with the hope it might be useful.

Nonetheless it might behave erratically in any possible way.

Please be sure to have a decent backup of you database before using the tool



Motivation:

When using an “Access” type database for Thumbsplus, some day a limit is reached as the database file can't grow to more than 2 GByte in size.
“Compacting” the database will remove unused the information, that still might exist from previously deleted or replaced records, but one day the
database is just full.

TD4Squeezer will reduce the quality (and with that the needed space in the database) for a selected set of thumbnail images.

You can always recreate the full quality of the thumbnail image of one or more files when viewing them, just by hitting “F3” in Thumbsplus.


Usage:

At first you need to take care that the database is not completely full but there is some room for TD4-Squeezer top work with. To provide this, in
ThumbsPlus, you might need to delete the thumbnails of some of your pictures using the “Remove Thumbnail” function. Of course you don't need to delete the
pictures themselves. The files can stay unchanged in the directory.

Next you should compact the database using the “Database” menu in Thumbsplus.

Now you can select a number of files you are not likely to use in the near future and assign them to a gallery. Here, the “Find” menu can be helpful to
select complete directories with their subdirectories, or to select files older than some appropriate date. The default gallery name used by TD4-Squeezer is
“__squeeze” (starting with two underscore characters). It is not recommended to move a huge count of thumbnails in that gallery. This might not be very
handy. Instead you can quickly add files or even delete the gallery and recreate it at any point in time.

After this preparation TD4-Squeezer is started. It will ill reduce the quality of those thumbnail images that are in the gallery (default name “__squeeze”), if
they are bigger than a selectable count of bytes (500, 1000, 1500, or 2000). Hence starting a new squeezing (with the same size) run will only handle this
Thumbnail images that are not already small enough. Depending on the picture content, it might be impossible the squeeze the thumbnail image to the selected size.
In that case the process is stopped with a quality factor of 1 %. Those images will (unsuccessfully) be re-tried with the next run of TD4-Squeezer, if they are
still given in the gallery.

After the thumbnail images hve been squeezed, the database nees to be compressend in ThumbsPlus to finally reduce it's file size .



Details:

The database file is given in a text input field. By default it is “thumbs.TD4” in the same directory as the TD4-Squeezer program file.

The “Find” Button can be used to to locate the database file.

If the “Fake” checkbox is checked the modified information is not written into the database. This way the run for “TD4-Squeezer” can be tested.

If “Pause to watch” is checked each thumbnail picture will be shown in it's original and in its modified state will be shown for half a second. The checkbox
can be toggled while “TD4-Squeezer” is running.

The “Gallery” text field holds the name of the gallery you want to use to select the thumbnails to be squeezed. If this field is empty, all tumbnails that have a size
bigger than the max size will be squeezed. The default gallery name is “__squeeze”

The “max Size (Byte)” selection defines the quality of the resulting thumbnail images. It can be set to 500, 100, 1500 or 2000 Bytes. The default is 1500. The
JPEG quality factor is automatically set to a value that results in a size slightly smaller than the given limit.

The “max Conversion Count” stops the squeezing run after the given number of converted thumbnail pictures.

The “Start” button starts the squeezing according to the given parameters. When running it shows “Abort” and hitting it will stop the squeezing. The
process can be restarted with the same or new parameters if it had stopped because of the Conversion Count limit or the “Abort” button.

The “Count” display shows the current record counter and the cound of found records to be converted (if possible).

The “Converted” display shows the count of thumbnails already squeezed in this run.

In the bottom right corner the current thumbnail picture is displayed in the original and then in the squeezed state.

In the bottom left corner the status display shows the SQL statement used to find the appropriate thumbnails and then for each thumbnail the name,
width, height, JPEG quality (in %), information size (in Bytes) as well in the original as in the squeezed state.

mschnell

  • Full Member
  • ***
  • Posts: 98
    • View Profile
TD4 Squeezer Tool
« Reply #10 on: 2017-05-13 02:27:36 »
Astonishing, that seemingly nobody else has this problem any more. It had been discussed here several times.

-Michael

Daan van Rooijen

  • Administrator
  • Sr. Member
  • *****
  • Posts: 933
    • View Profile
TD4 Squeezer Tool
« Reply #11 on: 2017-05-13 05:43:27 »
Is the tool finished and have you released it? Does it work well?
I'm volunteering as a moderator - I do not work for Cerious Software, Inc.

mschnell

  • Full Member
  • ***
  • Posts: 98
    • View Profile
TD4 Squeezer Tool
« Reply #12 on: 2017-05-13 06:56:53 »
Yep I would not publish a description unless it would be working.

A single glitch that I don't seem to be able to avoid is a database engine error that sometimes happens. It does not do much harm. Sometimes, after restarting the action the error does not happen with the the same picture and some more thumbnails are converted. Sometime the process always stops at the same picture. In that case same can be excluded (by deleting it from the gallery used by the process).

I doubt that it makes sense to do some "public" release. But if anybody is interested I will provide the executable on my Website (It should work on Windows > 7, no installation necessary.)

-Michael