Kinook Software Forum

Go Back   Kinook Software Forum > Ultra Recall > [UR] General Discussion

Reply
 
Thread Tools Rate Thread Display Modes
  #1  
Old 03-26-2006, 10:53 AM
kzerba kzerba is online now
Registered User
 
Join Date: 02-05-2005
Posts: 11
importing many emails

(1) converted ~2500 thunderbird emails to .eml files (time: ~10 min)
(2) imported into UR (time ~4 hrs)
(3) deleted same emails from UR (time ~ 4 hrs)

(2) and (3) pretty required not doing anything else on this computer during this time (IBM T30, 512 mb ram, WinXp SP2)

Is there any way to make this kind of operation faster?

kzerba
Reply With Quote
  #2  
Old 03-27-2006, 05:05 PM
dasymington dasymington is online now
Registered User
 
Join Date: 06-15-2005
Posts: 287
It would be good if it could be faster. I find even importing one or two .eml files can take up to 30 seconds or so. It must be because UR has to do a lot when it's importing - keywording, etc.

Presumably you'd only import such a large number initially and then you'd just be topping up with new messages, and normally you wouldn't be deleting so many at a time.
Reply With Quote
  #3  
Old 03-28-2006, 06:40 AM
kinook kinook is online now
Administrator
 
Join Date: 03-06-2001
Location: Colorado
Posts: 6,003
There is a fair amount of overhead to parse each message (and attachments) for keywords and other info, and supporting undo/redo also adds overhead. Some things that could speed up importing would be disabling the options under 'Tools | Options | Import (More) | File Import' and/or similar options on the Import Wizard (depending on how you are importing). Of course, you'll lose some functionality by doing that. Encrypting your database also adds some overhead.

But that averages out to about 10 seconds per message, which seems high, and 30 seconds is excessive to import just one or two. How large are the messages being imported? Are the databases located on network shares or local paths? What kind of hardware (CPU & HD) are your systems? I have an AMD 64 x2 Dual Core desktop w/ 2GB RAM and RAID0 HD, and I averaged about 3 messages per second to import 350 messages (ranging in size from 1K to 175K) from Outlook Express to a 100MB unencrypted database on a local drive (and 15 seconds to permanently delete).
Reply With Quote
  #4  
Old 03-28-2006, 02:38 PM
kzerba kzerba is online now
Registered User
 
Join Date: 02-05-2005
Posts: 11
Some of the 2500 emails were large. Email is one of my biggest headaches and I was experimenting with possibly scaling up in use of UR to manage my emails more effectively and reduce the headache. I don't want to lose the functionality of the import options to enhance the speed of the import.

Import context: IBM T30 laptop, Pentium M 1.5GHz processor, 512 mb ram, 37 GB hd with 7 GB free, local paths, no encryption, urd file size 32 mb.

To make an import comparable to your 350 email example, I selected emails < 200 kb from the 2500 and then took a representive sample of 350 from those (size 1-192 kb). This import took 15.5 min or 1 email every 2-3 sec, which is closer to your result.

So, size does matter, but so do the obvious things such as processor, ram, hd speed, etc. This experiment has taught me some things that I will consider in deciding how many emails at one time, which sizes of email and when to import to UR.

One thing I noticed is that if I import as a folder with the emails in the folder, UR always leaves one email in the folder that was imported. If I import as files just selecting the emails from the folder all is fine.

kzerba
Reply With Quote
  #5  
Old 03-28-2006, 04:50 PM
dasymington dasymington is online now
Registered User
 
Join Date: 06-15-2005
Posts: 287
Sorry for hijacking the thread; it's about importing many emails whereas I was talking about a few emails at a time. UR does import messages faster when they're in bulk, as you can see from the figures below.

My "30 seconds or so" was a bit subjective: a measured test shows that importing 3 .eml messages (4.30, 7.12 and 2.12 KB respectively) took 20 seconds. This is dragging the messages from Explorer and dropping them onto a folder in UR. The UR database is on a local drive. The PC is a 3.2GHz Pentium 4 with 1 GB RAM. The local partition on which the UR database resides is 97 GB with 60 GB free. The UR database is 655 MB and unencrypted.

Doing a bulk import in the same way for 60 messages (average 3 KB), took 1 min 30 secs. So that's about 6 seconds per message when only a few messages and 1.5 seconds per message when there's a larger number. No doubt the average time would drop if there were 100s or 1000s of messages.

Six seconds a message is perhaps not much, but it can seem a while the way I work, which is basically to store all important message in UR at various points throughout the day as I deal with them.
Reply With Quote
Reply

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



All times are GMT -5. The time now is 02:59 PM.


Copyright © 1999-2023 Kinook Software, Inc.