Quick Search, Advanced Search: "OR" and (explicit) "AND" not working

[kinook]

UR is developed with Visual C++, and I don't see how a Windows update could affect searching within UR. The SQLite library is statically linked into UltraRecall.exe.

I'm still unable to reproduce the problem with AND and OR not working. Can you post the details from

https://www.kinook.com/Forum/showthread.php?t=3038

[Spliff]

My question re framework was because 25 years ago, .Net hadn't been there yet; I hadn't thought of MS tools before .Net; thanks for clarifying!

Sorry, my bad: The "stick version" (= situation e) = a fith one) is as faulty as the (newly and "from scratch") installed version, "new" items created on purpose weren't "found" since they didn't contain "or", as string, so "a or or", being processed in all 5 situations a)...e) as "a AND b AND or", wasn't found, obviously: after hours of trying, unsuccessfully, I then unfortunately tend to make even very stupid mistakes; a good night's sleep afterwards then clears the mind.

Btw, I tried with "dieses" and "jenes" (= "this" and "that", but certain to not introduce a "stop word" problem), and then, neither of the items just containing "dieses" OR "jenes" was listed as a "find", BUT when I then selected either of those two items manually, the "dieses" or "jenes" (in the text = content) was highlighted (which is only expected for search results, not also for the first item the user navigates to, after a search), whatever it was: if I selected the "dieses" item after the search (title = "v1"), "dieses" was highlighted, and if I selected the "jenes" item after the search (title = "v2"), "jenes" was highlighted, and vice versa: highlighted "search find" (but not also appearing within the search results) for the very first item selected after the search, whatever it was.

For a "dies?s" search, or for a "jen?s" search, I get some search results indeed, but with nothing being highlighted, since the whole content of any of those "hits" doesn't contain any string which could be translated to "dies[a-z]s" or "jen[a-z]s", ditto for searches for "dies[a-z]s" > "contains keywords" > LONG "hit" lists which don't make any sense, without any hit, neither placeholder- nor literally-wise, and then, changed to "matches wildcard" manually, no hits anymore - all this with fresh installs, not only after my "adjusting" the UR settings to my wishes.

The next step would obviously be to buy a third W10/11 PC.

(As said, OR works fine by AS in 2 independent rows, and I would have needed the ?-placeholder for "tag-pivoting": sometagA[a-z] being possible by searching for sometagA, but for searching / filtering by sometag[a-z]A I would have needed a "working" ?-placeholder, and, obviously, such "compound" tags would be (=for other UR users; would have been, for me) extremely useful for FILTERING-by-searching.

I then thought I could replace them by TWO tags instead, then searching = filtering for them by implicit AND:
sometag* someothertagA [1]
or
someothertagA sometag* [2]
= instead of
uniquetag?A
that is, and the * assuring the switch to "matches wildcard" will be made (it is made indeed).

Since 1 didn't work, I tried 2 (i.e. the * at the end), and that didn't work either, i.e. correct translation to AS, but no result, and:

When, in AS, I then click on the "Quick" button, I see that the QS search string has been enclosed in double quotes, so it reads now
"someothertagA sometag*"
(for both versions, 1 and 2, i.e. the * being "within" the string, or at the end of the string, does NOT have any implication).

Thus, it's obvious that the code which translates the search string, makes false assumptions, caused by the * here, or then caused by the translation to "matches wildcard", itself (correctly) caused by the *). (When I search for "sometagA someothertagB", it's found, and back to QS, it's without the "", but of course, we're in, and stay in, "contains keywords" then.)

Thus, even * does not work correctly BUT when it's only one single search string, with the * at its end:
sometagA* [that works indeed]

When the two tags are in the same line of the content, i.e. not in different lines, then
sometagA someothertag*
is successful ("matches wildcard" is done, and item is found), but back to QS, it now reads
"sometagA someothertag*" [= phrase search]

whilst (logically then)
someothertag* sometagA
doesn't find the item (which contains sometagA and someothertagB)
since, whilst AS shows, matches wildcard someothertag* sometagA
back-to-QS button brings
"someothertag* sometagA"
adding (unwanted) phrase search to placeholder/ wildcard use again, but possibly treating the * literally here because it's not at the very end of the (as said, unwanted) phrase search;

on the other hand, when I put
someothertag* sometagA
into the item content, into the same line = as "phrase", and expecting that "since" UR treats the * literally, that "phrase" should be found then, AS
matches wildcard someothertag* sometagA
does NOT find anything, albeit, back to QS, the search now reads,
"someothertag* sometagA"
where clicking on "Start" brings "0 matching items", and back to AS, it now reads
matches wildcard "someothertag* sometagA"
and back to QS, it now reads
""someothertag* sometagA""
and with this back-and-forth, I can multiply the "" at will, whilst it has never been intended as a phrase item to begin with!


I hope you can replicate (parts of?) my "a * brings unwanted double quotes (but doesn't treat the * literally anyway, and treats it as placeholder just at the very end of the forced phrase)" problem (which seems to indicate the search string might be processed faultily, internally)?

(Obviously, I closely verified my tag strings, and my search strings, in order to not make any mistake here.)

[kinook]

Again, please provide the info from

https://www.kinook.com/Forum/showthread.php?t=3038

[Spliff]

Since I only know my 5 systems (PC 1 before clean re-install and after*, PC 2 before clean re-install and after*, stick*; *=here at least, the thread-3038 data obviously is the "factory" one, cf. the "Trial" in the above screenshots' caption), I can't speak for other UR users' systems here, so with this reservation:

Before, control-end systematically (?) went to the very LAST row-AND-column (= "Value", so to the very last value of them all), then "typing" (even by macro, i.e. without previous F2) started to change the "Value"'s string value; that's why in the other thread, I suggested to put string values within the very LAST row of Advanced Search.

Now, with your implementation of my suggestion of some Discard / "Exclude" (yours is the better wording indeed) column, whilst I had had in mind "Exclude" as the very FIRST column (i.e. even before "Relationship"; but without specifying that, I admit), you added that column as very last one, so quick "value" input is excluded now, the user or macro have to fiddle around with navigation within the AS rows / columns, beyond a simple control-end, and, worse,

sometimes, the focus in the AS grid is STUCK within "Item/Attribute" drop-down, and then, NO key navigation whatsoever (and with any modifier key) will LEAVE that drop-down, and, for example, both "up" and "down", AND "left" and "right" navigate up and down within that drop-down's entries, whilst at least "left" and "right" would be expected to LEAVE that drop-down, and set focus to "Relationship" or "No", respectively, i.e. go to the adjacent column. (From one AS to the next, sometimes, the new focus is where it was placed on leaving the previous AS, but more often than not, it's re-set on the Item/Attribute column, and then, the errance within the drop-down starts...

This makes AS macros unreliable since the macro "thinks" it's in the grid's intended target column/row, while in fact erring within the Item/Attribute drop-down, and the user has no other way than to use their mouse, in order to navigate off that drop-down.


As for (and always speaking of my 5 systems only) the "Vapor-Func" (OR, *, ?, [...]) described above, to resume:

Even * is unreliable, sometimes it works, but triggering the switch AND phrase search, then again not, seems to depend not only on the place of the * but also on the search terms, so it's not usable, so none of them all work:

For OR searches, I need multiple rows in AS (with those new additional problems described here; I'll have to go back to a UR version before even Feb, 2024 (since not even the March version, but also the Feb version already have the "Exclude" column, complicating things);

ditto for tag searches, only xab (with x meaning, "it's a tag", a being the category, b being the value) works, but works indeed;

as said, implicit AND works fine, so I can put several such tags into the items, or rather into the titles, since that way, I always see them, i.e. know any item's tags without searching, and then some xab xch xgj (even Quick) search (implicit AND) will work, whilst for xab OR xac OR xah, I need an AS grid with 3 rows, so I make several ASs, with 2 OR rows, 3 OR rows, etc., and IF I hold it all within the very same indentation level, even tree order is preserved, so that can I work from within the search results, as I would otherwise do from within the Data Explorer (which becomes unrealistic by experience, considering that filtering (in tree order) is necessary, but that, given the fact that any item just can bear one flag, but several tags, whilst Data Explorer filtering is just possible by flags, not also by tags: thus, working from within the search result grid is necessary BUT doable indeed - just the practical use of the AS grid, again and again, doesn't come really that handy, to put it as "light" as it gets.

MyInfo(App)'s search input field, together with its what you might call "scope codes", and with its (expected, and working) parentheses to group when necessary, obviously are the way to go, since that's the nearest thing to the target SQL "select" code, AND slight changes / variations (which are complicated fiddling around with the AS grid in UR in many cases) are done almost immediately, be it manually by the user, or then by some macro input into that input field.

I admit that UR's input field, together with working OR/?/[...] (i.e. if those ain't Vapor-Funcs as in my 5 systems), is not that bad to begin with, but leaves out any "more-than-just-one-scope" which then need the AS grid... which can't be regarded as "user-friendly" as it is.

This is NOT to advertise MI/MIA here - its search results do NOT respect tree order (and miss most of UR's sort variants) -, so it's questionable if MI's better search input (!) will really be so much more valuable in the end, given its search output (!), but its "tree filter" (by strings = tag or any other word-start-string), is infinitely more valuable than UR's "tree filter" (by flags).

Thus, copying that functionality from the competitor would certainly be UR's most important enhancement in years, and seems to be doable:

Tree order (as in UR for tree filtering) is respected (obviously)
Find-as-you-type, i.e. "live" display of the results
space is "AND"
NO */?/etc
NO sub-strings, just word-start-strings (but not necessarily title-start-strings, and btw: otherwise no implicit AND would be possible), i.e. x finds xa, xar, etc., but e.g. ar will not find xar

JUST for titles, and that makes it possibly very much easier than with "strings-from-everywhere", since the titles are listed in traditional sql tables (i.e. no fts problems), can be indexed accordingly, and UR currently IS able to respect the tree order, by filtering by flags, here in the tree, so the same should be possible with title string bits.

AND, as said before, it should be done in a flat list, but without discarding deeper-down items (i.e. parent items may, or may not, fulfill the condition(s).


Information management being about information retrieval in the end, the most perfect retrieval possible should be the object.

And yes, the ostentatious "lack" of interest (or better: so they insist on simulating) into all these matters, from UR co-users, is so much de-motivating - why should they be given anything then they don't pay extra for, right?! - that I can only qualify it as "appalling" indeed... which spares me to have to use that nasty "pearls before ..." expression, yeah.


ADD-ON:

To clarify: 20 flags seems "not too bad", the number 20 divided by 20 "values"... of the SAME category, but once you divide that number by several (!) categories (!), i.e. color, italics, bold, underline, then combine those (e.g. bold instead of regular for one category, different values then by different colors; another category by italics, combined with the aforementioned categories, by regular/bold, by color for the category represented by colors, and so on), you find yourself with "almost nothing", and then you have to renounce on categories, just have enough possible values-in-all, i.e. 20, to combine two categories, one of the two with a strict minimum of possible values; thus, for categories asking for many values, or even for sub-categories, you are forced to resort to tagging by text-tags... and that will set you back to Search, forces you to LEAVE the "Data Explorer", and then, Search's functionality's problems get into your, the user's way... (That being said, just formatting plus colors for filtering bear an inherent number limit, different formats not being available in number, and colors not being discernable in big numbers by the eye... whilst (additional) text-tags (as explained above) don't come with such bounds but with virtually no number restrictions; thus for more detailed filtering, beyond very basic filtering for just one category, they are the better solution to begin with, or in other words, sharply multiplying the number of flags instead wouldn't have been but a pis-aller anyway. Oh, and I should have made this clear indeed: MI's tree filtering by text-tags (=as many such tags as you need), whilst being much better than UR's tree filtering by flags (just ONE flag possible per item, and just up to 20 all combinations combined), because of its lacking "OR" (at least), isn't that formidable in the end, so "just copying" the MI func couldn't then, unfortunately, be considered "state of the art" yet, and I think it's better that I say so myself, instead of blatantly inviting to some, "and when that's done, you'll continue to complain", unavoidable otherwise:

At the end of the day, working on a really good Search functionality would be so much more worthwhile / beneficial:
- (Alternative) Input by input field, with e.g.: t:some for title, c:some for content, e:some or just some for everywhere, and so on (=scope), parentheses just like in SQL (i.e. the SQL target string anyway)
- Tree preservation upon demand (= by option) and, in case, by re-ordering after the search if it's really that much more complicated to do from search start - from the NON-observation of tree order, within the search results from MI, RN (RightNote) AND UR, I infer that all PIMs have adopted the Adjacency List Model in its most basic flavor (whilst UR's items' titles are stored in three different locations, though: so much for strict redundancy avoidance), and that makes me think that even with some minor ALM enhancement (lots of such ALM-to-"Common Table" extension examples being available in the web), i.e. one with not-too-much additional "write" tasks, already could be some BIG help in refining the Search?

Also, let's face it: On the "Mac" side, tools with "Search Results with the (first-per-item) relevant paragraph" are available; RN offers "n words (!) before and/or m words behind the search term", which makes the results more or less unreadable, since the n before and/or the m before is always much too few or much too many, and I have not found any word wrap, so, and...

to share at least immediately constructive idea here - goodwill! - UR users are invited to put - and, if necessary, even to copy! - the core info (incl. text-tags if they don't prefer to put them systematically into the titles) of any relevant (!) item into the very first paragraph, then browsing through the search results (by up and down) will present that core info much faster, than by having to then, one-by-one for (almost) every "find", SCROLL to the core parts, or rather, those parts where the search string(s) are to be found; I suppose it might be possible though to introduce automatic scroll up to the first "find" in UR instead (i.e. screen display would start with the start of the paragraph which contains the "find"), upon option (i.e. of just highlighting the finds)? (Alternatively, the user might use a big screen, with full-height content pane, AND and large search results pane on its side, so as to hopefully see the relevant parts even without having them on top and/or having to scroll down to them - the success of this strategy obviously depends on the user's average item text length...)

And the user might also display the "Item Text" column, since then, a mouse-over will - fast! - display the "full" text (i.e. as far as there is room for it) within some fly-over pane, albeit in a quite tiny font size, and without formatting, the latter being perfectly understandable since obviously, that "quick-n-dirty" text preview is retrieved directly from the sql table column... which then should make it even less helpful when you store the plaintext of your items in all-lowercase (writing from memory here, not having tried out recently) - be that as it might, font resizing by user setting should be easy to implement, to make this goody finally worthwhile, but many user will probably not even have discovered it in its current form? So they're informed now, but as said, the kind / lack of "interaction" with UR co-users here makes it ultra-difficult to consider them "fellow UR users", which obviously would be the ideal.


Oh, and - OT here - I "upped" my "move-to" script by (after the obligatory ^x, obviously, then) first inserting a {down}, then only have it make the navigation to the target(s) (if there are intermediate targets, too, you can count those, then adjust the number of necessary "Go - Go back"s accordingly), then do the ^v, and then do (at least one) GoBack (shortcut at your leisure), and, heureka, you can do other moves from where you left before the current one; the problem here being that this just works if you do the moves one-by-one, even when they go to the very same target, OR if at least the very last one of the items you will have selected for moving, is not immediately situated above another item, to be moved (and thus, if you move some "block", pay attention that you select the "bottom" item as last one), since otherwise, your "GoBack"(s) will stay within the target area, not go back to the "area of origin" (similar if you send an {up} instead of the down-key) - such being the sudden problems you encounter once you try to put some "outliner" to "power use"... (And, think on it: even knowing the number of the selected items - which can programmatically be retrieved indeed: it's an element of "window text" in case -, will not help since the {down} (similar for {up}) will not work from the visually-"last" item, but from the last-selected one, and if that's within a block, contiguous or not, you'll end up "somewhere"; ditto for copying items, obviously.) - In other words: From the "outside", "only so much"* can be done, at the end of the day, whilst constant optimization of code can "do it all", and this example is just a blatantly simple one... (*=meaning here: if you remember to select the "bottom" of several selected item last, it works as expected, except of course if you work from end-of-tree, but then, a simple {end} after the move will do) - You see here that co-users might have gotten lots of additional goodies if they had made just and strictly even minimal efforts onto what you might call "collaboration" (and just another example: within "global text replace", you must insert some "trick" at some position of recurring part of the script, and then, global replace "from the outside" works fine, but nada... not even one single "thank you", let alone adding / sharing your ideas, hints: nada: Third-party UR users, you call that "constructive"? Wanna make me laugh? ;-) (And when then, on top of that, I pile up lots of problems with the software I can't resolve, for my own needs, I'm more than just fed-up... understandably, I might dare say.)

[cnewtonne]

Spliff ...

What I have found to be most effective way of reporting issues is to ...

1. Use a numbered list of reproduction steps vs. long paragraphs & pages.
2. Describe steps using the UI command words. Use same terms the app is using.
3. Use screenshots and recordings if you want to show reproduction steps/errors.

In this thread, you have described a usability issue using 6,526 words. Using default MS Word settings, these are 13 pages! Still, it is hard to understand what you're describing. I thought I understood you from the very first post, but you lost me later.

Software is science. As such, it is concise and is described in exact steps and words. Make it easy for yourself and your fellow users by adhering to these simple steps, please!

Thank you very much.

[Spliff]

( QS/AS being QuickSearch / Advanced Search, respectively, and obviously: )

A)

In practice, it's NOT possible to "flatten" the hierarchy in UR, since:

a) compare this construct:

'a...
____.b...
________Something:

.b has 'a (or more precisely: \'a) in its lineage, and
Something has both \'a... and \.b... in its lineage, and
Something, and also anything within the possible sub-tree of Something, can be found by QS (triggered from 'a... or from .b) Something (with "LS" checked), "LS" being the checkbox "Limit search to selected item in Data Explorer pane", in other words, "Local search", i.e. "just in the current sub-tree";

b) with this construct:

.a - ...
.ab - ...
____Something:

even .ab (which corresponds to above's 'a .b, obviously) does NOT have .a (or then, \.a) in its lineage, all the less so would Something have \.a in its lineage, thus:
triggering, from .a, an "LS" QS search, for .ab - ... or any of .ab -...'s child items (including Something, and any of Something's child items, and so on), would fail; you would have to do a global search instead, triggering lots of unwanted "finds", in case.

Obvious solution to this problem: You build an AS instead (if the * works on your system that is), with:
- item - contains keywords - Something (or whatever string the "target" item may contain, in its title, content, "Notes", etc., anywhere below .ab in general, or below Something in particular)
- AND - Lineage - matches wildcard - \a* (or \ab* if you want to do an even more specific search)

Thus, with strict technical (=indentation-level) hierarchy only, you can avoid AS, whilst with just a "virtual" hierarchy", .a and .a[a-z] being placed onto the same indentation level, you will need the AS... which is a nightmare if you do perhaps hundreds of "searches" (i.e. "greps", filterings) a day (which is my case).

B)

But this "flattening out" the hierarchy in UR would be of interest indeed, considering that:

Whenever you want to [/B]navigate[/B] to some "item" (i.e. some "inter-title", prefixed with some "marker", as the ' and . used above), it's evident that a simple entry .a, or then .ar will be oh so much more "easy" - if those "."-entries are immediately below your source item, i.e. in level 1, than to navigate first to some entry 'a, then to its child item .b (without knowing if item 'a has already been expanded or not)... and then, if you are mistaken of the existence / naming of an item .b "within" parent range 'a, and if some other range 'whatever is expanded at the same time, your macro will "target" the .b... inter-title within, let's say, the 'c... parent range, since your macro will use the in-built tree-navigation, by "targeting" the "very next" .b... item, be that within its intended range 'a..., or then within the 'c... range if that one is expanded, and some .b... item is existant within that range...

And yes, you can then, i.e. "after the facts", trigger a "UR command line" retrieval, and then check if your ".b..." find (or further find below .b...) has a \'a within its lineage, but that costs 1,000 or 1,500 ms more (since your macro will have to retrieve, by clipboard, that "UR command line" first, before being able to analyse it), and this procedere cannot be called "elegant" or "smooth" in any way...

C)

On the other hand, within the file system, such macros are without problems, since in both situations, a) and b), you just work with * (i.e. a*\b*\*/something or then, ab*\*/something) and thus you have free choice between the paradigms "deep" or "flat", and both procedures work fine, by any professional code: I do both, i.e. all four, both in AHK and in Everything, the * being a placeholder for some sub-routine in case but which, on a modern PC, just takes some ms indeed, and NO markers like ' or . being needed, since the \ will lead the code, whilst in a db like UR, the LEVEL is ambiguous, i.c. can not be counted fast by the number of \ within the path, and then, the (navigational, move, copy, link) TARGET is specified by that path, and reached immediately, whilst in a db like UR, by scripting from the outside, it will have to be reached, step-by-step, and time-consuming path verifications being necessary at every such step then.

D)

Thus, yes, in DBs like UR, and especially in such DBs, a "flattened" hierarchy is largely preferable (since it does away with the need for "command line" verification, "is the target .b... really a child item of intermediate target 'a...?", all these not being needed within a file system navigation / move / etc), but, frankly spoken, UR's AS "costs" you several seconds "on top" of what would have been necessary, for every single search, and thus, if there was available some ENHANCED search box, as in MI and other software, and which would allow for entering
l:a* Something [l: = "within lineage", or better: p: for "path"]
or then, if you want it to be more precise:
l:ab* Something

this would not only speed up searching / filtering enormously, but it would also spare you to first navigate to the "parent item" in question, then check the "LS" (see above), but you could trigger this QS (as you could trigger the respective, but enormously time-consuming AS now) from anywhere, and a search "from current item", i.e. within the sub-hierarchy of that current item, could be triggered by:
l: Something
i.e. without placing any value behind the l:

And similar for any other QS:

Kyle, could you please consider such an enhanced Search Box, but with publishing a list of all its codes you consider to implement / use in it, in order for allowing us to have a look upon them, and to give our view?

And yes, such a system would sacrify the possibility for searching for some:some, but then, nobody really would need to search for some:some, except if they had used that string-form for tags, and why couldn't they replace those tags then by some.some indeed?

And yes, the some before the : should be as short as possible (hence my suggestion to publish them here, before implementation, or then, why not invite us to make such list, open to suggestions then, from the current "Item/Attribute" drop-down):

pt:some (always without any space around the ":") would be parent-title:some, and
pt: would be parent-title:current-item's-title, as any empty value after the : would be "current-value" wherever it makes sense.

The necessary "core" code for such a UR Search Revolution had already been implemented, by your transcription from AS Search grid to SQL string, and, frankly, UR's current AS Search grid is a nightmare, costing an additional 5s even when applying a script, and costing minutes for manual AS search: the absence of a FAST entry for the variable part(s) of those "stored searches" being the main problem, the "grouping by indentation instead of by parentheses" being the secondary one...

and the good news is, twofold:
- user's parentheses within the enhanced search box could be transferred 1-by-1 to the sql code (i.e. no additional coding work needed), and
- the current AS grid could be preserved without any additional effort, for people who prefer that, well, "very original" (and very time-consuming), way of searching. ;-)

[kinook]

Regarding the original topic about AND and OR searches not working, I did find a bug. There is logic to support case-insensitive Unicode searching (see SearchNonAsciiTextNonCaseSensitive), which is enabled by default, and this logic lowercases the indexed text and search terms that are entered, and it was incorrectly lowercasing AND and OR in the search phrase, which prevented the FTS search engine from treating these as AND and OR search clauses. This has been fixed in the latest download (v6.3.0.14). A workaround is to disable that registry setting if you don't need case insensitive Unicode search capability.

Regarding the other topics discussed here, can you boil this down to plain English? I am not quite following what you wrote.

Note: Updated in v6.3.0.15 to also handle NOT and NEAR commands.