Any way to create a compound (flag) search preserving the tree structure? Or in tree?

[Spliff]

a) Within the TREE itself,

I can HIDE several flags, so that in the end, in theory, I have just a tree with the flags I want to appear. I can't hide regular (non-flagged) items though, so in order to realize what I need, I first have to assign a special, "regular" flag to all regular (non-flagged) items, then hide that "regular" flag, too.

It goes without saying that this comes with lots of side-effects, since your default item would now be a specially flagged item. Also, all this is entirely manual, since there are no STORED hide sets; a little menu with some 5, 6 (or even 12) entries for such sets would be welcome, with entry 1 = "everything is displayed", entry 2 = "hide flags 2, 4, 7", and the like.

This being said, it would be, or have been preferable to have such sets not in the form "hide flags x, y, z" but "SHOW flags x, y, z"...

BUT at the end of the day both versions would be practically the same IF the format "NO flag" (= regular = default item) could be INCLUDED in the flag list(s), be it for "show" or be it for "hide", in other words, in the (currently unique) flag list (currently for hide) it's the "NO flag" which is missing, so cannot be selected for hiding.

Adding this feature, "hide No-flags", within the unique hide-flags list, might be quite easy to implement? And, as said, some different such lists (sets), available by a menu, for different "views", would be welcome.


b) Within the SEARCH (compound searches, stored searches),

it currently seems impossible (?) to preserve the tree structure; several times in my 2 years with UR, I have tried, for hours, but I always hope I have overlocked the necessary "trick", or then, would it be possible to implement some system attribute giving access to tree preservation? (Of course, I ask for b) since current a) solution is so awkward, so if a) could be amended, b) would not be so important anymore, or then the other way round: if b) is / could be made possible, a) is not so important anymore.)

We now know that for a SIMPLE search, preservation of tree structure would be realized by sorting by lineage, then by tree order, e.g.
Item Title // Lineage (sort 1) // Tree Order (sort 2, means "child number") // other columns in case

This FAILS for compound searches, also when I try (perhaps mistakenly) with additional columns (IsObject, IndentLevel, or others):

Let's have a (stored) search for
FLAG = Red (for construction elements in different, higher-up indentation levels)
OR (which means "and" in the sense of "show items of either flag value")
FLAG = Green (for specific child/grandchild... elements for which I want to see the "distribution" within the structure)

Even in this really simple compound search the tree structure is NOT preserved: the "green" elements, by tree being direct or indirect children or grandchildren, etc of the "red" elements, do NOT appear "in place", even when there is NO missing = filtered-out element in-between, in other words, even if the green element is a direct child of a red element, it's not displayed directly beneath its parent but elsewhere = further down, and similar for the "construction" = red elements, which seem to be sorted by indentation level in-between themselves, and even some green elements appear in-between them IF their indentation level is high enough to appear in this upper part of the result list.


Of course, the b) SEARCH solution is of high interest here since they can be stored = be easily accessed then, and filtering out even regular (non-flagged) elements is a no-brainer here, whilst the (in theory, ideal) a) TREE solution (where filtering preserves the tree) currently has no such "stored" procedures (filter presets), and presents the additional "fail" of regular (default) elements not being filtered out even when necessary (interesting though, the filter code within the tree functions the way I would have expected it to work in search, too). But there hopefully might also be a current UR search solution, adapted to UR's specific search code internals?


EDIT

This might all be a big misunderstanding, since even a single flag search does NOT preserve the tree structure, with
ItemTitle, Lineage (sort 1), TreeOrder (sort 2)
listing all shorter lineages before longer lineages (i.e. ordering by indentation level even when that column is not even selected from within the "Choose columns" dialog; for finding elements (records), I might have not discovered it before, just assuming
ItemTitle, Lineage (sort 1), TreeOrder (sort 2)
displayed the search results in tree order (meaning: preserving the tree order from the data explorer), but for my needed filtering tasks, the fact that the structure of the tree is in part scrambled becomes obvious to the point of the result display not being of any use (the above described combination red-green becomes somewhat "better" visually (since then red and green are so much interwoven that you could think at first sight that it's all in tree order, but it's not, not at all) if I sort by TreeOrder (which in fact is "SiblingsNumber") first, then by Lineage, but it's just scrambled otherwise). Is there something wrong with my UR installation?

I am perfectly aware that for SIMPLE searches, the above does not correspond to what had been said in
https://www.kinook.com/Forum/showthread.php?t=5758
which shows that for me it's been a problem for more than 2 years now, and in
https://www.kinook.com/Forum/showthread.php?t=5462
which shows that it had been a problem for others even before.

So after 5758, we all thought that the problem, for SIMPLE searches, had been resolved. For COMPOUND searches, though, I then had thought this was a new problem, not having been treated before, whilst I now discover, on my system, that even for simple searches, the problem persists (btw, ALL ordering always in the way that the triangle shows UP (which means ABC), not down (which would mean ZYX)):

simple search for red flag, sort first by lineage, secondarily by "treeorder" (numbers 1 and 2 clearly visible, so I don't just assume but see on screen).

Real Tree Order (with indent levels):
0 (1) (first level below source-item)
p (2)
c (3)
z (3)
m (3)
y (3)
1 (2)
2 (2)
3 (2)
4 (2)
x (2)

Search result:
0 (1)
p (2)
1 (2)
2 (2)
3 (2)
4 (2)
x (2)
c (3)
z (3)
m (3)
y (3)

(For compound searches then, the (faulty) search results are as expected from the above simple-search results, but I now think that if the problem is solved, for simple searches, the compound search will produce flawless results, too.)

My data above representing (sic!) real data, i.e. longer strings, but with, in the real (!) tree, any greater indentation number (all 2's after the 1, all 3's after the 2) being a real child of the preceding lesser number (the 1 above them; the 2 above them), meaning the respective substrings before the respective "/" in the real lineage strings are identical, in other words, I just shortened the real data, but the example comes from real, real-world data.

[Spliff]

(Can't edit / add anymore to the above)
New (default) db, just 4 items. Obviously, our assumption (after the 2 linked forum threads above) it could be that easy, was wrong, but we just didn't notice, whilst it becomes evident with (but is not caused by) compound searches.

[kinook]

In the latest download (6.3.0.7), an option has been added at Tools | Flags to hide unflagged items when not showing flagged items.

Regarding fully replicating the tree content in the related items / search results pane, the available columns don't provide enough information to accomplish that.

[Spliff]

Well...


1)

I know /assumed there currently is not enough column info, that's why in-between I had suggested adding a new column; this was visible for some time here, but was part of a part I later edited out, I see now; I've been (and am) aware the necessary code will be very compact AND very demanding, the respective SQL is highly complicated (as I have known by my efforts to write such code myself, I just got to 90 p.c. of the wanted result); but then, since you do it (almost, see below) right in the tree ("Data Explorer"), it obviously can be done, albeit not on-spot.

I made a mistake by saying to do it within the tree itself was "ideal"; even before trying, just after reading you, my mistake occurred to me: In fact, we need the filtered-tree ASIDE to the tree, not in place of it, why? Because from the filtered-tree, we see what we (might) want to do within our tree, thus, if we have the two in two concurrent panes, we can toggle between filtered-tree, for (further) analysis, and tree, for doing changes;

technically, the same is true if we switch forth and back within one (the tree) pane, but if filtered-view, we (regularly, not systematically) need the complete = expanded (but filtered) tree, and thus the toggle "View - Show flagged items" will then show that complete, expended tree > "lost in hyperspace" > lots of searching / navigating after every one such toggle-back;

also, an exclusive (sic!) tree solution, = a) above instead of b) above (instead of both solutions, eventually), represents the inconvenient that tree preservation would only remain possible for filtering by flags, but not also for filtering by other criteria, e.g. "tags", e.g. some tag codes within the titles or elsewhere, or any other search element; it's obvious such functionality - tree preservation for ANY search / filtering would be more than welcome (all the more so since even, see the two links above, it was asked for by other users already in 2017, and assumed by all of us as "being there" while in fact it was not; it's just that currently, me trying a combination of two flags, one for the "construction" and the other one for "content specifics", it's become clear as day that the solutions we all believed in, in 2017 and 2021, don't give the results we're after.

Also, see 3) below in this respect, since in tree-preserving filtering, we will have problems within the tree which are absent within the (flat) search result list.


2)

Thus, I suggest to create the necessary code for tree preservation in searching / filtering, perhaps even then as an click-box, overriding the current column sorts, working as a toggle then (even if for the toggled view, the search has to be started again).

Btw, the new column could be a (= another) virtual one, and just be filled (in work memory) in cases the toggle "preserve tree structure" is set to "on", so as to simplify (and fasten) the proceedings when it's not needed - since in that intermediate SQL "view" (behind the scene), some tree column 1.1.1.1, 1.1.1.2, etc. would be necessary to be built I suppose.


3)

"since you do it (almost, see below) right in the tree" - why did I say that? Because I had assumed that a flag-filtered tree would show ALL NON-filtered items (I called them "elements" above, I mean "items", of course), and that's currently not the case in your current solution a) (I have checked.).

As I have already mentioned above, in the FLAT tree-preserving search results list, we don't encounter any problem to display ALL NON-filtered items, we just leave out the filtered items.

Otherwise in a filtered tree (which preserves the tree structure visually): Here, we have 3 possibilities (as the designer / developer):

a) We can simply leave out non-filtered items, too, if the necessary parenting (sub-) structure is filtered out currently, and that's been your current, quick-n-dirty "solution", which unfortunately makes the filtered view unusable, since a filtered view is about prominence (sic!) of the non-filtered items, not about aleatorically hiding them with the rest I'm afraid.

b) We can leave any partial substructure (= the intermediate-level parenting items), needed for correct tree positioning of the non-filtered items, untouched, perhaps (just for the screen representation) "formatting" them in light grey, and of course, the caption would become ("Data Explorer - Filtered View"); this obviously is, code-wise, the most elaborate solution (and would also imply that all un-necessary siblings of those intermediate parent items would be filtered out, of course).

Of interest here: According to the situation, this very elegant solution would not necessarily be the user's wanted one since if the "intermediates" are too numerous, they would possibly make necessary a lot of scrolling, and then, the question arises, both on the developer's as on the the user's side: To what "degree" should the "intermediary skeleton parts" be preserved? At least somewhat, at the higher-up levels, they would probably say, but then, whenever most of the non-filtered items are within some sub-structure, it would become evident that the "in-betweens" especially within that sub-structure should be preserved...

In other words, this would become highly-complicated stuff, for at the end of the day, little effect; of course, the only realistic (sic!) solution b) would be to implement a toggle (sic!), between "complete b)" (= show ALL necessary "intermediates" (in light grey) for and together with the non-filtered items), and c):

c) Even in the "Data Explorer" = tree pane, show a filtered flat list: easy, and also sufficient in itself, albeit a (toggled) combination with b) would be "ideal"...


4) BUT we've got additional PROBLEMS here, for our "tree solution", both for the developer as for the user:

How much "editing" will be allowed in those "tree solutions" a, b and c (except for simple renames that is: moving items?)? A code nightmare, AND a perfect occasion for the user to definitely (even if just partly) scramble their tree, even if the code is totally correct.

And that brings us back to the tree-preserving search/filter solution, b) on top: Concurrently (sic!) available: the (tree-preserving) "flat list" search results (and here, as in the "tree solution c", just the non-filtered items, but all of them), for deciding WHAT to do, AND the tree (in its current, and malleable, collapse-expand state) for DOING what we've decided to do, and whenever we then decide the two panes have got too much out of sync, we simply trigger the same search again.

It's clear as day that THAT would be a smooth workflow, AND it would not just apply to flag filtering, but to ALL searches / filterings.

The current "tree solution a) a)" (which even hides non-filtered items) being unusable as such, and the alternative "tree solutions" a)b) and a)c) coming with their problems of their own:

Please, Kyle, implement the necessary code for optionally preserving, by sort-overriding toggle, the tree structure in search results lists!

Here again, some might say, "not needed!", but as I have said, again and again, UR might NOT only be a "data repository", as which it's perceived by most of the general public, but could be the very best "write app" out there (Macworld included), and that would apply to ALL sorts of writing.

[kinook]

It's not clear what you are requesting. Can you explain it again, in plain English, with fewer words if possible. It seemed clear what you requested previously, but now you're saying what was added isn't useful. Thanks.

[Spliff]

I try to do a summary here, numbers as above:


1) TREE filtering instead of search/filter is not ideal (my bad in my first post) since user (and/or developer) would get into trouble by editing the filtered view, so no editing but endless forth and back between filtered view and complete view, and the complete view is the whole, expanded tree, so navigation problems in order to do the wanted edits: user's work would be possible but very cumbersome.


2) Thus, I suggest to add the necessary code to SEARCH. Can be done by an intermediate sql "view" with (virtual = just here, not stored) numeric path column by numeric path representation, then sorting by that (1.1.4.1 after 1.1.4 and before 1.1.5, currently it would come after 1.1.5 which breaks the tree preservation).


3) I present 3 ways of doing TREE filtering (see 1), a, b, c:

c being a flat list (possible if you really don't want to do the SEARCH thing (see 2),

b would be too demanding code-wise since for also graphically maintaining the tree, you would need to also show all "hidden" items (in light grey) but which would be necessary for the graphical tree representation of the non-hidden items

a (which you currently chose) is unacceptable since some non-hidden items ARE hidden though, since they are children / grandchildren / etc of hidden items (which are left out), so if you don't want to do c = flat list but graphical tree representation, you can't left out non-hidden items but must also preserve "hidden" items to "hold", to "situate" the non-hidden items = solution b = lots of code

To make this as clear as it gets: Currently, you do a = seemingly preserve the tree but hide "non-hidden" items which "depend" from hidden items, so it currently does NOT make sense, example:
hide everything except red and green
a is red, child b is green, you correctly show both
but a has another child, c, which is black, and which has a green child d on its turn;
you hide the non-to-be-hidden item d since you hide its black parent c

Hence the (complicated) solution b, or the (simple) solution c, here in TREE solution.


4) BUT as said, solution c in the tree (= flat list) would correspond to the SEARCH solution (from which you refrain because of the necessary, demanding coding), AND I say, since it's the similar code anyway, in the SEARCH results solution AND in the TREE solution, please do it (= the flat list, preserving the tree structure) per SEARCH results, since then

- the user can work concurrently on BOTH panes: for their deciding what to edit / move / etc they would consult the search results, and for their edits / moves / etc they would have the tree available in the state they need it to be for these edits, moves, etc

- the user can use ANY search for this work (e.g. searches for keywords, tags, whatever) for this editing-and-shuffling work, whilst if you do the flat list within the tree pane, this would only apply to flags, so the "usefulness scope" of filtering-while-preserving-the-tree-structure would be greatly enhanced, compared to the flat-list Data Explorer solution (with no extra developing work compared to the latter).


As said, "TREE a", your current "solution", is not a solution, since it hides items which are NOT to be hidden, just because they have the "wrong" parentage (which in a flat list would be easily hidden indeed).

So, "TREE b" (= also show the necessary "in-between" items) is complicated and not worthwhile, "TREE c" (=flat list in Data Explorer) would be possible but not easier to code than the flat-list "SEARCH" solution, and quite awful for the user, hence my suggestion to implement a SEARCH OPTION "preserve structure" (or "preserve hierarchy") (in the flat list, not also graphically of course), as another checkbox in the pack "Search titles only", "Match whole words", etc., and which would OVERRIDE any current column sort settings.

(I also said the additional code, with the creation of the intermediate SQL "view" from which then the "hierarchy sort" is done, should, of course, only triggered whenever the option "preserve structure" has been set for the search / filtering.)

This would then be available for ANY search / filter, not only for flags.

I hate to say it, but the developers of RightNote and MyInfo both implement quite nice features on a quite regular basis (especially MyInfo, whilst RightNote lately added the option to have TWO, concurrent Data Explorers, for alternative views, and which ironically, if we had them in UR, would invalidate my reservations re the TREE solutions (Tree b and Tree c, whilst current Tree a cannot be), whilst on some core functionality NOT being in the class of UR, so it's not a bad idea to ENHANCE and develop the STRONG points in UR's interface and functionality, unparalleled over there, and structure preservation, even as a default, might be considered a core functionality anyway - by which I don't wanna say that alternatively, all the numerous, but all structure NON preserving columns sorts weren't needed.

Again, "do it right" instead, in the Data Explorer, would be possible, but not less work than to put hierarchy preservation as an option into the search... which would be so much more useful for the user: much better workflow than Data Explorer toggle, and availability for ALL searches / filterings.