File Manager - misc finds

Just three misc things that I found while going through File Manager.
Even thou you locked out System Directory and you can’t get inside with a click, I still could just type in “System” in address bar and it would open it.
Directory “token_pools” within System is open for read/write while others are not - possibly an oversight
When I create a hidden file with help of a terminal it still shows up as “-hidden-”. Attempt to open it with editor gives me “invalid path” at magic button prompt, but also prevents me from opening any other file. I have to press sync. After that, things work fine until I try to open hidden file again.

That’s it for now, cheers.

1 Like

No option for edit anymore so I’ll just include it in the reply:
New “Open with” option via right-click displays a modal properly, but when I click away without selecting anything, subsequent right clicks do nothing - sync fixes that
When I try to left-click open *.html file or ID file it instantly starts the UIAppTemplate with App that was ran in it - Hello World since it was the only custom App I ran.

1 Like

I cannot recall anyone disabling access to the System directory from within of the File Manager UI dApp. Looking into that…


Yeah… it seems like there is some sort of a bug, seeing those strange characters in tooltip once the System directory gets hovered?


and indeed, I’ve included some fix-ups in JavaScript eliminating empty \u0000 characters from the end.
@Major you might want to take a look why these characters are being sent from full-nodes to the UI in the fist place. Marking this point as resolved.

I have disabled the timeout after which it was impossible to edit a post.

image
That is now confirmed. I will try to assess this further, @Major , requesting your assistance.

Further it seems that particular Token Pools in the System directory also have invalid permissions assigned.

That stays in contrast with Identity Tokens which seem all fine:
image

So, I’ve found the following code fragment:

Looks like some ‘system-folders’ are deployed during initial bootstrapping, within of the System state-domain. These ‘special’ folders have a system-only security descriptor assigned.

Thus all sub-folders and files created within these folders inherit these privileges.

Seemingly ‘token_pools’ directory is not on the list.

I vote for having all elements within of the System State-Domain inherit ‘system-only’ security descriptor by default.

Yet again, @Major - your assistance here would be highly appreciated.

While we both know that the very reason for you to be summoning me from around here is to see whether I’ve stopped using Signal.

I know we have been transitioning towards Open-Source and of course I do appreciate.

Now, repercussions stemming from this bug-report have been boggling my mind since yesterday.

It is not the very ‘bug’ or a bogus scenario with the very folder at hand as the reasons are obvious - someone forgot to set appropriate security descriptor onto the ‘token_pools’ folder.

While I have already implemented a ‘dirty’ work-around, I am not satisfied by it.

The work-around checks whether the file / folder accessed resides in System state-domain and sort-of returns a falsified security descriptor.

I need to do lots and lots of additional testing, within of my mind, before I get back.

Before I proceed any further I want to be assured that there’s a Working back-up of the public test-net @vega4 .

A backup copy has been prepared. Green light to initiate an approach.

Ok folks so me and @CodesInChaos, we’ve been playing around like a lot researching the current state of the file-system and seeing into what could be done to aid not only this bogus situation but any future situations resulting from the fact that… we’ve been having little experience with modifying directories.

When you think of it, we’ve been adding files, removing files, changing files… I thought that some logic for modification of directories themselves was missing I even began introducing some changes to code responsible for the file system… all the mechanics based on Merkla Patricia Tries right in front of me… and I was scrapping my head on how to restore images of the sub-items when the folder-lead-node gets modified.
So I wrote code that that would make a backup copy of all of the images /hashes of the sub-elements… it quickly occurred to me that it was not needed to be done manually since like two years ago we have implemented some crazy cool copy constructors and stuff…

so I was like ok… make a back-up of the Merkle-Patricia root node, meaning the once within a specific directory; replace the current directory and transplant back the root-element from the previous directory.

I wrote all the code.

I was about to test it.

And then all of the sudden @CodesInChaos pointed out that this logic prevents sub-folders and files from being deleted since the code would always restore the previous.

So I ended up deleting …err… uncommenting all of the changes.

Leaving some in-code rationalization for posterity.

it follows.

/*
		[Theory - Modification of Directories] - BEGIN

		-----------------------
		Update 16.07.22
		What follows is not needed.

		Reason? Had we implemented what follows it would not allow for deletion of files and sub-folders.
		All it takes for folders to be explicitly modifiable is for the exact to-be-replaced-with instance to be provided to
		this very function ( addNode() ).

		The drawback is performance, since the current version of the node needs be queried for beforehand.
		That apriori retrievd copy though, is to contain images of all the sub sub-nodes [Assumption 2].
		ToDo:That needs to be verified as there might be some optimization checks scrapping this data as it is made available
		outside of the Trie.

		If, only a Security Descriptor is provided, only it is injected.

		In any case, changes are made to propagate to the Root of the Trie.

		There never is a need to modofy and /or update any of the sub-elements of the folder, in this scenario.

		-----------------------

		Notice: a leaf-node might as well be a 'directory' i.e. an instance of a CTrieDB.
		
		In order to support explicit modification of directories, as a directory is modified (i.e. its name 
		or its security descriptor gets changed), we need to make sure that the updated version of the directory
		'holds' all of the child-nodes. For this, we need to check for all of these and make sure references in the modified (upper-level, currently modified)
		directory are valid.

		Child-nodes, while residing in the cold storage, they do not hold any references to parential objects.
		Such references are generated on-the fly as data gets retrieved, instead.
		Thus, there is no need to be modifying child-nodes, in such a scenario at all.

		[Assumption 1]: *Typically, whenever files are explicitly modified, such an operation triggers an update which propagates as far as the Root-node goes.*
		
		This holds when modifying driectories, as a diretory is 'just' a leaf-node within a parental Merkle-Patricia-Trie.

		So, whenever a directory is explicitly updated:
		1) either A) a new instance of CTrieDB is provided *OR* B) just a new security descriptor is provided.
		2)  A) -> we replace the current CTrieDB retaining references (images/hashes!) of all of the child-nodes. 
			B) -> set the new security descriptor within of the present instance and be on our way
		3) we rely on the exisiting mechanics assuring propagation of changes, upwords and towards the Root of the Trie [Assumption 1].
		[Theory - Modification of Directories] - END

		*/

So… seems like everything needed was already in place.

As for tackling the current bug, our first approach relied on introducing falsified security descriptors, dynamically for files that were being retrieved from the ‘token_pools’ directory.
It worked.

Together with @CodesInChaos we’ve decided it was too dirty an approach and it kept introducing a performance hit, of course, as the check was being made each time any file on the system was accessed.

Anyway, here’s the code:

//Retrieves security descriptior associated with an object.
std::shared_ptr<CRightsToken> CTrieNode::getPermToken()
{
	std::lock_guard<std::recursive_mutex> lock(mGuardian);

	//System State-Domain - BEGIN
	std::string path = this->getPath(true);
	std::string systemDir =("token_pools/");
	if (path.rfind(systemDir,0)==0)
	{//hot-fix to mitigate a missing security descriptor on the 'System' state-domain.
	//this should be a 'one-time'process, after which a security descriptor should be inherited just fine during consecutive access attempts.
			mPermToken= CRightsToken::genSysOnlyDescriptor();
	
	}
	//System State-Domain - END


	return mPermToken;
}

So we decided to introduce a #GridScript patch, to be broadcasted across the network, signed through Overwatch privileges. The ‘patch’ would remotely alter security descriptors within the Token Pools system directory and of all of the files within.

I need yet to code it and ask for a permission from his holiness @TheOldWizard

Copy. Over and out.

Together with @Major we’ve prepared an update; which once executed on Test-Net would render the security descriptors to report as follows:

@Major already filed for an Overwatch authorization request.

From what I can see we’ve already received 2 out of 3 signatures required.

I’ll play around by the morning just a little bit; after that and once authorization received I will proceed with the update.

After post-mortem analysis; it turns out that there never was an actual security breach.

So what was wrong? Reasons aside, it turned out that we had wrong security descriptors assigned to a directory containing Token Pools (used for off-the-chain payments). Since we do not want even owners to be able to modify these directly, once deployed, and used, these files should have a a security descriptor associated giving write access only to the System i.e. decentralized consensus.

Question is: was security compromised in any way?

It was not.
image
The above permissions indicate that only owner used to have write permissions on the ‘token-pools’ folder.

The important questions is, who was the actual owner.

Since neither ‘token-pools’ directory nor the files within used to have explicit security descriptors associated, implicit ones would be used. And these would render the ‘System’ state-domain as the actual owner.

Thus, as a result the person who registered the token pool would have no control over files located within the System State Domain.

In any case for security reasons it undeniably was good that we’ve taken care of this by assigning explicit system-only security descriptors as this imposes an additional layer of security.

I went ahead and described the entire update procedure right over here.

Can you please provide a reproducible example showcasing how you create a hidden file?

Sure:
Start terminal
Enter your own state domain
bt
data hidden_content
2
data hiddenfile
setVarEx
ct

LESS easily views its contents if you know the proper file name. (in my case - my state domain - file is called “hidden” ;))

1 Like


I have verified and confirmed the issue.
While being able to open a file, knowing its name, should definitely be possible, I do not think that we should be listing even the information that there is a ‘hidden file’ at least by default.

The error in the File Manager UI dApp stems from the fact that a meta-data entry, through the VM Meta Data exchange sub-protocol, with a file-name set to ‘-hidden-’ is being delivered to the JavaScript main sub-system, running within of the web-browser.

The UI dApp is thus notified about a non-existing ‘-hidden-’ file and then attempts to open it.

I will go ahead and alter the system so that no information regarding hidden files is shown, by default, that should take care of both the issues. Proceeding now.

an interesting find, I spotted within the source code.

If a file is ‘hidden’ it actually does NOT have any filename assigned.

It can be only thus addresses through an image/hash of its actual filename, which is being stored nowhere though. Thus, once one does ‘less’, the utility computes a hash of the name user provided and attempts to traverse the Merkle Patricia Trie through this very value.

It was more writing in this post then actual coding :rofl:
image

Let me know verify how it works…

image

and well it works. @Alpacalypse feel invited to re-evaluate. I’m putting a tick next to this point.

1 Like

Warming up, anyway, I’m on it!

From what I recall, there is a bitfield, associated with each file, information about which is delivered through the VM Meta-Data sub-protocol. I believe one of these flags indicates that the file is hidden.

We could hide it thus, based on this very flag, from within of the File Manager UI dApp, but I agree that it’s probably best not to generate a VM Meta Data entry for this file at all during ‘normal’ queries.

Update: I’ve been working a lot on these mechanics. All in all I knew it was not perfect the day I’ve had made the functionality available; just was hoping… it wouldn’t be that evident and /or imminent.

The thing is… handling the context menu… the way we do… with all those animations happening… with elements residing within of the Shadow DOM… one could thing it would be easy but it turns out extremely complicated.

And unfortunately ( honestly, not to trying to push the blame onto others here…) it is because of Chromium…

We have been coping with the nature of Chromium since the day 1 we set out on implementation of the Web UI.

With great results, but at great development costs.

Chromium, sometimes would be forgetting to fire events.
Other times it would DECIDE not to fire events of a certain type entirely and sometimes for a single DOM element… to optimize performance.

That is how things are.

There is no other choice to go than Chromium engine that is certain.

But sometime we do need to be implementing extremely sophisticated heuristics to offset for the uncertainties imposed by the underlying web-engine.

It turns out that employment of Shadow DOM, while providing compartmentation of UI dApps (at least a sort of), it messes up with Chromium’s event-processing queues even more.

I’m not yet satisfied by the results, I’ll be keeping you updated.

1 Like

It could be said that Chromium prioritized performance over completeness of actions and events .

Interestingly that behavior should be less visible as users keep switching onto newer faster computers.

On the other hand, yes there are bugs in Chromium, we’ve been squeezing lots and lots out of it…to extreme extent we all know that.

Shadow DOM still remains an exotic use case scenario and we’ve got it all around. And sometimes events propagate from shadow DOM out to the main tree and things get real crazy.

If you test contest menu on desktop it is always fine.

When the file manager UI dApp with its shadow D sub tree comes around… well… things might get messy.

You are just underlying your achievements ; just say it…

Blah blah blah and it would be done… you know it I know it…

:thinking:… Whatever…
even if so, everything holds true.

Were it not for these adversities, we would be reaching the optimum much faster. That’s the point. At lower costs.

I have found the main thing messing things up in terms of the context-menu not being shown at all on consecutive right clicks. I have also immensely improved the fallback heuristics.

In addition, since it still might not be perfect, in certain scenarios, thus I consider this an ongoing (long-term?) topic to be discussed and researched over here.

Update: I have played around little bit more and now I believe the current results might be considered as solid, under normal usage scenarios.

I was able to verify that after an app based on the template gets installed and launched, files with an ‘.html’ extension would be opened through it. Proceeding with verification of whether it is a ‘bug’ indeed, or whether it is ‘by design’. One thing is certain. The template-based app does register a content-handler. And I have verified that that hander can be successfully overridden by explicitly selecting another UI dApp through the context-menu.

Let us see, as describe in the Tutorial, the app registers a custom, implicit file handler.

Now, looking at the CAppSelector component (part of the system), we can see that first, it tries to find an explicit handler, once a file is attempted to be opened.

Still, if no explicit file handler for ‘.html’ files is defined, it would traverse implicit handlers in the reverse in a LIFO (Last-In-First-Out) order. Notice that the custom UI dApp that you’ve installed was indeed the last one to register a file-handler.

The only thing remaining to be verified is whether the Editor UI dApp registers an explicit ‘.html’ file handler. Let us see…


No explicit, file handler, registered by the Editor UI dApp indeed.

Thus the behaviour is expected. It is by design, indeed.

@Alpacalypse or have I misunderstood something?

For now, I’ll place a tick next to this point.