Exclude System or Hidden Folders from Mac Dir List

Here is the attributes for the System Volume Information directory on a Mac disk.

How can I get a Dir listing of all folders on a Mac drive but exclude hidden or system folders.

Note (see 2) ... System Attribute for a System file is Nan... So, if I cannot use the attribute function how can I establish (reliably) what are system or hidden files.

AVF_INFO and 'System...' are hidden directories . DCIM and PRIVATE are 'normal' dirs I expect to find in a dir list.

Note 2. I have already eliminated lots of directories which I also needed to exclude from my Dir list.

MyDirList = HD_FullList([HD_FullList.isdir]);  %remove files from list
MyDirList=MyDirList(~ismember({MyDirList.name},{'.','..'}));% remove . & ..
MyDirList=MyDirList(~ismember({MyDirList.name},{'$RECYCLE.BIN'}));% remove recycle bin
MyDirList=MyDirList(~ismember({MyDirList.name},{'.Trashes'}));% remove trash dirs
MyDirList( startsWith({MyDirList.name}, '.') ) = [];  %exclude hidden files ie starting with .

And excuse me for venting a little rant here.... Why is it so ridulously difficult to get dir lists of files and directories on Mac and Win. This is excruciatingly painful... and seriously error prone. This might be my last MatLab project.

[dpb]

I'll join in a little of the rant -- I've never understood and always thought (and submitted enhancment requests multiple times years and years ago before finally giving it up as lost cause) dir() should honor the underlying OS switches to give control to the user; ideal of "platform-independent" be d---d. If the user has to deal with multiple platforms, that's part of their need/duties to deal with; give the user the tools to do so.

In general, if I do need more (or less) than what the normal dir() will retrieve; I'll just pass the command to the underlying OS and parse the result instead.

I "know nuthink!" per Sgt. Schultz about the Mac so can't comment further on specifics for it...

[Matt O'Brien]

Thank you for your feedback. Appreciated. We are completely on the same page.

I think the absense of a proper MatLab dir function which works on both Mac and Win, with switches to manage System and Hidden files is a massive own goal.

My only workaround is to hard code for files or directories I find which do not have the attribute set as per their real world status. This makes producing future proof or bug free code not only impossible, but implies code is flawed from the start.

To me, this signals the end of my adoption of Matlab and my current project is now relegated to the status of prototype.

There is a lot to like about MatLab.... but there are too many barriers to overcome.

I will no longer recommend Matlab as a small footprint IDE as I have been doing.

I have tried to write my own Dir function, but if I cannot properly identify system or hidden files on a Mac.... what is the point. I am disappointed, because I have invested so much time on this and now must committ to washing all this effort down the sink.

I will log a ticket with Matlab... in case there is something obvious I am missing..

Sad, but so be it.

.

[Walter Roberson]

What is a "system" file or folder on MacOS ?

[Matt O'Brien]

Two examples included in the list above.

1. 'System Volume Information' is a directory included when I use Dir on a Mac. I assume this is both a Sytem file and perhaps hidden file. I just need to make sure that I can remove it from my Dir list (eg. I do not want to copy this dir by mistake, or include this dir for a user to select.. and subsequently encounter uncontrollable error conditions.
2. AVF_INFO. This is a hidden directory created by Sony when they format Sd cards in their cameras. It contains a database of the images and video files stored on the SD card and allows the Sony cameras display thumnnails and other key features on a modern digital camera. Again, this is invisible, by default in most scenarios.... but it is included in the contents of a Dir command. What is worse..... the attributes function returns a value of NaN for this directory.... so other than hard coding this directory name, I cannot exclude it from a dir list... so my app will fail at some stage when it tries to take inappropriate actions on these directories.

If you search many of the queries re flaws with the Dir command on Mac... they recommend you exclude any file or directory which starts with a full stop (ie '.').

That will remove a lot of hidden files... but it is not a safe strategy... as I have tried and it does not exclude the 2 examples referenced here. What is worse... I have no way of knowing what other system or hidden directories my app might bump into in the future...with totally and unacceptably unpredictable results.

Thanks for your query and allowing me to explain this further.

[Walter Roberson]

Notice the DCIM directory. That is a memory card for a digital camera. The standards say that camera memory cards will be formatted by one of DOS FAT file systems. "system volume information" is not generated by MacOS, it is there as part of the DOS file system. MacOS does not offer "system" files or directories.

If you are expecting MATLAB to populate Windows file system attributes for a Windows file system mounted on a Mac, I am not sure that is a reasonable thing to expect.

[Matt O'Brien]

You are correct .... these are SD cards, which conform to DCIM standards, used globally by the camera / video industry, which includes Fat and ExFat standards. Millions of users around the world are using these cards in Macs and Windows machines, directly in a Mac SD reader or via backups of these cards taken in the field. It is an outdated standard, I strongly would be in favour of this standard been upgraded, but we can only use what we have.

There are large number of MatLab users reading their image files from such cards.

The following screen grab is from a Mac screen.

Note that Mac treats the System / Hidden files differently (faded font in the Yellow boxes) ... so the Mac OS knows the difference.

I am not asking that MatLab should recognise and deal with Windows file attributes ... but MatLab should recognise and allow an app to distinguish between these directory / file types.

If MatLab are not prepared to do that then it reinforces in my mind that MatLab is not a place I want to invest time and money.

I will accept a workaround... but so far I have not got one.

I have posted a support case to MatLab. Lets see what MatLab say...

I am currently hard coding my app to deal with this .... but that is guaranteed to break down at some point in the future .. so it would be immoral for me to share my apps as a result.

I do appreciate that you have taken the time and effort to respons and I am grateful for that.

[dpb]

What does the MacOS system 'ls' return for those files? Unless it returns some attribute or has some switch to differentiate them, then you're pretty-much hosed independently of MATLAB.

The MATLAB dir() function on Windows does return hidden/system files whereas the normal OS CMD dir does not; the "/a:hs" switches are negated by default for those two attributes whereas TMW has chosen to set them for their internal dir() function. That has never made any sense to me; particularly when they won't pass a switch and then don't return the attributes field to let one screen by. Of course, Walter just reminded me a day or so ago that even the .Folder property is a moderately recent addition as it came up in another Q? of a poster using R2013x and running into that problem.

I don't know what MacOS does, but I would think the workaround has to be to use the OS-supplied functions and that you'll run into the same issue on whatever platform you develop if it doesn't identify those as eventually it gets passed down to the underlying OS APIs that are fundamental.

As noted, I began a crusade some 30 years ago by submitting enhancement requests; after 10 years of no action, finally just gave up that quest as hopeless. But I still fail to see why TMW is so apparently klewless on this front...

[dpb]

"I will accept a workaround... but so far I have not got one."

The obvious workaround is to use the underlying OS equivalent -- as noted above, however, unless it can tell the difference, then you're pretty-much out of luck no matter what; the differences in the above screen image would seem to imply there is some attribute that is distinguishable or else the mounted file system has installed its own extension into the UI as well and is bypassing the standard OS API.

I never used a Mac and have never done anything with the digital camera stuff even on Windows, so I've not direct knowledge about what/how it is done, but anything you do inside or outside of MATLAB would seem to have similar issues no matter what programming language you choose.

I found a perhaps helpful <Apple link on the file system>; in it is the following --

Creating and Handling Invisible Files

In macOS, you make a file invisible to the user by prepending its name with a period character. Adding this character signals to the Finder and other display-related interfaces that they should treat the file as invisible under normal conditions. You should use this capability sparingly, though, and only in very specific cases. For example, the process for localizing a custom directory name involves placing hidden files with the localized names at the top-level of the directory. You should never use this technique to hide user documents or other data your app creates.

Even if you mark a file as invisible, there are still ways for the user to see it. Users can always view the true file system contents using the Terminal app. Unlike the Finder, the command line in the Terminal window displays the actual names of items in the file system and not their display names or localized names. You can also show hidden files in the Open and Save panels (using the setShowsHiddenFiles: method) in cases where you want the user to be able to select them for viewing or editing.

Regardless of whether a file contains a leading period, your app’s code can always see hidden files. Methods that enumerate the contents of directories generally include hidden files in the enumeration. The only exceptions to this rule is the meta directories . and .. that represent the current and parent directories. Therefore, any code that enumerates the contents of a directory should be prepared to handle hidden files and deal with them appropriately, usually by just ignoring them.

From the above, I presume that probably the particular files of interest may begin with the dot in the filename for hidden files; not sure about system files although I presume they would also be hidden.

Good luck, but looks to me like you'll have similar issues no matter what you use to write the application in; doesn't appear this is anything that will be unique to MATLAB although I still agree they could supply more regarding what switches are set/not and let the user control). The ls man page has the list of switches; looks as though it should be feasible to return the limited list desired and/or use the annotation of the type. The big issue with unix-like filenames is that such abominations as \n are allowable legal characters in a filename so you can't rely 100% on parsing a text listing since it uses \n to separate records -- and one of those could just have been part of a filename--Catch 22!

[Matt O'Brien]

A few commemts on the above.

1. I have started to look at the ls command... but not optimistic. Mac documentation for this command is extremely sparse and refers to general linux to get more info... which so far has not revealed a magic option... but have not given up on this.
2. In the code window above... you can see that I have already removed all the dot and dot dot dirs, everything beginning with a dot. Sony have obviously found a way to flag a folder as hidden (AVF_INFO). No doubt Canon and Nikon and Fuji will all use similar techniques to keep their SD card housekeeping directories away from casual misuse. I know in Sony's case, if the AVF_INFO is missing the camera will give you a message... pls wait ... while it rebuilds it. My fear is that my code may need to be hardcoded for every brand of camera.
3. There is a Mac keyboard combo of Cmd Shift + which reveals / hides System / Hidden files. I would love to be able to replicate this in code.
4. I intend to investigate how Apple Swift / Xcode handles hidden files. The danger is that if I go that far it might be easier for me to write an Apple native app.
5. The problem I have is that I have the app working for Windows but running aground trying to resolve this issue on the Mac.

Thanks for posting the above info. I will explore further.

[Walter Roberson]

Hiding dot files is a convention, not something enforced by the OS.

MacOS HFS+ and APFS file systems support "flags" ( https://www.maketecheasier.com/use-file-flags-modify-file-behavior-macos/ ) that can be set with chflags https://ss64.com/osx/chflags.html -- but not queried with that command. At the moment all I see for querying the flags is ls -lO (lower-case L, capital Oh), but I would imagine there are other ways to get the flags. Ah, I see they can be read with stat see https://unix.stackexchange.com/questions/640422/api-syscall-to-read-or-list-bsd-macos-file-flags

The supported flags do include hidden but not system

It would not be unreasonable for a MacOS filesystem layer reading a Windows system to examine the windows attributes and fill in the hidden attribute of the MacOS stat() call, but MacOS does not have a system attribute in stat(), so in order to code the system attribute MacOS would have to go even further afield, such as synthesizing an extended attribute https://ss64.com/osx/xattr.html .

I would suggest that even if MacOS went to that trouble, that expecting MATLAB's dir() call to examine that kind of information would be much the same as expecting MATLAB's dir() call to fill out the USERREAD USERWRITE USEREXECUTE fields based upon interpreting Access Control List https://www.techrepublic.com/article/introduction-to-os-x-access-control-lists-acls/ . For example some might say that for a file, USEREXECUTE should be false if the file has com.apple.quarintine associated with it. But other people would say that the results of dir() should reflect only the basic attributes associated with the file -- and on MacOS, going to look in the flags for hidden or (hypothetically) looking in extended attributes to see if some kind of system attribute has been synthesized is not basic attributes associated with a file.

Question: is there a standard for how third party filesystem drivers such as FUSE (for Linux ext* filesystems) or Paragon's NTFS drivers should import non-MACoS filesystem flags and attributes and access control lists? A standard that Mathworks could follow so that it always gets the flags in dir() "right" ?

[dpb]

From what I saw, looks like #5 above is going to be an issue regardless how you attack it; that you're working in MATLAB seems to be irrelevant to the issues you're running into.

If you can't find some other code hacks to handle such on someplace like GitHub or StackOverflow, you may have to resort to native code via mex or somesuch route. However, it seems unlikely this hasn't been solved before as much of a platform for image processing is Mac.

One other link I just found that might be of some use <ImageCaptureCore> on the Apple developer site...I've not looked into it at all that may be that toolset you're looking for, I don't know.

[Matt O'Brien]

Thanks for your efforts to dig into this.....

I suspect it is beyond my skill level to delve too deep into this space. I am a very recent Mac user (because of their MacAir laptop range and having been burned badly twice buying high end Win laptops) and not a Mac O/S specialist or developer.

It is likely I will give up on this... but will probably explore a little bit further and revisit from time to time.

I am keen to get the functionality working on Mac.... so I can see myself exploring alternatives to MatLab if other tools can more easily deal with this challenge.

Thanks again.

[dpb]

OBTW...just a thought -- since apparently the mounted DICOM file system is DOS-based, you may be in luck in that the \n conundrum can't bite you for these folders because it isn't allowable under that file system.

My only experience w/ Linux was 30 years ago now with a Foxboro IA control system upgrade to Unit 9 at the TVA Kingston Fossil Steam Plant; supported/converted a realtime heat rate calculation monitoring system from VAX to run on then NT4; it was virtually all Fortran with some low-level C for the communications. I remember even then "issues" dealing with the file system on the Silicon Graphics machine it was built around...from VMS, the difference was jarring to say the least! I never had reason to deal with anything more in that line and can't say as I feel I've missed anything... :)

Anyways, good luck! Looks like you'll need it, but I don't think your problems are really MATLAB-related, but idiosyncracies of the OS.

[Matt O'Brien]

Our messages are overlapping...so apologies if my comments are slightly out of synch.

Thanks for the previous link. I did start to look at more generic scenarios and reading directly from the card in a device, whether it be a camera or a card reader is an option.

However, for my needs, I need to be able to read the image or video data from a card reader (most frequent situation) or (less regularly, but important) a portable storage device (ie Sdd) which contains backups of multiple cards. The scenario I am catering for usually involves 10,000's of 50MB images per card, so it is high volume processing and thus the reason to streamline / automate aspects of this process.

However, these tools may lead to some cluses or work arounds.

I started off in the 70's writing Fortran iv (Civil Engineering apps). I then began coding directly in machine code, advanced to assembler and then on to 3rd gen languages (Cobol was my fav) before moving to mainframes. At that stage I left coding behind as I became responsible for teams, projects, enterprise computing, etc..

Thanks for all your contributions.

Typical,,, what starts off as a 50 line script and a basic idea becomes a good idea and a monster....

I always had a healthy respect for Digital and the PDP/11. I remember reading The Soul Of A New Machine (Data General) as it was the early days when alternative to mainframe computing started to emerge.

https://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine

Regards.

[dpb]

We're not that far apart, then although I predate you by probably about 10 years or so...first mainframe was a Philco 2000; the class scientific computing machine of its era -- no disk or durm, but 27(!) 7-track tape drives. Fortran II and assembler for it running the proprietary versions of the nuclear design codes from Bettis for a commercial nuclear power vendor. The Philco was already in the process of being retired when I arrived but it took over two years to get the CDC 6600 installed and able to run all the needed codes before could pull the plug on the old Philco. Other than the commercial timeshare still in Philly, I think it well may have been the last one in a production environment. Fairly shortly after it was replaced by a Cyber 76...after only a couple years with the latter, I moved on to the consulting gig and the other end of embedded/standalone systems, mostly around the M6809E running Forth. Quite a number of various specific-purpose "instruments" built with this configuration from loose-parts monitoring to neutron-noise systems looking for things like inner liner vibration/movement, etc., ...

[Walter Roberson]

@Matt O'Brien

Note that Mac treats the System / Hidden files differently (faded font in the Yellow boxes) ... so the Mac OS knows the difference.

At most MacOS knows about hidden, not about system.

'System Volume Information' is a directory included when I use Dir on a Mac. I assume this is both a Sytem file and perhaps hidden file.

System Volume Information is a Windows artifact; on Windows it is system folder (possibly hidden too, not sure right now) used to contain restore points.

Are you expecting that the MacOS built-in FAT filesystem drivers will convert the DOS system attribute into some kind of flag or attribute or extended attribute that MATLAB should know to check? If so, then what are you expecting that MacOS already converts the Windows system attribute into, that MATLAB is being negligent in checking? Or are you expecting that MATLAB should have its own FAT filesystem drivers, that it should detect that it is a Windows filesystem and should do its own block-level reading of the files and interpret the binary attributes so that it can fill in the system attribute for the dir() call ? Or are you saying that Mathworks has been irresponsible in not already having (successfully) pushed Apple to convert the DOS FAT hidden attribute into something that MATLAB could have gotten ahold of for the purposes of filling out the dir() information ?

[Matt O'Brien]

Yes... I noticed the behaviour of the 'Hidden' files on Mac.... When I discovered 'hidden' folders using the MatLab dir function on a Mac... I immediately learned how to hide / reveal these directories using the keyboard short cut .. which told me that Apple know how to identify these files / directories.

I had already discovered the attrib features, from testing my app on Windows. I assumed (dangerous) that this feature would also work on Mac.... but learned very quickly that it does not. This brings me to the start of my query (ie first post/query above.

I am not expecting MatLab or Mac to change any of its behaviour in relation to how it handles such cards / directories / files. My app has an option to copy from card to disk (where backup cards are retained within a project structure). I have used Beyond Compare to compare the source card and the destination on a hard drive. They match perfectly. I have checked this with multiple cards, multiple times on multiple machines (and also on Windows).

So parts of my app are delighted these System / Hidden files are available through the Dir command. However, I also need to analyse the contents of the directories. Unfortunately, while DCIM standards exist, every Camera maker has their own naming convention for raw or jpg files and there does not seem to be any convention for video files. Also video codecs change all the time as new standards and the limits of hardware continues to advance.

So it is not feasible for me to build an app that can cater for all the naming conventions of every camera / video maker in the world... so I cannot use Folder names or file extensions as a filtering mechanism.

Most people can build apps to process these images either by hardcoding the file types or be selective in terms of the file extensions they use.

In my case I cannot do that. At the same time I cannot have system files or hidden files appearing in reports or analyses or as part of any image processing routines.

Also.... inceasingly .... lots of monitoring and measuring systems are storing data on embedded SD cards ... so the ability to be able to select only files which are not system or hidden becomes increasingly important.

The key answer to the question you asked is ....

I do not need any special processing or handling ...... the way the system handles the files (eg copy from SD to disk is perfect) but I do need a means to recognise what files are system or hidden, so they can be excluded in certain cases and included in other scenarios. I am not too concerned as to what method is used to do that (ie I do not need the attrib function/feature modified)

Hopefully that makes sense. I am happy to clarify any points.

When I get a chance, I am going to look for a Mac utility which will list all dirs and files on an SD card and can include / exclude system or hidden files. Maybe I can get Beyond Compare or similar to create a text file of hidden or system files. My app can run that app, store the results and I can then use the information within the logic of my app.

PS... I suspect the system files are also hidden files.

[Walter Roberson]

Why is it so ridulously difficult to get dir lists of files and directories on Mac and Win. This is excruciatingly painful... and seriously error prone. This might be my last MatLab project.

Use the ** directory specification, such as

dinfo = dir('/Volumes/Card52312/**');

This will return everything from Card52312 and later, all in one list. dinfo.folder will give the names of the containing folders and dinfo.name will give you the names inside the folder, and dinfo.isdir will tell you whether the entry is a directory or not.

Is dir() a flexible function that will allow you to filter the returned information based on various criteria? No, it is not. It is closer to the unix stat() call, which returns a list of whatever is there, expecting you to do any post-processing that might be appropriate.

Is MATLAB being especially negligent in not returning information that other programming languages makes readily available? No: C++ (2017) std::filesystem does not provide any facility for searching directories for folders/files with particular attributes, and the set of permissions supported by C++ does not include system or hidden or any analogues; https://en.cppreference.com/w/cpp/filesystem/perms

People who need filtered file information on Unix-type systems code the search themselves, or find a pre-written library that will do the search for them -- or use the standard Unix utility find https://ss64.com/osx/find.html . For example,

find . -flags +hidden -print

./DCIM/114NIKON/._DSCN2398.JPG

./DCIM/114NIKON/._DSCN2399.JPG

./DCIM/114NIKON/._DSCN2406.JPG

./NIKON001.DSC

./._.Trashes

./.Trashes

./.fseventsd

./System Volume Information

find . -type d -flags +hidden -print

./.Trashes

./.fseventsd

./System Volume Information

... which still does not permit you to look for system

(Side note: Using ** or using the find utility might potentially cause MacOS to "materialize" dataless files and directories; See https://github.com/fish-shell/fish-shell/issues/8399 )

[dpb]

"Is dir() a flexible function that will allow you to filter the returned information based on various criteria? No, it is not. ..."

That it is what it is, is certainly true, but that it doesn't at least pass down the available equivalent OS common switches isn't much in keeping with the "rapid development" paradigm that's supposed to be MATLAB's forte and reason to exist...that one can parse the results oneself is also certainly true, but one can also rewrite a sort algorithm from scratch every time, too; but why should one have to do so?

[Walter Roberson]

There may be room for a findobj() type of operation... but dir() was never designed to be that. I would have a hard time coming up with any computer language that had those kind of operations built in.

[Matt O'Brien]

I did a very quick Google search for options to identify hidden files in Swift / Xcode (ie native App on Mac).

On my first search I found the following .... I highlighted the most interesting sections of this example.

I cannot be sure this will work... but it looks promisiong.

https://stackoverflow.com/questions/27721418/getting-list-of-files-in-documents-folder

At this stage, I am not spending any more time on this. I will put my use of MatLab on hold... but expect I will abandon MatLab completely.

I will also warn my colleagues and interested parties in relation to this major black hole when trying to process image data from Media cards. My purpose in writing the app was to be able to help others manage high volume image processing. As I know how to manage this stuff myself I see no point in providing an app which is doomed to failure in due course. I can hard code my app if I wish to deal with cameras I use myself to test if I have identifed and coded for all system and hidden files.

I am going to take a rest from this.

My next step will be to see if I can find a Mac Utility which will identify hidden / system files, but I have spent way too much time on this (and learning MatLab) already.

Thanks to Walter for his time and efforts.

Regards.

[dpb]

I yet fail to see why/how this is actually related to MATLAB per se, though.

Can one call SWIFT from C? If so, then a mex gateway should be doable...

But, if you use/move the app to SWIFT for Apple, what do you do for everything else -- would take multiple apps to support alternate platforms; afaict SWIFT doesn't port to Windoes...

[dpb]

@Walter Roberson -- dir() has to eventually devolve to the underlying OS filesystem interaction API calls, anyway, nothing other than using the same switches as does the CMD(*) "dir" or Linux "ls" would be all I'm asking for.

(*) Is TMW ever going to recognize powershell, I wonder? Although it's dreadfully arcane, it makes a lot more features available, it seems. It's always been another frustration that they ignore the COMSPEC environment variable so if use another default command processor, one doesn't get it instead but CMD; one has to launce the other shell explicitly first.

[Matt O'Brien]

Can some one show me how to execute the following 'ls' commands from inside MatLab, along the following lines.

MyLsList = ls -a MyDirString

MyLsList = ls MyDirString

so I can capture the related ls output in the variable MyLsList and MyDirString contains the target directory of my choice.

I can get the Ls command to work but unable to add in the -a parameter. If I can get these two Ls commands to work, one should contain all all entries incl hidden and the second may be the list I am looking for.

Or perhaps I need to recognise the difference between these lists to find the hidden entries.

This would be a possible workaround for me.

[dpb]

I don't have a non-Windoes OS to illustrate with, but the format should suffice to see syntax.

>> [~,s]=system('dir /a:h /b c:\*.sys')
s =
    'hiberfil.sys
     pagefile.sys
     swapfile.sys
     '
>> 

For a variable string use something like (be sure to be careful to keep embedded spaces where needed between command switches and the filename, etc., ...). For example,

cmd=strcat('dir /a:h /b ',fn);

looks just right, doesn't it? But, it has the nasty behavior that it does an internal strtrim() on the arguments before doing the catenation so you end up with the filename string being directly appended to the last command switch and Boom! the call fails and you'll spend a long time looking to figure out why... :( To make the behavior even more obtuse, if you use a cellstr or string class, then it doesn't trim--go figure.

>> fn=fullfile('c:','*.sys');
>> cmd=['dir /a:h /b ' fn];
>> [~,s]=system(cmd)
s =
    'hiberfil.sys
     pagefile.sys
     swapfile.sys
     '
>> 

NOTA BENE: the above is one string with embedded \n -- remember the caveat that hopefully nobody was nasty enough to have used one in an actual filename on your system...that's a nasty.

>> split(s,newline)
ans =
  4×1 cell array
    {'hiberfil.sys'}
    {'pagefile.sys'}
    {'swapfile.sys'}
    {0×0 char      }
>>

That's with MS CMD processor that MATLAB uses "out-of-the-box" -- another aberration is that TMW ignores the COMSPEC environment variable so one has to go thru gyrations to get another shell. I use the JPSoft (shameless plug for super product; not related, just 20+ year user) replacement which has more functionality included a "/f" switch besides "/b" to remove the header/trailer info; it returns the fully-qualified name for all.

@Walter Roberson is bound to come along before long and can/will give a specific example, undoubtedly! :)

Answer 1

[Matt O'Brien]

Thanks for the syntax.

I got the following to work ... in principle ...

-a includes hiddenitems

-R recursive mode ...

clc
MyLsString= strcat('ls -a -R /Volumes/MoB_WD_5TB/DeeAprilTravel2023');
[~,s]=system(MyLsString);
MyList = splitlines(s);
MyLen = length(MyList);
z=0;    

But I am not sure I have the skills or trust my efforts to untangle the output.

Splitlines.

I used split lines to get lines raather than a single long text string. This seems to work ... but not sure I can trust the output....

More troubling....

Data Returned.

No doubt this data can be unscrambled... but I would not trust myself to do this.

Also ... I see lots of funny scenarios.

eg. Line 7 DCIM is a very important sub directory ... but it is on the same line as 'fseventsd-uuid'

Maybe this is a splitline issue.

So ... my summary ... Good to know how to issue system commands from Matlab .... but looks scary to try and convert this into useful data to resolve my issue.

[dpb]

Well, you get what the OS tools will give you...first thing I'd do would be to compare to the output format at the command line to see how it formats the output; as noted before there are "issues" with OS'es that allow such abominations as a \n as an allowable character in a filename -- or, perhaps some of the problems may be there are other nonprinting characters embedded in those -- only more spelunking will let you know what it is that is in some of those filename strings for certain.

Good luck; it reminds me why I an always glad despite its warts in other areas to not have to have dealt with the --ix file system.

From the stuff you posted before, it would look to me still like the more sure path would be to use a mex gateway into the SWIFT OS calling stuff you found above, although I'd definitely to some more exhaustive looking at the details of what ls returns and the actual returned formatted output. I've been know to use output redirection and capture a file in the past, too...what does the output of

ls -a -R /Volumes/MoB_WD_5TB/DeeAprilTravel2023 >lsOut.txt

look like? Is it unwrapped differently?

BTW in

MyLsString= strcat('ls -a -R /Volumes/MoB_WD_5TB/DeeAprilTravel2023');

the strcat function is totally superfluous...but, you probably already knew that...

More potentially problematical is

MyLen = length(MyList);

length is dangerous function; it is defined as max(size,x)) so it can potentially return either number of rows or columns. In this case, splitlines will always return a cell array so it works, but as a defensive coding technique, avoid using length and ask for the specific size direction of interest. It's a remnant of very first implementations but would not have been defined as such later...

[Walter Roberson]

You cannot use ls together with text splitting reliably determine filenames in any situation in which there might be spaces in valid file names. You instead need to use the '-1' (digit one) option, such as

!ls -1Ra ..

..:
.
..
mss.system.xRKeyY

../mss.system.xRKeyY:
.
..

The -a flag and the -1 flag are not supported on Windows.

It is not clear to me why you would want to be using -a to get all files when your complaint is about weakness in not having hidden files automatically filtered out.

In my opinion, if you are going to use OS system-specific code then you should be using find for this task

My next step will be to see if I can find a Mac Utility which will identify hidden / system files,

NO you will not find any such utility. Mac has no understanding of system files. Such a utility would have to be something like Paragon Software's NTFS drivers for Mac that treated the device as a file system.

[Matt O'Brien]

All good info.

I also take your point re use of -a.... my intention was to create 2 lists (include exclude hidden ) and compare the difference, but got sidelined with the splitline issue.

I spotted the -1 option, but did not grasp its use.

I will play around further with ls tomorrow, more to understand it further than to expect to find a solution.

[Matt O'Brien]

All good points... good tip re 'length' .... use of strcat was to allow me play with different combinations of parameters... in this case no parameters ... so looked superfluous....

Here is a screen grab of result of -1Ra ... have not figured out yet how to to try and structure the output...

This is the result of just hovering over the s value....

I am going to take a rest on this for a while... I am not going to try and convert the text into a structured dataset.... I can see some clues how to do that ... but not keen to follow that path...

I will keep you posted on any developments...

I sent you a PM ... just fyi...

Regards...

Matt O'Brien

Answer 2

[Matt O'Brien]

Update. Explore finding Hidden and Visible files/folders using Apple native tools.

I have spent considerable time exploring the Apple native apps such as Swift/Xcode and AppleScript.

My objective is two fold.

1. Explore, using native Apple apps, if I can separate visible and hidden dirs using a script or app
2. Recce the possibility of aborting MatLab and reverting to Swift/Xcode and build my apps on Mac / Ipad /Iphone.

I built the very basic and crude Apple script below.... It opens a folder selection dialog, where I select an SD card in a connected card reader.

It has 2 basic loops, both just at the top folder level (ie non recursive). The first looks for hidden folders and the second loop looks for visible folders. In each case it lists (logs) the folders found. The output is extremely crude... will make it pretty later.

Here is the results. It is a screen grab from the AppleScript IDE. Clearly, I can select and identify hidden folders and/or visible folders.

My Key Point.

I had to learn Apple developing and scripting tools from scratch. If I can do this .... then it must be seriously possible for MatLab to add a feature to the current Dir function or create a new function (to preserve compatibility / avoid bugs) which will enable the basic ability to be able to list visible / hidden files/folders with simplefunction option.

I am horrified that such an important feature in such an important function is missing and that I have to go to this level of effort to resolve in a reliable way.

In AppleScript I can identify visible / hidden folders with a single clear statement.

"set MyList to folders of the_folder whose visible is false"

I am enclosing the script I used (it is crude) just for your info.

My Next Action.

I plan to refine this script in due course, get it working for all subfolders.... and then begin to migrate my field apps from MaLab to work on native Apple code. A winter project. This will allow me to use my MacAir very efficiently in the field for ingesting high volume media files and manage the digital assets accumulated. I may even be able to get my app to work on iPhone or iPads, especially with USB C iPhones in the pipeline. That is all nice to have. What is critical for me now is I have a solution which is fit for purpose and not dependent on hard coding to cater for the infinite variety of hidden files or folders stored by different camera companies when saving images / video to digital media.

I will leave this post here for a week or so. If no other solution found by then I plan to mark this post as the answer to my query.

-- Get all files in a selected folder
-- Explore visible / hidden folders
log "Hidden"
-- Get user folder 
set the_path to choose folder with prompt "Choose your folder..."
my file_to_folder(the_path)

on file_to_folder(the_folder)
	
	tell application "System Events"
		-- Get all folders = hidden
		set MyList to folders of the_folder whose visible is false
		
		repeat with MyItem in MyList
			
			-- Remove extension of the file name
			set the_file_ext to name extension of MyItem
			set the_name to name of MyItem as string
			
			log the_name
			end repeat
	end tell
	log " "
	log "Visible"
	tell application "System Events"
		
		-- Get all folders = visible
		set MyList to folders of the_folder whose visible is true		
		repeat with MyItem in MyList
			
			-- Remove extension of the file name
			set the_file_ext to name extension of MyItem
			set the_name to name of MyItem as string
			log the_name
		end repeat
	end tell
end file_to_folder
-- Ending dialog
display dialog ("It's done!") buttons {"Perfect!"}

[Walter Roberson]

I already showed in https://www.mathworks.com/matlabcentral/answers/1950983-exclude-system-or-hidden-folders-from-mac-dir-list#comment_2717723 the (shell-level) command

find . -flags +hidden -print

that finds and lists all files and directories relative to the current directory that have the filesystem flag hidden set. One item is listed per line. No need to learn AppleScript or the like -- just use of one of the most common unix utilities.

[Walter Roberson]

function dinfo = dir_no_hidden(dirpath)
   if ~exist(dirpath, 'dir')
       dinfo = [];
       return;
   end
   dinfo = dir( fullfile(dirpath, '**') );
   cmd = sprintf("find '%s' -flags +hidden -print", dirpath);
   [status, msg] = system(cmd);
   if status == 0
       hidden_files = regexp(msg, '\r?\n+', 'split');
       dinfo_fullnames = fullfile({dinfo.folder}, {dinfo.name});
       need_to_exclude = ismember(dinfo_fullnames, hidden_files);
       dinfo(need_to_exclude) = [];
   end
end

[Matt O'Brien]

I am grateful for this (and your patience)... apologies if I missed it in the conversation.

I will explore this further.

I picked up the vibe that trying to establish hidden/system files from an SD card on Mac was not feasible from within MatLab , due to the Dos origins of the SD card structures. (Ps... I do not care to distinguish between hidden or system files, I just want to be sure I am excluding all hidden files/folders and including all visible files/folders).

I do remember you referring to using external system commands.

Regards...

[Walter Roberson]

Figuring out files are marked system is not easy; figuring out which files are marked hidden is much easier.

Your Swift / Applescript code is working with hidden and not system

[dpb]

@Walter Roberson, could the " ... One item is listed per line." be a potential problem for the allowed (I think on Apple?) aberrent file containg an embedded \n in that it can't be definitively proven the split is correct?

This may not be an issue for the OP if the SD cards are using DOS naming conventions so isn't possible, just a question that I saw raised during the previous looking I did earlier.

[Matt O'Brien]

Re Hidden / System. Yes... I accept your point completely re system / hidden. I am hoping that removing hidden files will give me a working solution.

I will work on the script above tonight (and maybe in the morning and will revert back).

The fact that I can get this to work within a function in MatLab is fab.

It is easier for me to check this out on my Win setup (32 inch screen and better ergonomics) and then check on my MacAir.

[dpb]

I still agree with the initial point that TMW should have either incorporated the common OS switches in the existing dir() function LONG ere now (like in the beginning) or have produced alternate functionality. It is, after all, the user-friendly rapid-development environment that is what it's all about; leaving this to the user to deal with is, simply put, "just plain rude!" as one of my granddaughters is wont to opine...it's doable, yes, but why can't we simply just use "/h" or "/-h" in some form other than calling the OS and parsing it ourselves?

Back years ago, I had done for Windows; unfortunately, it was another of the utilities that got left on the corporate machine when retired to the farm and forgot to offload that stuff before leaving...what I really miss is the mex extension that took a Fortran FORMAT string and variables then passed on the the Fortran i/o library --a MUCH nicer solution than writing those ugly C formatting strings with no repeat counts, etc., ...without paying consulting client, it hasn't been enough of an issue to have bothered to regenerate although at the moment I can't even get a Fortran compiler to link mex; not having the "high-priced spread" of the Intel compiler. Since they support gcc, there's no reason the Fortran compiler shouldn't be supported as well, but I've been unable to solve the name-mangling issues to link. Anyways, sorry for the side rant; we old fogeys tend to wander off...

[Matt O'Brien]

Aside. I cut my teeth using Fortran. There is a gas pipeline in Ireland whose concrete coating thickness (to keep submerged when crossing rivers, lakes and estuaries) I calculated with a few lines of Fortran ... which I believe is still doing the job it was designed to do.

I strongly believe the fundimental point that the dir() function needs the required switches or an sdir (Super Dir) function is implemented.

My dream language to use was Cobal for so many reasons. Brilliant for combining logic and data structures (but geared for large scale data processing).

Answer 3

[Matt O'Brien]

I am keen to supply feedback asap .... realised I needed to run the script in a Mac environment...

I copied the struct output from your script above into a table and saved to Excel.

With the test data I currently have on the card / card reader ... I have approx 700 lines of files/folders...

My initial dive into this is encouraging... with a little twist...

The folders, listed in these screen grabs, I am keen to exclude .... It looks like the script has successfully excluded these folders but left the files (or some of them) from within these folders. I only found these 3 examples.

I need to do a more thorough job of checking the test data and maybe testing a few other scenarios (eg different cameras). I may be missing some outliers in the other 700 rows, but initial inspection is looking very promising.

No need to try and tweak the script at this point. I will revert when I am comfortable I have digested the results.

I may need to get the list of 'hidden directories' and then specifically exclude / ignore the content of this directories.

(Ps1. Just FYI. there were dot and double dot folders included in the list ... which I am comfortable getting rid of.)

(Ps2. I will try to acquire a better grasp of the system 'find' command).

Answer 4

[Matt O'Brien]

Just a quick update.

I am getting very confusing results. While the system Find results were initially promising, the MatLab dir command is now giving me identical results as using the System find command, as per the the script above. Exactly the same number of files / directories returned by both (ie 676, so a decent volume of data). I expect the MatLab dir command would include all the hidden folders.... but now... they are excluded ?????

My suspicision (worry) is that setting the option (in Finder) to display or hide hidden objects is having an effect on the respective find and dir commands when executed from MatLab. I need to check this further. If true, this is a disaster.

The second point of confusion is the following scenario, hinted in a note above.

Sony creates a hidden AV-Info directory (might be spelled wrong, writing on my ipad). This contains several files. Both Dir and Find lists do not include the folder name, but do include the folder files. My simple assumption is that the folder is flagged on the system as hidden but the respective files in the folder are flagged as visible. If so, it means that I have to specifically identify folders which are hidden and then remove all their contents from the list supplied via Dir or Find.

I need to dig deeper.

This is horrendously time consuming... I am not sure at what point I give up. I will continue for the moment.. but growing weary.

Answer 5

[Matt O'Brien]

I am posting this... just in case there is interest ...I do not expect anyone to dig thru this stuff.

I found a glitch above and have completed a detailed comparison of the List genberated by MatLab Dir and System Find.

The result is close to the desired ... but there are a few glitches.

I am attaching an Excel spreadsheet which has a detailed line by line comparison of both lists, where I have commented on the differences (most at top and bottom of the list).

I am also enclosing a pdf with a few screen grabs, incl a screen grab of most of the folders expaned on the SD card, using Finder (and showing hidden items).

I need to digest this myself.... but looks like the Find list is close but will need to exclude visible files/folders stored in hidden folders (which may be several folders deep... ie the parent folder maybe visible but the parent of the parent may be hidden).

I will leave this for a few hours now and will review myself with a fresher pair of eyes.