Today I was wrestling with Azure Data Factory to get a list of folders; filter out an unwanted folder, and then iterate over these folders to do some magic. Getting the list of all folders gave me some headache, so I want to share my findings to do it right the first time.
- Create a dataset
- Get a folder list
- Add a filter component
Create the dataset
Create a new dataset
Choose type binary
Do not create a dataset with a file type of CSV or one of the other filetypes. This will give you a list of the files.
Enter the container name; in my case “costmanagement”.
Get folder list
Add a Get metadata component
Add the “Child Items” in the field list; nothing else necessary.
When you run now; you can see the output of the metadata component filled with a list of all folders within the container.
Enter the filter settings:
Items = @activity(‘GetListofBlobs’).output.childItems
Filter out all folders starting with an underscore: @not(startswith(item().name,’_’))
I’ve tried the same with a datset of CSV and still don’t understand why I get the content of one file. Let me know when you have an explanation. But the above binary dataset works for me.