Copy files based on wild card file name; from SFTP to Blob storage – Azure Data Factory
I have followed the below steps, the pipeline runs fine, but the files are not moved:
created a new pipeline in Azure data factoryadded the ‘Copy and Move’ activitycreated source data set and linked service to connect SFTPcreated sink data set and linked service to connect blob storagein the source, i have chosen the wild card path and used the above regex in the wild card file name.
The pipeline runs successfully but it doesn’t move the matching files. In the SFTP server i have 3 csv files, but only two of those match the wild card. I was expecting to see those two in the storage container, but it wasn’t the case. I’m afraid the wild card pattern is wrong. Here is the sample file name: SCUST_TXT20240417-032938-891.CSV.
If i change the wildcard file name as: *.CSV, the pipeline moves all the three files to the destination (sink). So i assume it’s a problem with the wild card.
I have a list of files on an SFTP server, and I would like to copy and move some of those to the blob storage container. The filenamess should match the wild card: (SCUST_TXT)[0-9]*(-)[0-9]*(-)[0-9]*(.)(CSV).I have followed the below steps, the pipeline runs fine, but the files are not moved:created a new pipeline in Azure data factoryadded the ‘Copy and Move’ activitycreated source data set and linked service to connect SFTPcreated sink data set and linked service to connect blob storagein the source, i have chosen the wild card path and used the above regex in the wild card file name.The pipeline runs successfully but it doesn’t move the matching files. In the SFTP server i have 3 csv files, but only two of those match the wild card. I was expecting to see those two in the storage container, but it wasn’t the case. I’m afraid the wild card pattern is wrong. Here is the sample file name: SCUST_TXT20240417-032938-891.CSV.If i change the wildcard file name as: *.CSV, the pipeline moves all the three files to the destination (sink). So i assume it’s a problem with the wild card. Read More