Copy files if file names match a regex; from SFTP to Blob storage – Azure Data Factory
I have followed the below steps, the pipeline runs fine, but the files are not moved:
created a new pipeline in Azure data factoryadded the ‘Copy and Move’ activitycreated source data set and linked service to connect SFTPcreated sink data set and linked service to connect blob storagein the source, i have chosen the wild card path and used the above regex in the wild card file name.
The pipeline runs successfully but it doesnt move the matching files. In the SFTP server i have 3 csv files, but only two of those match the regex. I was expecting to see those two in the storage container, but it wasnt the case. I’m afraid the regex patter may be wrong. Here is the sample file name: SCUST_TXT20240417-032938-891.CSV.
If i change the wildcard file name as: *.CSV, the pipeline moves all the three files to the destination (sink). So i assume it’s a problem with the regex.
I have a list of files on an SFTP server, and I would like to copy and move some of those to the blob storage container. The filenamess should match a regex such as: (SCUST_TXT)[0-9]*(-)[0-9]*(-)[0-9]*(.)(CSV).I have followed the below steps, the pipeline runs fine, but the files are not moved:created a new pipeline in Azure data factoryadded the ‘Copy and Move’ activitycreated source data set and linked service to connect SFTPcreated sink data set and linked service to connect blob storagein the source, i have chosen the wild card path and used the above regex in the wild card file name.The pipeline runs successfully but it doesnt move the matching files. In the SFTP server i have 3 csv files, but only two of those match the regex. I was expecting to see those two in the storage container, but it wasnt the case. I’m afraid the regex patter may be wrong. Here is the sample file name: SCUST_TXT20240417-032938-891.CSV.If i change the wildcard file name as: *.CSV, the pipeline moves all the three files to the destination (sink). So i assume it’s a problem with the regex. Read More