Error using mkdir with “lock folders” in a parfor loop
I am writing parfor loops and using folders as "lock files" for thread safety because mkdir is atomic and I believe is therefore the best option in MATLAB for a "lock file". However, I am getting the following error:
Error using makeLockFile
The system cannot find the path specified.
This error occurs on the line in makeLockFile containing
mkdir(pidPath);
Here is my code for makeLockFile (top), releaseLockFile (middle), and the helper function getMATLABPids (bottom):
What I am trying to do is:
Create a "lock folder". This folder has the same path as a .mat file (except the extension of course). If it exists, that means a MATLAB instance is using the corresponding .mat file.
If successful, create a subfolder named as the instance’s process ID, and return. This is where the issue is!
If unsuccessful, that means another process has locked this variable. In that case, remove any previous (now inactive) process’s lock subfolders that were left behind due to interrupted processes (this ensures that a process that is no longer running is not the sole reason for a lock). Then try deleting the lock folder. If that fails, pause and repeat.
When I encounter the above error, I can restart the process and then it runs all the way through, so I know it’s not a path correctness issue.
I don’t know how to debug this any further, I’ve been using disp() but can’t make heads or tails of why mkdir would ever fail in this context.
function [pidPath, status] = makeLockFile(lockFolderPath)
%% PURPOSE: CREATE A LOCK FOLDER, BECAUSE MKDIR IS ATOMIC.
% Clean the lockFolderPath and add the process ID.
[folderPath, folderName] = fileparts(lockFolderPath);
pid = num2str(feature(‘getpid’)); % Get MATLAB instance’s process ID.
lockFolderPath = [folderPath filesep folderName];
pidPath = [lockFolderPath filesep pid];
status=false;
while ~status
disp([‘PID: ‘ pid ‘ Lock Folder: ‘ lockFolderPath]);
[status, message, messageID] = mkdir(lockFolderPath);
try
if ~status
% The variable is locked.
% Get the listing of all folders in the variable folder
items = dir(lockFolderPath);
folderItems = items([items.isdir]);
folderNames = {folderItems.name};
folderNames(ismember(folderNames, {‘.’, ‘..’})) = []; % Remove ‘.’ and ‘..’
% Get all of the active MATLAB PID’s
pids = getMATLABPids();
% Remove the inactive (leftover from previous processes) PID’s
inactivePIDs = folderNames(~ismember(folderNames, pids));
for i = 1:length(inactivePIDs)
rmdir([lockFolderPath filesep inactivePIDs{i}]);
end
% Try releasing the lock folder. Has no effect if folder is not
% empty (i.e. active process folders present)
releaseLockFile(pidPath);
continue;
else
% Make the folder for this specific process ID
mkdir(pidPath);
end
catch ME
disp([‘ERROR PID: ‘ pid ‘ Lock Folder Path: ‘ pidPath]);
disp({ME.stack.file});
disp({ME.stack.name});
disp({ME.stack.line});
rethrow(ME);
end
end
end
function [] = releaseLockFile(pidPath)
%% PURPOSE: RELEASE THE LOCK FILE.
status = rmdir(pidPath); % PID-specific folder.
pid = feature(‘getpid’);
lockFolderPath = fileparts(pidPath); % Parent folder (the lock folder)
status = rmdir(lockFolderPath); % Remove the folder only if it is empty.
end
function [pids] = getMATLABPids()
%% PURPOSE: GET ALL OF THE PROCESS ID’S FOR ALL MATLAB INSTANCES
if ispc==1
% Execute the command to get a list of all running MATLAB processes
[status, result] = system(‘tasklist /FI "IMAGENAME eq matlab.exe" /FO LIST’);
% Check if the command was successful
if status == 0
% Process the result to extract PIDs
lines = strsplit(result, ‘n’);
pids = [];
for i = 1:length(lines)
line = strtrim(lines{i});
if startsWith(line, ‘PID:’)
pid = extractAfter(line, ‘PID:’);
pids = [pids; {pid}];
end
end
else
error(‘Failed to execute system command. Status: %d’, status);
end
else
% Execute the command to get PIDs of running MATLAB processes
[status, result] = system(‘pgrep matlab’);
% Check if the command was successful
if status == 0
% Process the result to extract PIDs
pids = strsplit(strtrim(result));
else
error(‘Failed to execute system command. Status: %d’, status);
end
end
endI am writing parfor loops and using folders as "lock files" for thread safety because mkdir is atomic and I believe is therefore the best option in MATLAB for a "lock file". However, I am getting the following error:
Error using makeLockFile
The system cannot find the path specified.
This error occurs on the line in makeLockFile containing
mkdir(pidPath);
Here is my code for makeLockFile (top), releaseLockFile (middle), and the helper function getMATLABPids (bottom):
What I am trying to do is:
Create a "lock folder". This folder has the same path as a .mat file (except the extension of course). If it exists, that means a MATLAB instance is using the corresponding .mat file.
If successful, create a subfolder named as the instance’s process ID, and return. This is where the issue is!
If unsuccessful, that means another process has locked this variable. In that case, remove any previous (now inactive) process’s lock subfolders that were left behind due to interrupted processes (this ensures that a process that is no longer running is not the sole reason for a lock). Then try deleting the lock folder. If that fails, pause and repeat.
When I encounter the above error, I can restart the process and then it runs all the way through, so I know it’s not a path correctness issue.
I don’t know how to debug this any further, I’ve been using disp() but can’t make heads or tails of why mkdir would ever fail in this context.
function [pidPath, status] = makeLockFile(lockFolderPath)
%% PURPOSE: CREATE A LOCK FOLDER, BECAUSE MKDIR IS ATOMIC.
% Clean the lockFolderPath and add the process ID.
[folderPath, folderName] = fileparts(lockFolderPath);
pid = num2str(feature(‘getpid’)); % Get MATLAB instance’s process ID.
lockFolderPath = [folderPath filesep folderName];
pidPath = [lockFolderPath filesep pid];
status=false;
while ~status
disp([‘PID: ‘ pid ‘ Lock Folder: ‘ lockFolderPath]);
[status, message, messageID] = mkdir(lockFolderPath);
try
if ~status
% The variable is locked.
% Get the listing of all folders in the variable folder
items = dir(lockFolderPath);
folderItems = items([items.isdir]);
folderNames = {folderItems.name};
folderNames(ismember(folderNames, {‘.’, ‘..’})) = []; % Remove ‘.’ and ‘..’
% Get all of the active MATLAB PID’s
pids = getMATLABPids();
% Remove the inactive (leftover from previous processes) PID’s
inactivePIDs = folderNames(~ismember(folderNames, pids));
for i = 1:length(inactivePIDs)
rmdir([lockFolderPath filesep inactivePIDs{i}]);
end
% Try releasing the lock folder. Has no effect if folder is not
% empty (i.e. active process folders present)
releaseLockFile(pidPath);
continue;
else
% Make the folder for this specific process ID
mkdir(pidPath);
end
catch ME
disp([‘ERROR PID: ‘ pid ‘ Lock Folder Path: ‘ pidPath]);
disp({ME.stack.file});
disp({ME.stack.name});
disp({ME.stack.line});
rethrow(ME);
end
end
end
function [] = releaseLockFile(pidPath)
%% PURPOSE: RELEASE THE LOCK FILE.
status = rmdir(pidPath); % PID-specific folder.
pid = feature(‘getpid’);
lockFolderPath = fileparts(pidPath); % Parent folder (the lock folder)
status = rmdir(lockFolderPath); % Remove the folder only if it is empty.
end
function [pids] = getMATLABPids()
%% PURPOSE: GET ALL OF THE PROCESS ID’S FOR ALL MATLAB INSTANCES
if ispc==1
% Execute the command to get a list of all running MATLAB processes
[status, result] = system(‘tasklist /FI "IMAGENAME eq matlab.exe" /FO LIST’);
% Check if the command was successful
if status == 0
% Process the result to extract PIDs
lines = strsplit(result, ‘n’);
pids = [];
for i = 1:length(lines)
line = strtrim(lines{i});
if startsWith(line, ‘PID:’)
pid = extractAfter(line, ‘PID:’);
pids = [pids; {pid}];
end
end
else
error(‘Failed to execute system command. Status: %d’, status);
end
else
% Execute the command to get PIDs of running MATLAB processes
[status, result] = system(‘pgrep matlab’);
% Check if the command was successful
if status == 0
% Process the result to extract PIDs
pids = strsplit(strtrim(result));
else
error(‘Failed to execute system command. Status: %d’, status);
end
end
end I am writing parfor loops and using folders as "lock files" for thread safety because mkdir is atomic and I believe is therefore the best option in MATLAB for a "lock file". However, I am getting the following error:
Error using makeLockFile
The system cannot find the path specified.
This error occurs on the line in makeLockFile containing
mkdir(pidPath);
Here is my code for makeLockFile (top), releaseLockFile (middle), and the helper function getMATLABPids (bottom):
What I am trying to do is:
Create a "lock folder". This folder has the same path as a .mat file (except the extension of course). If it exists, that means a MATLAB instance is using the corresponding .mat file.
If successful, create a subfolder named as the instance’s process ID, and return. This is where the issue is!
If unsuccessful, that means another process has locked this variable. In that case, remove any previous (now inactive) process’s lock subfolders that were left behind due to interrupted processes (this ensures that a process that is no longer running is not the sole reason for a lock). Then try deleting the lock folder. If that fails, pause and repeat.
When I encounter the above error, I can restart the process and then it runs all the way through, so I know it’s not a path correctness issue.
I don’t know how to debug this any further, I’ve been using disp() but can’t make heads or tails of why mkdir would ever fail in this context.
function [pidPath, status] = makeLockFile(lockFolderPath)
%% PURPOSE: CREATE A LOCK FOLDER, BECAUSE MKDIR IS ATOMIC.
% Clean the lockFolderPath and add the process ID.
[folderPath, folderName] = fileparts(lockFolderPath);
pid = num2str(feature(‘getpid’)); % Get MATLAB instance’s process ID.
lockFolderPath = [folderPath filesep folderName];
pidPath = [lockFolderPath filesep pid];
status=false;
while ~status
disp([‘PID: ‘ pid ‘ Lock Folder: ‘ lockFolderPath]);
[status, message, messageID] = mkdir(lockFolderPath);
try
if ~status
% The variable is locked.
% Get the listing of all folders in the variable folder
items = dir(lockFolderPath);
folderItems = items([items.isdir]);
folderNames = {folderItems.name};
folderNames(ismember(folderNames, {‘.’, ‘..’})) = []; % Remove ‘.’ and ‘..’
% Get all of the active MATLAB PID’s
pids = getMATLABPids();
% Remove the inactive (leftover from previous processes) PID’s
inactivePIDs = folderNames(~ismember(folderNames, pids));
for i = 1:length(inactivePIDs)
rmdir([lockFolderPath filesep inactivePIDs{i}]);
end
% Try releasing the lock folder. Has no effect if folder is not
% empty (i.e. active process folders present)
releaseLockFile(pidPath);
continue;
else
% Make the folder for this specific process ID
mkdir(pidPath);
end
catch ME
disp([‘ERROR PID: ‘ pid ‘ Lock Folder Path: ‘ pidPath]);
disp({ME.stack.file});
disp({ME.stack.name});
disp({ME.stack.line});
rethrow(ME);
end
end
end
function [] = releaseLockFile(pidPath)
%% PURPOSE: RELEASE THE LOCK FILE.
status = rmdir(pidPath); % PID-specific folder.
pid = feature(‘getpid’);
lockFolderPath = fileparts(pidPath); % Parent folder (the lock folder)
status = rmdir(lockFolderPath); % Remove the folder only if it is empty.
end
function [pids] = getMATLABPids()
%% PURPOSE: GET ALL OF THE PROCESS ID’S FOR ALL MATLAB INSTANCES
if ispc==1
% Execute the command to get a list of all running MATLAB processes
[status, result] = system(‘tasklist /FI "IMAGENAME eq matlab.exe" /FO LIST’);
% Check if the command was successful
if status == 0
% Process the result to extract PIDs
lines = strsplit(result, ‘n’);
pids = [];
for i = 1:length(lines)
line = strtrim(lines{i});
if startsWith(line, ‘PID:’)
pid = extractAfter(line, ‘PID:’);
pids = [pids; {pid}];
end
end
else
error(‘Failed to execute system command. Status: %d’, status);
end
else
% Execute the command to get PIDs of running MATLAB processes
[status, result] = system(‘pgrep matlab’);
% Check if the command was successful
if status == 0
% Process the result to extract PIDs
pids = strsplit(strtrim(result));
else
error(‘Failed to execute system command. Status: %d’, status);
end
end
end parallel computing, parallel computing toolbox, mkdir, path, lock, system MATLAB Answers — New Questions
