I've been reaching the limits of the clustering program, as I am trying to cluster many years of contiguous trajectories. I have hit some sort of limit to the CLUSTER file size, and was wondering about the cause.
Thus far, I have been successful in clustering 41 years of daily trajectories (120 hours, 12 h intervals, skip = 2). This leads to a total of 7487 passes in the clustering program. I have to assign mean trajectories myself (>5000 trajectories, beyond the trajmean limit), but even beyond this number of trajectories I don't have issues with memory.
When trying a clustering attempt that requires ~8033 passes (44 years of data, supplied as above; 120, 12, 2), the 'CLUSTER' file abruptly ends at line 60,173,034 (and is 2,081,279 KB in size), which is partway through a pass (7487, funnily enough - but I think the match to the previous attempt is not significant). Is this line # or file size at some limit for data output from the cluster program? The cluster.exe process completes without error, and I can view the DELPCT data, so presumably the program is running fine past this point. Yet, the cluster file ends abrubtly and does not seem to contain any data after this point. As a result, I cannot use the CLUSTER file to assign trajectories to clusters myself as the data is missing after line 60,173,034.
So, my question is: is there a set limit to the cluster program's output ability (in the CLUSTER file), perhaps in terms of file size (around 2 GB)?
Thus far, I have been successful in clustering 41 years of daily trajectories (120 hours, 12 h intervals, skip = 2). This leads to a total of 7487 passes in the clustering program. I have to assign mean trajectories myself (>5000 trajectories, beyond the trajmean limit), but even beyond this number of trajectories I don't have issues with memory.
When trying a clustering attempt that requires ~8033 passes (44 years of data, supplied as above; 120, 12, 2), the 'CLUSTER' file abruptly ends at line 60,173,034 (and is 2,081,279 KB in size), which is partway through a pass (7487, funnily enough - but I think the match to the previous attempt is not significant). Is this line # or file size at some limit for data output from the cluster program? The cluster.exe process completes without error, and I can view the DELPCT data, so presumably the program is running fine past this point. Yet, the cluster file ends abrubtly and does not seem to contain any data after this point. As a result, I cannot use the CLUSTER file to assign trajectories to clusters myself as the data is missing after line 60,173,034.
So, my question is: is there a set limit to the cluster program's output ability (in the CLUSTER file), perhaps in terms of file size (around 2 GB)?
Statistics: Posted by Harris_MRP — February 20th, 2025, 9:11 pm