If you expect (or find) that nodes are requesting the same data more than once, perhaps you could benefit from a caching strategy? Especially where some data is used much more often than others, so you can target only the most frequently-used information.
If the data is mutable, you also need a way to confirm that it hasn't changed since the last request that's less expensive than repeating the request.
This is further complicated if each node has its own separate cache. Depending on the nature of your system and task(s), you could consider adding a node dedicated to serving information between the processing nodes, and building a single cache on that node.
For an example of when that might be a good idea, let's suppose I retrieve some data from a remote data store over a low-bandwidth connection, and I have some task(s) requiring that data, which are distributed exclusively among local nodes. I definitely wouldn't want each node requesting information separately over that low-bandwidth connection, which another node might have previously requested. Since my local I/O is much less expensive than my I/O over the low-bandwidth connection, I might add a node between the processing nodes and the remote source that acts as an intermediate server. This node would take requests from the processing nodes, communicate with the remote data store, and cache frequently-requested data to minimize the use of that low-bandwidth connection.
The core concepts here that may be applicable to your specific case are:
- Eliminate or reduce redundant I/O;
- Take advantage of trade-offs between memory use and computation time;
- Not all I/O is created equal.