1 year ago
#357824
khubull
How to define remote Ray function on the driver instead?
I have nested remote Ray functions defined inside a class as methods to the class. The code runs properly but I get the following warning...
WARNING import_thread.py:131 -- The remote function 'process.d2' has been exported 100 times. It's possible that this warning is accidental, but this may indicate that the same remote function is being defined repeatedly from within many tasks and exported to all of the workers. This can be a performance issue and can be resolved by defining the remote function on the driver instead. See https://github.com/ray-project/ray/issues/6240 for more discussion.
I referred to the github link but unfortunately I couldn't make sense of what to do here.
My code is structured like so...
class SomeClass():
@ray.remote
def main(self):
while True:
tasks = []
for _ in range(10):
tasks.append(ray.get(self.d1.remote(self)))
ray.get(tasks)
@ray.remote
def d1(self):
for _ in range(10):
self.d2.remote(self)
@ray.remote
def d2(self):
for _ in range(10):
self.d3.remote(self)
@ray.remote
def d3(self):
self.do_something()
This is just an example. The reason for doing all these nested remote functions in the actual code is because each nested function access a different object from the same class attributes. I have main defined as a remote function as well because I am running the whole process in parallel with some other code.
How should I go about redefining the class methods on the driver instead? Would they need to be defined before I create an instance of the class?
python
ray
0 Answers
Your Answer