1 year ago

#343519

test-img

Yourim Lee

How I import class module in concurrent.futures ProcessPoolExecutor map?

I try to get sequence alignment score using processpoolexecutor in concurrent.futures module. But I have an error below.

from concurrent.futures import ProcessPoolExecutor
from Bio import Align

def main():
    aligner = Align.PairwiseAligner()

    with ProcessPoolExecutor(max_workers=3) as executor:
        result = list(executor.map(aligner.align, ['fasd', 'fdas'], ['gdfs', 'fdsa']))
    print(result)

if __name__ == '__main__':
    main()

I got a TypeError. Error is

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/bion/.conda/envs/project_env/lib/python3.9/concurrent/futures/process.py", line 208, in _sendback_result
    result_queue.put(_ResultItem(work_id, result=result,
  File "/home/bion/.conda/envs/project_env/lib/python3.9/multiprocessing/queues.py", line 372, in put
    obj = _ForkingPickler.dumps(obj)
  File "/home/bion/.conda/envs/project_env/lib/python3.9/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'Path generator' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/bion/playground/notebooks/multiprocessing_test.py", line 12, in <module>
    main()
  File "/home/bion/playground/notebooks/multiprocessing_test.py", line 8, in main
    result = list(executor.map(aligner.align, ['fasd', 'fdas'], ['gdfs', 'fdsa']))
  File "/home/bion/.conda/envs/project_env/lib/python3.9/concurrent/futures/process.py", line 559, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/home/bion/.conda/envs/project_env/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/home/bion/.conda/envs/project_env/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/home/bion/.conda/envs/project_env/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
TypeError: cannot pickle 'Path generator' object

I think This issue is that imported class not function. Do you have any idea?

I imported function. It works

def foo(a, b):
    return a+b


with ProcessPoolExecutor() as executor:
    result = list(executor.map(foo, [1, 2, 3], [4, 5, 6]))
print(result)

my code flow is this

seqA = 'asdf'
seqB = 'asgf'

seq_length = min(len(seqA), len(seqB))
alignments = aligner.align(seqA, seqB)
similarity_score = alignments[0].score / (seq_length * aligner.match_score)

and I want to run many number of seqA and seqB sequences_A = ['adf', 'agsdaf',...] sequences_B = ['gafs,', 'asgdasdf', 'afasdf',...]

python

class

dictionary

biopython

multiprocess

0 Answers

Your Answer

Accepted video resources