Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
434 views
in Technique[技术] by (71.8m points)

python 3.x - How to divide two large numpy matrix elemtwise without getting killed?

I have two matrices A,B both of shape 150000 X 150000

I want to divide each element of A with each element of B, element wise. The way I currently do it is -

res=A/B

I do get the output for small matrices, but for large matrices as mine. The process gets killed. Any suggestions on how to do this efficiently ?

sample data

A =
[
[2.2,3.3,4.4]
[2.2,3.3,4.4]
[2.2,3.3,4.4]]

B=
[
[1,3.3,4.4]
[2.2,1,4.4]
[2.2,3.3,1]]

res = 
[
[2.2,1,1]
[1,3.3,1]
[1,1,4.4]]



This is a 3 X 3 matrix, I'm working with 150000 X 150000 matrix


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

you could try to use pandas and set the type of the values to something small memory wise, or at least check the memory allocated for a value, usually Python uses float64 or so, which is in some cases way too much.

use

pd.to_numeric(s, errors='coerce') 

or

pd.to_numeric(column, downcast='integer')

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...