python (65.2k questions)

javascript (44.3k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (13k questions)

Questions - precision

Intel 8087 floating point numbers

Good morning, I have to change the constant values "wired to code" for an old machine based on intel 8086 and intel 8087 coprocessor. I have the source code, written in the RTL/2 language. U...
test-img

Andak

assembly

floating-point

precision

intel

x87

Votes: 0

Answers: 0

numpy.linalg.solve for Decimal in Python

Is there any way how to solve equations when a matrix A and a vector b is composed of decimal.Decimals? My A: array([[Decimal('-5266125828.168885444558615257'), Decimal('11312418445.6961242883...
test-img

Dominik Bolerác

python

numpy

precision

equation-solving

Votes: 0

Answers: 1

Latest Answer

I don't see a reason why you couldn't use the code in from here. You said some lines don't work with decimals, but with some small adjustment it seems to work just fine. b = b.reshape(-1, 1) n = len(A...
test-img

Faber

How to use exact values of axes' using matplotlib in python

I have taken some graphics using matplotlib in python. But, y values cannot be understood easily. The picture is given below enter image description here How can I change these coordinate scripts. How...
test-img

Zohidjon

python

matplotlib

precision

axes

Votes: 0

Answers: 0

Find least significant digit in a double in Python

I have a lot of financial data stored as floating point doubles and I'm trying to find the least significant digit so that I can convert the data to integers with exponent. All the data is finite, e.g...
test-img

Spedley

python

precision

rounding-error

Votes: 0

Answers: 2

Latest Answer

You have a couple of options, Firstly and most preferably, use the stdlib Decimal, not builtin float This fixes most errors related to floats but not the infamous 0.1 + 0.2 = 0.3...4 from decimal impo...
test-img

RossM

Posts

Questions

Blogs

Jobs