Introduce float 8 types by xadupre · Pull Request #14731 · microsoft/onnxruntime

xadupre · 2023-02-17T13:49:07Z

Description

The PR implements FloatE4M3FN, FloatE5M2, FloatE4MEFNUZ, FloatE5M2FNUZ as described in PR onnx/onnx#4805. It uses CUDA API to cast float/half to float8 if CUDA>=11.8, a custom implementation if CUDA<11.8.

It implements, Cast, QuantizeLinear, DequantizeLinear for all types on CPU, only for types FloatE4M3FN, FloatE5M2 on CUDA.
It extends the supported types for control flow operator, Shape, Reshape, Identity, If, Loop, Scan, Reshape
It implements Equal(19).
Cast, QuantizeLinear, DequantizeLinear operators now support a parameter saturate only valid for float 8 types. It is true by default. In that case, any value out of range is converted into the maximum float 8 value. If false, it is infinite.
QuantizeLinear, DequantizeLinear now supports multiple scales on CUDA (and ROCm by extension), scale = 1D tensor with one scale per channel

Motivation and Context

Supports latest onnx version.

Fixes AB#15395

snnn · 2023-06-01T21:22:36Z

@xadupre, the "Windows GPU Reduced Ops CI Pipeline" fails since this change. Would you please help fix it?

### Description The PR implements FloatE4M3FN, FloatE5M2, FloatE4MEFNUZ, FloatE5M2FNUZ as described in PR onnx/onnx#4805. It uses CUDA API to cast float/half to float8 if CUDA>=11.8, a custom implementation if CUDA<11.8. * It implements, Cast, QuantizeLinear, DequantizeLinear for all types on CPU, only for types FloatE4M3FN, FloatE5M2 on CUDA. * It extends the supported types for control flow operator, Shape, Reshape, Identity, If, Loop, Scan, Reshape * It implements Equal(19). * Cast, QuantizeLinear, DequantizeLinear operators now support a parameter `saturate` only valid for float 8 types. It is true by default. In that case, any value out of range is converted into the maximum float 8 value. If false, it is infinite. * QuantizeLinear, DequantizeLinear now supports multiple scales on CUDA (and ROCm by extension), scale = 1D tensor with one scale per channel ### Motivation and Context Supports latest onnx version. Fixes [AB#15395](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/15395) --------- Co-authored-by: Xavier Dupre <xadupre@microsoft.com@orttrainingdev8.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net> Co-authored-by: Randy Shuai <rashuai@microsoft.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: Scott McKay <Scott.McKay@microsoft.com>

### Description onnx/onnx#6318 and onnx/onnx#6283 added FP4 support to ONNX. This change introduces the FP4 type in ORT and adds type support to one relevant operator (`Cast`) as a proof-of-concept for the type integration into ORT. More op support will be added on a need-basis. This change took inspiration from the following PRs: #14731 #22228 #20362 Some notes: 1) Only `tensor` type gets support for FP4 initially. Secondary types like `seq(tensor)`, `sparse_tensor`, `optional` do not get support (so as to not introduce unnecessary bloat to the framework without a solid use-case) 2) Flatbuffer related files receive no updates in this PR ### Motivation and Context Be able to run FP4 models with ORT

onnx/onnx#6318 and onnx/onnx#6283 added FP4 support to ONNX. This change introduces the FP4 type in ORT and adds type support to one relevant operator (`Cast`) as a proof-of-concept for the type integration into ORT. More op support will be added on a need-basis. This change took inspiration from the following PRs: microsoft#14731 microsoft#22228 microsoft#20362 Some notes: 1) Only `tensor` type gets support for FP4 initially. Secondary types like `seq(tensor)`, `sparse_tensor`, `optional` do not get support (so as to not introduce unnecessary bloat to the framework without a solid use-case) 2) Flatbuffer related files receive no updates in this PR Be able to run FP4 models with ORT

* Support fp4 type in ORT (microsoft#25767) onnx/onnx#6318 and onnx/onnx#6283 added FP4 support to ONNX. This change introduces the FP4 type in ORT and adds type support to one relevant operator (`Cast`) as a proof-of-concept for the type integration into ORT. More op support will be added on a need-basis. This change took inspiration from the following PRs: microsoft#14731 microsoft#22228 microsoft#20362 Some notes: 1) Only `tensor` type gets support for FP4 initially. Secondary types like `seq(tensor)`, `sparse_tensor`, `optional` do not get support (so as to not introduce unnecessary bloat to the framework without a solid use-case) 2) Flatbuffer related files receive no updates in this PR Be able to run FP4 models with ORT * [Core] Fix debug node input output compilation after Fp4 support was enabled in ORT (microsoft#25940) ### Description As title ### Motivation and Context Follow-up fixes to microsoft#25767 * Link FP4 types between OnnxRT and MIGraphX APIs Do this so that MIGraphX can take in fp4 types from input/output tensors and then use that to perform an inference via the MIGraphX API. --------- Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com>

### Description onnx/onnx#6318 and onnx/onnx#6283 added FP4 support to ONNX. This change introduces the FP4 type in ORT and adds type support to one relevant operator (`Cast`) as a proof-of-concept for the type integration into ORT. More op support will be added on a need-basis. This change took inspiration from the following PRs: #14731 #22228 #20362 Some notes: 1) Only `tensor` type gets support for FP4 initially. Secondary types like `seq(tensor)`, `sparse_tensor`, `optional` do not get support (so as to not introduce unnecessary bloat to the framework without a solid use-case) 2) Flatbuffer related files receive no updates in this PR ### Motivation and Context Be able to run FP4 models with ORT

xadupre and others added 6 commits February 13, 2023 16:06

cast

ebd515b

a few more steps

e9fd662

float 8

0f44b84

add floate4m3, floate5m2 everywhere

5968590

Merge branch 'main' of https://github.com/microsoft/onnxruntime into f8

beae359

first unit test with cast

dc226b0

xadupre requested a review from a team as a code owner February 17, 2023 13:49

github-advanced-security AI found potential problems Feb 17, 2023

View reviewed changes

Comment thread onnxruntime/test/python/onnxruntime_test_float8.py Fixed

Xavier Dupre and others added 8 commits February 17, 2023 16:09

change onnx source

22345bb

add two new float 8 types

39ad5d3

fix float8.h

2e61452

remove onnx submodule

8eeb826

delete onnx submodule

770e4cf

add submodule onnx

8f387e1

fix build issues

84337b3

isort

728a24a

github-advanced-security AI found potential problems Mar 15, 2023

View reviewed changes

Comment thread onnxruntime/test/python/onnxruntime_test_float8.py Fixed

xadupre added 13 commits March 24, 2023 15:29

Merge branch 'main' of https://github.com/microsoft/onnxruntime into f8

da8ed7f

check ReferenceEvaluator as well

7b884b0

better message

4555d1d

new mssage

4f56a6d

import test

1b65a33

split unit test

ef0976a

update float8.h

8f30839

float8

606f452

error

39276aa

error

a306351

error

95202d6

error

1323339

error

477c8a1

xadupre added 7 commits May 26, 2023 13:55

add flag DISABLE_FLOAT8_TYPES

1ef646e

lint

05bf92a

lint

96748a6

fix disable float8

ff8feca

missing S

d58c9cb

disable if on rocm

4f6ea36

fix two compilation issues

e54e85c

jcwchen mentioned this pull request May 26, 2023

Skip not implemented failures for tests with ORT 1.15 onnx/onnx#5261

Merged

edgchen1 approved these changes May 26, 2023

View reviewed changes

edgchen1 previously approved these changes May 26, 2023

View reviewed changes

Comment thread include/onnxruntime/core/framework/float8.h

Comment thread include/onnxruntime/core/framework/float8.h

Comment thread include/onnxruntime/core/framework/float8.h

Comment thread include/onnxruntime/core/framework/float8.h

xadupre added 2 commits May 27, 2023 10:30

Merge branch 'main' of https://github.com/microsoft/onnxruntime into f8

beed7a5

enable float8 tests

b2f6b6d

xadupre dismissed edgchen1’s stale review via b2f6b6d May 27, 2023 10:33

xadupre added 4 commits May 27, 2023 16:13

disable float8 types for qnn, dnnl

bfcd61b

avoid disabling tests twice

d1e9eb1

change opset in function generate_size_op_test

cf5faa7

fix json and lint

ccc6876

askhade approved these changes May 30, 2023

View reviewed changes

edgchen1 approved these changes May 30, 2023

View reviewed changes

snnn approved these changes May 30, 2023

View reviewed changes

jchen351 approved these changes May 30, 2023

View reviewed changes

askhade merged commit e726151 into microsoft:main May 30, 2023

hariharans29 mentioned this pull request Aug 21, 2025

[Core] Support fp4 type in ORT #25767

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introduce float 8 types#14731

Introduce float 8 types#14731
askhade merged 254 commits into
microsoft:mainfrom
xadupre:f8

xadupre commented Feb 17, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snnn commented Jun 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

Uh oh!

Conversation

xadupre commented Feb 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snnn commented Jun 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

xadupre commented Feb 17, 2023 •

edited

Loading