Feature/support older intel mac book pro with gcc 13 by nezda · Pull Request #1085 · intel/intel-extension-for-transformers

nezda · 2023-12-27T11:37:41Z

Type of Change

Initiated discussion #1081 to while trying to get this project to compile on a 2019 Intel-based MacBook Pro.

Currently segfaults when running inference on mistralai/Mistral-7B-v0.1 quantized as WeightOnlyQuantConfig(ggml=True)

feature or bug fix or documentation or others
API changed or not

Description

detail description
JIRA ticket: xxx

Expected Behavior & Potential Risk

the expected behavior that triggered by this PR

How has this PR been tested?

how to reproduce the test (including hardware information)

2019 Intel-based MacBook Pro 2.3 GHz 8-Core Intel Core i9 on macOS 13.5 (22G74) using brew install gcc (13.2.0), export CC=/usr/local/opt/gcc/bin/gcc-13, export CXX=/usr/local/opt/gcc/bin/g++-13 and building with pip install -v .

Dependency Change?

any library dependency introduced or removed

…en/products/docs/accelerator-engines/what-is-intel-amx.html) anyway per intel#1081 (comment) Signed-off-by: Luke Nezda <lnezda@gmail.com>

…ツ)_/¯ Signed-off-by: Luke Nezda <lnezda@gmail.com>

Signed-off-by: Luke Nezda <lnezda@gmail.com>

…ntel#1081 (comment) Signed-off-by: Luke Nezda <lnezda@gmail.com>

…in this scope; did you mean 'SYS_sysctlbyname'? ld: library not found for -lrt collect2: error: ld returned 1 exit status [2/164] Building CXX object models/mpt/CMakeFiles/mpt.dir/__/model_utils/util.cpp.o FAILED: models/mpt/CMakeFiles/mpt.dir/__/model_utils/util.cpp.o /usr/local/opt/gcc/bin/g++-13 -DNE_GELU_USE_VEC -DNE_SIMD_VEC_DOT_F16 -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/core/. -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas -O3 -DNDEBUG -std=c++17 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk -fPIC -mf16c -mfma -mavx -mavx2 -fopenmp -MD -MT models/mpt/CMakeFiles/mpt.dir/__/model_utils/util.cpp.o -MF models/mpt/CMakeFiles/mpt.dir/__/model_utils/util.cpp.o.d -o models/mpt/CMakeFiles/mpt.dir/__/model_utils/util.cpp.o -c /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp: In function 'int32_t get_num_physical_cores()': /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp:36:16: error: 'sysctlbyname' was not declared in this scope; did you mean 'SYS_sysctlbyname'? 36 | int result = sysctlbyname("hw.perflevel0.physicalcpu", &num_physical_cores, &len, NULL, 0); | ^~~~~~~~~~~~ | SYS_sysctlbyname [3/164] Building CXX object models/llama/CMakeFiles/llama.dir/__/model_utils/util.cpp.o FAILED: models/llama/CMakeFiles/llama.dir/__/model_utils/util.cpp.o /usr/local/opt/gcc/bin/g++-13 -DNE_GELU_USE_VEC -DNE_SIMD_VEC_DOT_F16 -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/core/. -I/Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas -O3 -DNDEBUG -std=c++17 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk -fPIC -mf16c -mfma -mavx -mavx2 -fopenmp -MD -MT models/llama/CMakeFiles/llama.dir/__/model_utils/util.cpp.o -MF models/llama/CMakeFiles/llama.dir/__/model_utils/util.cpp.o.d -o models/llama/CMakeFiles/llama.dir/__/model_utils/util.cpp.o -c /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp: In function 'int32_t get_num_physical_cores()': /Users/nezda/code/itrex/intel_extension_for_transformers/llm/runtime/graph/models/model_utils/util.cpp:36:16: error: 'sysctlbyname' was not declared in this scope; did you mean 'SYS_sysctlbyname'? 36 | int result = sysctlbyname("hw.perflevel0.physicalcpu", &num_physical_cores, &len, NULL, 0); | ^~~~~~~~~~~~ | SYS_sysctlbyname Signed-off-by: Luke Nezda <lnezda@gmail.com>

Signed-off-by: Luke Nezda <lnezda@gmail.com>

…ents Signed-off-by: Luke Nezda <lnezda@gmail.com>

Signed-off-by: Luke Nezda <lnezda@gmail.com>

…s too Signed-off-by: Luke Nezda <lnezda@gmail.com>

…n_hidden_size, rope_scale, rope_theta - tried to fix write_vocab_only but Params here missing lots too Signed-off-by: Luke Nezda <lnezda@gmail.com>

DDEle

Great work!

DDEle · 2023-12-29T16:46:41Z

intel_extension_for_transformers/llm/runtime/graph/scripts/convert_llama.py

@zhenwei-intel Can you check it and check if there are similar problems in other models?

DDEle · 2023-12-29T16:47:10Z

intel_extension_for_transformers/llm/runtime/graph/scripts/convert_mistral.py

@intellinjun Can you check it and check if there are similar problems in other models?

requirements.txt

intel_extension_for_transformers/llm/runtime/graph/scripts/requirements/common.txt

examples/huggingface/pytorch/question-answering/pruning/group_lasso/modeling.py

DDEle · 2023-12-29T16:58:14Z

docker/README.md

 ```bash
 docker pull intel/ai-tools:itrex-1.3.0
-docker pull intel/ai-tools:itrex-devel-1.3.0
+docker pull intel/ai-tools:itrex-1.3.0-devel


Thanks for the correction. I'm not familiar with docker but it seems that you are right. To be confirmed by @tylertitsworth?

DDEle · 2023-12-29T17:11:09Z

intel_extension_for_transformers/llm/library/jblas/jblas/kernel_jit.h

copying your error msg

/Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/kernel_jit.h:1231:7: error: call of overloaded 'db(uintptr_t, long unsigned int)' is ambiguous 1231 | db(reinterpret_cast<uintptr_t>(nullptr), sizeof(intptr_t)); // case 0 should never occur | ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In file included from /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/jit_base.h:19, from /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/jit_blas_storage.h:15: /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/xbyak/xbyak.h:1133:14: note: candidate: 'void Xbyak::CodeArray::db(uint64_t, size_t)' 1133 | void db(uint64_t code, size_t codeSize) | ^~ /Users/nezda/code/itrex/intel_extension_for_transformers/llm/library/jblas/jblas/xbyak/xbyak.h:1129:14: note: candidate: 'void Xbyak::CodeArray::db(const uint8_t*, size_t)' 1129 | void db(const uint8_t *code, size_t codeSize) | ^~ ninja: build stopped: subcommand failed.

I actually want the first case. Can you try reinterpret_cast<uint64_t>(nullptr) (or just UINT64_C(0) if fails)?

For curiosity, what is uintptr_t in your side?

Hmm, original code gcc rejected with above was

db(reinterpret_cast<uintptr_t>(nullptr), sizeof(intptr_t)); // case 0 should never occur

my wild guess "fix" that is committed in this branch atm

db(nullptr, sizeof(intptr_t)); // case 0 should never occur

you've just requested

db(reinterpret_cast<uint64_t>(nullptr), sizeof(intptr_t)); // case 0 should never occur

which seems to compile fine ¯\_(ツ)_/¯ : I will push the change.

I re-confirmed the original doesn't compile with my gcc 13.2.0. I also confirmed I still get segfaults after re-building everything with this change.

As for what uintptr_t is in my env, I really don't know how to answer that - I asked Bard "What is the difference between uintptr_t and uint64_t ?" and it sounded fairly pedantic - like they could differ if we had 128 bit pointers or something.

…uirements/common.txt Co-authored-by: Yi DING <yi1.ding@intel.com> Signed-off-by: Luke Nezda <lnezda@gmail.com>

Co-authored-by: Yi DING <yi1.ding@intel.com> Signed-off-by: Luke Nezda <lnezda@gmail.com>

@DDEle

…ツ)_/¯ - adjust per code review @DDEle Signed-off-by: Luke Nezda <lnezda@gmail.com>

VincyZhang · 2024-05-13T06:06:50Z

@nezda @DDEle any update about this PR?

nezda force-pushed the feature/support-older-Intel-MacBook-Pro-with-gcc-13 branch 4 times, most recently from 63bd546 to 9c57593 Compare December 29, 2023 05:02

nezda added 13 commits December 29, 2023 09:32

Intel Mac's don't support [AMX](https://www.intel.com/content/www/us/…

00a4adb

…en/products/docs/accelerator-engines/what-is-intel-amx.html) anyway per intel#1081 (comment) Signed-off-by: Luke Nezda <lnezda@gmail.com>

gcc-13 said reinterpret_cast<uintptr_t>(nullptr) was ambiguous ¯\_(…

5f4e039

…ツ)_/¯ Signed-off-by: Luke Nezda <lnezda@gmail.com>

fix docker command

1b14304

Signed-off-by: Luke Nezda <lnezda@gmail.com>

disable the build of qbits and the deprecated executor on darwin per i…

f02ed60

…ntel#1081 (comment) Signed-off-by: Luke Nezda <lnezda@gmail.com>

don't link to -rt for APPLE either per intel#1081 (reply in thread)

46b64bd

Signed-off-by: Luke Nezda <lnezda@gmail.com>

typos

8f99e30

Signed-off-by: Luke Nezda <lnezda@gmail.com>

trying to make Intel mac work... (this didn't help?)

9087bfb

Signed-off-by: Luke Nezda <lnezda@gmail.com>

hacked requirements to strip "+cpu" suffix from torch==2.1.0 requirem…

5fe2be0

…ents Signed-off-by: Luke Nezda <lnezda@gmail.com>

hacked requirements to strip "+cpu" suffix from torch==2.1.0 requirem…

aae9843

…ents Signed-off-by: Luke Nezda <lnezda@gmail.com>

added missing rope_scale param

beb027b

Signed-off-by: Luke Nezda <lnezda@gmail.com>

attempt to deal with missing Linux-only os.sched_getaffinity for maco…

aaee454

…s too Signed-off-by: Luke Nezda <lnezda@gmail.com>

I think f_norm_eps should be rms_norm_eps; guessed still missing ff…

bfec292

…n_hidden_size, rope_scale, rope_theta - tried to fix write_vocab_only but Params here missing lots too Signed-off-by: Luke Nezda <lnezda@gmail.com>

nezda force-pushed the feature/support-older-Intel-MacBook-Pro-with-gcc-13 branch from c1f89be to bfec292 Compare December 29, 2023 15:54

DDEle reviewed Dec 29, 2023

View reviewed changes

nezda and others added 3 commits December 29, 2023 11:47

Update intel_extension_for_transformers/llm/runtime/graph/scripts/req…

4ff62db

…uirements/common.txt Co-authored-by: Yi DING <yi1.ding@intel.com> Signed-off-by: Luke Nezda <lnezda@gmail.com>

on mac (darwin) strip "+cpu" suffix from torch==2.1.0 requirements

cd05487

Co-authored-by: Yi DING <yi1.ding@intel.com> Signed-off-by: Luke Nezda <lnezda@gmail.com>

gcc-13 said reinterpret_cast<uintptr_t>(nullptr) was ambiguous ¯\_(…

05b66db

…ツ)_/¯ - adjust per code review @DDEle Signed-off-by: Luke Nezda <lnezda@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/support older intel mac book pro with gcc 13#1085

Feature/support older intel mac book pro with gcc 13#1085
nezda wants to merge 16 commits intointel:mainfrom
nezda:feature/support-older-Intel-MacBook-Pro-with-gcc-13

nezda commented Dec 27, 2023 •

edited

Loading

Uh oh!

DDEle left a comment

Uh oh!

DDEle Dec 29, 2023

Uh oh!

DDEle Dec 29, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DDEle Dec 29, 2023

Uh oh!

DDEle Dec 29, 2023

Uh oh!

nezda Dec 29, 2023

Uh oh!

VincyZhang commented May 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

nezda commented Dec 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of Change

Description

Expected Behavior & Potential Risk

How has this PR been tested?

Dependency Change?

Uh oh!

DDEle left a comment

Choose a reason for hiding this comment

Uh oh!

DDEle Dec 29, 2023

Choose a reason for hiding this comment

Uh oh!

DDEle Dec 29, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DDEle Dec 29, 2023

Choose a reason for hiding this comment

Uh oh!

DDEle Dec 29, 2023

Choose a reason for hiding this comment

Uh oh!

nezda Dec 29, 2023

Choose a reason for hiding this comment

Uh oh!

VincyZhang commented May 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

nezda commented Dec 27, 2023 •

edited

Loading