Skip to content

feat: add some params for rocksdb write#2538

Merged
AlexStocks merged 13 commits into
OpenAtomFoundation:unstablefrom
MalikHou:unstable
Mar 20, 2024
Merged

feat: add some params for rocksdb write#2538
AlexStocks merged 13 commits into
OpenAtomFoundation:unstablefrom
MalikHou:unstable

Conversation

@MalikHou

Copy link
Copy Markdown
Contributor

Add some new params for rocksdb

Background

Frequent write stop/stall in large-volume random write tests

Change

  • add level0_file_num_compaction_trigger
  • add level0_stop_writes_trigger
  • add level0_slowdown_writes_trigger
  • add min_write_buffer_number_to_merge
  • add max_bytes_for_level_base
  • add max_subcompactions

For the sake of compatibility, the parameters in MR still maintain the default parameters of Rocksdb. If necessary, you can test and adjust by yourself.

Result

In the case of a large number of random writes, the write/stall situation can be greatly alleviated through specific parameter combinations.

@github-actions github-actions Bot added the ✏️ Feature New feature or request label Mar 20, 2024
@MalikHou

MalikHou commented Mar 20, 2024

Copy link
Copy Markdown
Contributor Author

!!! This configuration is only reasonable for specific scenarios, please do not imitate it.!!!
eg:

db-instance-num:6
write-buffer-size : 512M
max-write-buffer-num : 3
max-write-buffer-size : 15G
min-write-buffer-number-to-merge : 1
level0-stop-writes-trigger : 10000
level0-slowdown-writes-trigger : 10000
level0-file-num-compaction-trigger : 4 
max_subcompactions :4
max-background-flushes : 6
max-background-compactions : 8
max-background-jobs : 12
target-file-size-base : 512M

before

WechatIMG129


after

WechatIMG693

@wangshao1

Copy link
Copy Markdown
Collaborator

我理解这些参数本身rocksdb已经支持了动态调节,您是否可以一块修改下pika代码,支持通过config set动态调节这些参数?

@Issues-translate-bot

Copy link
Copy Markdown

Bot detected the issue body's language is not English, translate it automatically.


I understand that rocksdb already supports dynamic adjustment of these parameters. Can you modify the pika code together to support dynamic adjustment of these parameters through config set?

Comment thread include/pika_conf.h Outdated
@MalikHou

Copy link
Copy Markdown
Contributor Author

我理解这些参数本身rocksdb已经支持了动态调节,您是否可以一块修改下pika代码,支持通过config set动态调节这些参数?

好的

@Issues-translate-bot

Copy link
Copy Markdown

Bot detected the issue body's language is not English, translate it automatically.


I understand that rocksdb already supports dynamic adjustment of these parameters. Can you modify the pika code together to support dynamic adjustment of these parameters through config set?

OK

@MalikHou

Copy link
Copy Markdown
Contributor Author

我理解这些参数本身rocksdb已经支持了动态调节,您是否可以一块修改下pika代码,支持通过config set动态调节这些参数?

pls view

@Issues-translate-bot

Copy link
Copy Markdown

Bot detected the issue body's language is not English, translate it automatically.


I understand that rocksdb already supports dynamic adjustment of these parameters. Can you modify the pika code together to support dynamic adjustment of these parameters through config set?

pls view

@Issues-translate-bot

Copy link
Copy Markdown

Bot detected the issue body's language is not English, translate it automatically.


Uploading screenshot 2024-03-20 16.58.15.png…

@MalikHou

Copy link
Copy Markdown
Contributor Author

截屏2024-03-20 16 58 15

@AlexStocks AlexStocks merged commit e6f9a53 into OpenAtomFoundation:unstable Mar 20, 2024
@AlexStocks AlexStocks added 3.5.5 and removed 3.5.4 labels May 15, 2024
bigdaronlee163 pushed a commit to bigdaronlee163/pika that referenced this pull request Jun 8, 2024
* feat:ad rocksdb write params

* add new param reload

---------

Co-authored-by: shengxin.hou <shengxin.hou@shopee.com>
cheniujh pushed a commit to cheniujh/pika that referenced this pull request Sep 24, 2024
* feat:ad rocksdb write params

* add new param reload

---------

Co-authored-by: shengxin.hou <shengxin.hou@shopee.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.5.5 4.0.0 ✏️ Feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants