Skip to content

Commit 0bd89fc

Browse files
tinswzytinswzy
committed
enhance: Use milvusStorageV2 to support various cloudstorage #64
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
1 parent fd5f377 commit 0bd89fc

99 files changed

Lines changed: 51321 additions & 30 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 4 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,4 @@
1-
# If you prefer the allow list template instead of the deny list, see community template:
2-
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
3-
#
4-
# Binaries for programs and plugins
5-
*.exe
6-
*.exe~
7-
*.dll
8-
*.so
9-
*.dylib
10-
bin/*
11-
12-
**/.idea/*
13-
**/.vscode/*
14-
15-
# Test binary, built with `go test -c`
16-
*.test
17-
18-
# Output of the go coverage tool, specifically when used with LiteIDE
19-
*.out
20-
21-
# Dependency directories (remove the comment below to include it)
22-
# vendor/
23-
24-
# Go workspace file
25-
go.work
26-
go.work.sum
27-
28-
# env file
29-
.env
1+
**/cmake_build/*
2+
**/.idea
3+
**/.vscode
4+
**/external/output

‎Makefile‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,42 @@ test: build
5454

5555
clean:
5656
rm -f $(BIN_DIR)/*
57+
rm -rf $(PWD)/cmake_build
58+
rm -rf $(PWD)/external/output
5759

5860
docker:
5961
docker build -t woodpecker:latest .
62+
63+
# ==================== CPP BUILD ========
64+
INSTALL_PATH := $(PWD)/bin
65+
LIBRARY_PATH := $(PWD)/lib
66+
OS := $(shell uname -s)
67+
mode = Release
68+
use_asan = OFF
69+
ifeq ($(USE_ASAN), ON)
70+
use_asan = ${USE_ASAN}
71+
CGO_LDFLAGS := $(shell go env CGO_LDFLAGS) -fno-stack-protector -fno-omit-frame-pointer -fno-var-tracking -fsanitize=address
72+
CGO_CFLAGS := $(shell go env CGO_CFLAGS) -fno-stack-protector -fno-omit-frame-pointer -fno-var-tracking -fsanitize=address
73+
MILVUS_GO_BUILD_TAGS := $(MILVUS_GO_BUILD_TAGS),use_asan
74+
endif
75+
76+
use_opendal = OFF
77+
ifdef USE_OPENDAL
78+
use_opendal = ${USE_OPENDAL}
79+
endif
80+
81+
# default git branch is dev
82+
export GIT_BRANCH=dev
83+
84+
ifeq (${ENABLE_AZURE}, false)
85+
AZURE_OPTION := -Z
86+
endif
87+
88+
89+
build-3rdparty:
90+
@echo "Build thirdparty using conanfile.py ..."
91+
@(env bash $(PWD)/scripts/3rdparty_build.sh -o ${use_opendal} -t ${mode})
92+
93+
build-external: build-3rdparty
94+
@echo "Building Milvus external cpp library ..."
95+
@(env bash $(PWD)/scripts/external_build.sh -t ${mode} -a ${use_asan} ${AZURE_OPTION} -o ${use_opendal} )

‎common/external/README.md‎

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
# Arrow FileSystem C Interface Documentation
2+
3+
This document describes how to use Woodpecker to unify the access of various types of cloud storage via the external dependency milvus-storage.
4+
5+
## Build the C++ Library
6+
7+
First, you need to build the C++ parts of the code:
8+
9+
```bash
10+
cd /path/to/woodpecker
11+
make build-external
12+
```
13+
14+
After building, the library files will be generated in the `external/output/lib/` directory.
15+
16+
## Environment Variable Configuration
17+
18+
Before running Go tests, you need to set up the following environment variables:
19+
20+
```bash
21+
export PKG_CONFIG_PATH=/path/to/woodpecker/external/output/lib/pkgconfig
22+
export CGO_LDFLAGS="-Wl,-rpath,/path/to/woodpecker/external/output/lib"
23+
```
24+
25+
Or set them directly when running tests:
26+
27+
```bash
28+
PKG_CONFIG_PATH=/path/to/woodpecker/external/output/lib/pkgconfig \
29+
CGO_LDFLAGS="-Wl,-rpath,/path/to/woodpecker/external/output/lib" \
30+
go test -v ./common/initcore -run TestStorageV2LocalFS
31+
```
32+
33+
## Go API Usage
34+
35+
### Initializing the File System
36+
37+
#### Local File System
38+
39+
```go
40+
import "github.com/zilliztech/woodpecker/common/initcore"
41+
42+
// Initialize local file system
43+
err := initcore.InitLocalArrowFileSystem("/tmp/storage")
44+
if err != nil {
45+
log.Fatal(err)
46+
}
47+
defer initcore.CleanFileSystem()
48+
```
49+
50+
#### Remote File System (MinIO/S3)
51+
52+
```go
53+
import (
54+
"github.com/zilliztech/woodpecker/common/config"
55+
"github.com/zilliztech/woodpecker/common/initcore"
56+
)
57+
58+
// Load configuration
59+
cfg, err := config.NewConfiguration("config/woodpecker.yaml")
60+
if err != nil {
61+
log.Fatal(err)
62+
}
63+
64+
// Configure MinIO settings
65+
cfg.Woodpecker.Storage.Type = "minio"
66+
cfg.Minio.Address = "localhost:9000"
67+
cfg.Minio.BucketName = "my-bucket"
68+
cfg.Minio.AccessKeyID = "minioadmin"
69+
cfg.Minio.SecretAccessKey = "minioadmin"
70+
71+
// Initialize
72+
err = initcore.InitStorageV2FileSystem(cfg)
73+
if err != nil {
74+
log.Fatal(err)
75+
}
76+
defer initcore.CleanFileSystem()
77+
```
78+
79+
### File Operations
80+
81+
#### Write File
82+
83+
```go
84+
data := []byte("Hello, Woodpecker!")
85+
metadata := map[string]string{
86+
"author": "test_user",
87+
"version": "1.0",
88+
}
89+
90+
err := initcore.WriteFile("test.txt", data, metadata)
91+
if err != nil {
92+
log.Fatal(err)
93+
}
94+
```
95+
96+
#### Read File
97+
98+
```go
99+
data, err := initcore.ReadFile("test.txt")
100+
if err != nil {
101+
log.Fatal(err)
102+
}
103+
fmt.Printf("File content: %s\n", string(data))
104+
```
105+
106+
#### Get File Info
107+
108+
```go
109+
info, err := initcore.GetFileInfo("test.txt")
110+
if err != nil {
111+
log.Fatal(err)
112+
}
113+
fmt.Printf("File size: %d bytes\n", info.Size)
114+
fmt.Printf("Metadata: %v\n", info.Metadata)
115+
```
116+
117+
#### Get File Size
118+
119+
```go
120+
size, err := initcore.GetFileSize("test.txt")
121+
if err != nil {
122+
log.Fatal(err)
123+
}
124+
fmt.Printf("File size: %d bytes\n", size)
125+
```
126+
127+
#### Check File Existence
128+
129+
```go
130+
exists, err := initcore.FileExists("test.txt")
131+
if err != nil {
132+
log.Fatal(err)
133+
}
134+
if exists {
135+
fmt.Println("File exists")
136+
}
137+
```
138+
139+
#### Delete File
140+
141+
```go
142+
err := initcore.DeleteFile("test.txt")
143+
if err != nil {
144+
log.Fatal(err)
145+
}
146+
```
147+
148+
## Running Tests
149+
150+
### Local File System Tests
151+
152+
```bash
153+
cd /path/to/woodpecker
154+
PKG_CONFIG_PATH=external/output/lib/pkgconfig \
155+
CGO_LDFLAGS="-Wl,-rpath,$(pwd)/external/output/lib" \
156+
go test -v ./common/initcore -run TestStorageV2LocalFS
157+
```
158+
159+
### Remote File System Tests (MinIO)
160+
161+
First, start the MinIO server:
162+
163+
```bash
164+
# Start MinIO with Docker
165+
docker run -d \
166+
-p 9000:9000 \
167+
-p 9001:9001 \
168+
--name minio \
169+
-e "MINIO_ROOT_USER=minioadmin" \
170+
-e "MINIO_ROOT_PASSWORD=minioadmin" \
171+
quay.io/minio/minio server /data --console-address ":9001"
172+
173+
# Wait for MinIO to start
174+
sleep 5
175+
176+
# Create a test bucket
177+
docker exec minio mc alias set local http://localhost:9000 minioadmin minioadmin
178+
docker exec minio mc mb local/woodpecker-test
179+
```
180+
181+
Then run the test:
182+
183+
```bash
184+
PKG_CONFIG_PATH=external/output/lib/pkgconfig \
185+
CGO_LDFLAGS="-Wl,-rpath,$(pwd)/external/output/lib" \
186+
go test -v ./common/initcore -run TestStorageV2RemoteFS
187+
```
188+
189+
If MinIO is not available, you can skip the remote test:
190+
191+
```bash
192+
SKIP_MINIO_TESTS=1 go test -v ./common/initcore -run TestStorageV2RemoteFS
193+
```
194+
195+
## C Interface Overview
196+
197+
The underlying C interfaces are defined in `external/src/segcore/arrow_fs_c.h`:
198+
199+
- `InitLocalArrowFileSystemSingleton(const char* c_path)` - Initialize local file system
200+
- `InitRemoteArrowFileSystemSingleton(CStorageConfig c_storage_config)` - Initialize remote file system
201+
- `CleanArrowFileSystemSingleton()` - Clean up the file system
202+
- `GetFileInfo(path, out_size, out_keys, out_values, out_count)` - Get file information
203+
- `ReadFileData(path, out_data, out_size)` - Read file data
204+
- `WriteFileData(path, data, data_size, metadata_keys, metadata_values, metadata_count)` - Write file
205+
- `DeleteFile(path)` - Delete file
206+
- `FreeMemory(ptr)` - Free memory
207+
208+
## Notes
209+
210+
1. **Memory Management:** Memory allocated by the C interface will be automatically released. There's no need to manually call `FreeMemory` in the Go code.
211+
2. **Thread Safety:** The file system singleton is thread-safe in multi-threaded environments.
212+
3. **Metadata Support:** The metadata writing feature is reserved and may not take effect in the current version.
213+
4. **Error Handling:** All functions return errors. Make sure to handle them appropriately.
214+
5. **Path Handling:** Paths are relative to the root path specified during file system initialization.
215+
216+
## Troubleshooting
217+
218+
### Build Error: milvus_core.pc not found
219+
220+
Make sure you have run `make build-cpp` and configured the correct `PKG_CONFIG_PATH`.
221+
222+
### Runtime Error: Library not loaded
223+
224+
Make sure you have correctly set `CGO_LDFLAGS` for rpath, or set `DYLD_LIBRARY_PATH` (macOS) / `LD_LIBRARY_PATH` (Linux) as appropriate.
225+
226+
### Test Failure: MinIO connection refused
227+
228+
Confirm that your MinIO server is up and running, and that the address and port in your configuration are correct.

0 commit comments

Comments
 (0)