opt: redesign the truncate effect logic to reduce memory cost in text mode #718

[chilli13]

#718

[chilli13]

@yuweizzz #731 can not solve issue memory cost too much when deal big files, as ew.writeEvent(e) keeps writing all data to ew.payload in function func (ew *eventWorker) Run(), this pull try to limit payload len before ew.writeEvent(e), please help to review, thks

[yuweizzz]

I think it is better to move this part into func (ew *eventWorker) writeEvent(e event.IEventStruct).

[cfc4n]

为什么不放在这里获取tzise，而是通过外部传递呢？

建议还原所有代码，之后只保留一下更改。

tsize := int(ew.processor.truncateSize)
//terminal write when reach the truncate size
	if tsize > 0 && ew.payload.Len() >= tsize {
		ew.payload.Truncate(tsize)
		_ = ew.writeToChan(fmt.Sprintf("Events truncated, size: %d bytes\n", tsize))
		return true
	}
	return false

[cfc4n]

Do you confirm that clearing the payload, moving from parserEvents to writeEvent, can optimize memory usage?

[chilli13]

Do you confirm that clearing the payload, moving from parserEvents to writeEvent, can optimize memory usage?

Yes, the test steps are as follows

1. run ecapture without --tsize
   2.access big file (1.7G+), watch ecapture memory cost, find the peak of memory consumption
2. run ecapture with --tsize 10240000 (about 10M), repeat step 2

#./bin/ecapture tls
...
#ps -p $(pidof ecapture)  -o rss,%mem
  RSS %MEM
1100172 13.5


#./bin/ecapture tls -t 10240000
# ps -p $(pidof ecapture)  -o rss,%mem
  RSS %MEM
252116  3.1

[cfc4n]

LGTM, Thanks.