Skip to content

Commit 060d39f

Browse files
jasnelljasnell
authored
Merge pull request #6 from tc39/jasnell/spec-draft-text
2 parents 3bdd000 + cabaae2 commit 060d39f

3 files changed

Lines changed: 3031 additions & 49 deletions

File tree

‎README.md‎

Lines changed: 168 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
1-
# TypedArray Concatenation
1+
# TypedArray, ArrayBuffer, and SharedArrayBuffer Concatenation
22

3-
ECMAScript Proposal for TypedArray concatentation
3+
ECMAScript Proposal for TypedArray, ArrayBuffer, and SharedArrayBuffer concatenation
44

55
This proposal is currently [stage 1](https://github.com/tc39/proposals/blob/master/README.md) of the [process](https://tc39.github.io/process-document/).
66

77
## Problem
88

9-
ECMAScript should provide a native method for concatenating TypedArrays that enables implementations to optimize through strategies that can avoid the current requirement of eagerly allocating and copying data into new buffers
9+
ECMAScript should provide native methods for concatenating TypedArrays and ArrayBuffers that enable implementations to optimize through strategies that can avoid the current requirement of eagerly allocating and copying data into new buffers.
1010

11-
It is common for applications on the web (both browser and server side) to need to concatenate two or more TypedArray instances as part of a data pipeline. Unfortunately, the mechanisms available for concatenation are difficult to optimize for performance. All require additional allocations and copying at inopportune times in the application.
11+
It is common for applications on the web (both browser and server side) to need to concatenate two or more TypedArray or ArrayBuffer instances as part of a data pipeline. Unfortunately, the mechanisms available for concatenation are difficult to optimize for performance. All require additional allocations and copying at inopportune times in the application.
1212

1313
A common example is a `WritableStream` instance that collects writes up to a defined threshold before passing those on in a single coalesced chunk. Server-side applications have typically relied on Node.js' `Buffer.concat` API, while browser applications have relied on either browser-compatible polyfills of `Buffer` or `TypedArray.prototype.set`.
1414

@@ -18,9 +18,8 @@ let size = 0;
1818
new WritableStream({
1919
write(chunk) {
2020
buffers.push(chunk);
21-
size += chunks.length;
22-
if (buffer.byteLength >= 4096) {
23-
// Not yet the actual proposed syntax... we have to determine that still
21+
size += chunk.length;
22+
if (size >= 4096) {
2423
flushBuffer(concat(buffers, size));
2524
buffers = [];
2625
size = 0;
@@ -35,49 +34,197 @@ function concat(buffers, size) {
3534
dest.set(buffer, offset);
3635
offset += buffer.length;
3736
}
37+
return dest;
3838
}
3939
```
4040

41-
```js
42-
const buffer1 = Buffer.from('hello');
43-
const buffer2 = Buffer.from('world');
44-
const buffer3 = Buffer.concat([buffer1, buffer2]);
45-
```
46-
47-
While these approaches work, they end up being difficult to optimize because they require potential expensive allocations and data copying at inopportune times while processing the information. The `TypedArray.prototype.set` method does provide an approach for concatenation that is workable, but the way the algorithm is defined, there is no allowance given for implementation-defined optimization.
41+
While these approaches work, they end up being difficult to optimize because they require potentially expensive allocations and data copying at inopportune times while processing the information. The `TypedArray.prototype.set` method does provide an approach for concatenation that is workable, but the way the algorithm is defined, there is no allowance given for implementation-defined optimization.
4842

4943
## Proposal
5044

51-
This proposal seeks to improve the current state by providing a mechanism that provides an optimizable concatenation path for TypedArrays within the language.
45+
This proposal provides three complementary static methods for concatenation:
46+
47+
1. **`%TypedArray%.concat(items [, length])`** — element-oriented concatenation of same-type TypedArrays
48+
2. **`ArrayBuffer.concat(items [, options])`** — byte-oriented concatenation returning an ArrayBuffer
49+
3. **`SharedArrayBuffer.concat(items [, options])`** — byte-oriented concatenation returning a SharedArrayBuffer
50+
51+
All three methods afford implementations the ability to determine the most optimal approach, and optimal timing, for performing the allocations and copies, but no specific optimization is required.
52+
53+
`%TypedArray%.concat` accepts only TypedArrays of the same type as the constructor (e.g., all `Uint8Array` for `Uint8Array.concat`), though those TypedArrays may be backed by either an ArrayBuffer or a SharedArrayBuffer. `ArrayBuffer.concat` and `SharedArrayBuffer.concat` accept any mix of ArrayBuffer, SharedArrayBuffer, TypedArray, and DataView inputs — the return type is determined by which method is called, not by the input types.
5254

53-
As a stage 1 proposal, the exact mechanism has yet to be defined but the goal would be to achieve a model very similar to Node.js' `Buffer.concat`, where multiple input `TypedArray`s can be given and the implementation can determine the most optimum approach to concatenating those into a single returned `TypedArray` of the same type.
55+
### `%TypedArray%.concat(items [, length])`
56+
57+
Concatenates multiple TypedArrays of the same type into a new TypedArray.
5458

5559
```js
5660
const enc = new TextEncoder();
5761
const u8_1 = enc.encode('Hello ');
5862
const u8_2 = enc.encode('World!');
5963
const u8_3 = Uint8Array.concat([u8_1, u8_2]);
64+
// u8_3 contains: Uint8Array [72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33]
6065
```
6166

62-
A key goal, if a reasonable approach to do so is found, would be to afford implementations the ability to determine the most optimal approach, and optimal timing, for performing the allocations and copies, but no specific optimization would be required.
67+
- `items` — an iterable of TypedArray instances, all of the same type as the constructor.
68+
- `length` (optional) — a non-negative integer specifying the element length of the result. If less than the total, the result is truncated. If greater, the result is zero-filled. Defaults to the sum of all input lengths.
6369

64-
### Differences from `set`
70+
All items must be TypedArrays of the same type as the constructor (e.g., all `Uint8Array` for `Uint8Array.concat`). A `TypeError` is thrown if any item is a different type. Items may be backed by either an ArrayBuffer or a SharedArrayBuffer.
6571

66-
Per the current definition of `TypedArray.prototype.set` in the language specification, the user code is responsible for allocating the destination `TypedArray` in advance along with calculating and updating the offset at which each copied segment should go. Allocations can be expensive and the book keeping can be cumbersome, particularly when the are multiple input `TypedArrays`. The `set` algorithm is also written such that each element of the copied `TypedArray` is copied to the destination one element at a time, with no affordance given to allow the implementation to determine an alternative, more optimal copy strategy.
72+
A `TypeError` is thrown if any item is a detached TypedArray. A `RangeError` is thrown if the total element count exceeds 2<sup>53</sup> - 1.
6773

6874
```js
75+
// Truncate to 5 elements
76+
const truncated = Uint8Array.concat([u8_1, u8_2], 5);
77+
78+
// Zero-fill to 20 elements
79+
const padded = Uint8Array.concat([u8_1, u8_2], 20);
80+
81+
// WritableStream coalescing example
6982
let buffers = [];
7083
let size = 0;
7184
new WritableStream({
7285
write(chunk) {
7386
buffers.push(chunk);
74-
size += chunks.length;
87+
size += chunk.length;
7588
if (size >= 4096) {
76-
// Not yet the actual proposed syntax... we have to determine that still
7789
flushBuffer(Uint8Array.concat(buffers, size));
7890
buffers = [];
7991
size = 0;
8092
}
8193
}
8294
});
8395
```
96+
97+
The `concat` method is available on all TypedArray constructors:
98+
99+
```js
100+
// Integer types
101+
Int8Array.concat([new Int8Array([-1, 127]), new Int8Array([0, -128])]);
102+
// → Int8Array [-1, 127, 0, -128]
103+
104+
Uint8Array.concat([new Uint8Array([0, 255]), new Uint8Array([128])]);
105+
// → Uint8Array [0, 255, 128]
106+
107+
Uint8ClampedArray.concat([new Uint8ClampedArray([0, 255]), new Uint8ClampedArray([128])]);
108+
// → Uint8ClampedArray [0, 255, 128]
109+
110+
Int16Array.concat([new Int16Array([-1, 32767]), new Int16Array([0])]);
111+
// → Int16Array [-1, 32767, 0]
112+
113+
Uint16Array.concat([new Uint16Array([0, 65535]), new Uint16Array([256])]);
114+
// → Uint16Array [0, 65535, 256]
115+
116+
Int32Array.concat([new Int32Array([-1, 2147483647]), new Int32Array([0])]);
117+
// → Int32Array [-1, 2147483647, 0]
118+
119+
Uint32Array.concat([new Uint32Array([0, 4294967295]), new Uint32Array([256])]);
120+
// → Uint32Array [0, 4294967295, 256]
121+
122+
// BigInt types
123+
BigInt64Array.concat([new BigInt64Array([0n, -1n]), new BigInt64Array([9007199254740991n])]);
124+
// → BigInt64Array [0n, -1n, 9007199254740991n]
125+
126+
BigUint64Array.concat([new BigUint64Array([0n, 1n]), new BigUint64Array([18446744073709551615n])]);
127+
// → BigUint64Array [0n, 1n, 18446744073709551615n]
128+
129+
// Floating-point types
130+
Float16Array.concat([new Float16Array([1.5, -0]), new Float16Array([Infinity, NaN])]);
131+
// → Float16Array [1.5, -0, Infinity, NaN]
132+
133+
Float32Array.concat([new Float32Array([1.5, -0]), new Float32Array([Infinity, NaN])]);
134+
// → Float32Array [1.5, -0, Infinity, NaN]
135+
136+
Float64Array.concat([new Float64Array([1.5, -0]), new Float64Array([Infinity, NaN])]);
137+
// → Float64Array [1.5, -0, Infinity, NaN]
138+
```
139+
140+
### `ArrayBuffer.concat(items [, options])`
141+
142+
Concatenates the byte contents of multiple ArrayBuffers, SharedArrayBuffers, TypedArrays, or DataViews into a new ArrayBuffer.
143+
144+
```js
145+
const ab1 = new ArrayBuffer(4);
146+
const ab2 = new ArrayBuffer(4);
147+
const ab3 = ArrayBuffer.concat([ab1, ab2]);
148+
// ab3.byteLength === 8
149+
```
150+
151+
- `items` — an iterable of ArrayBuffer, SharedArrayBuffer, TypedArray, or DataView instances. For TypedArray and DataView inputs, only the viewed portion of the underlying buffer is included.
152+
- `options` (optional) — an object with the following properties:
153+
- `length` — a non-negative integer specifying the byte length of the result. If less than the total input bytes, the result is truncated. If greater, the result is zero-filled. Defaults to the sum of all input byte lengths.
154+
- `resizable` — a boolean. If `true`, the result is a resizable ArrayBuffer where `length` specifies the maximum byte length (`maxByteLength`). The actual `byteLength` is the lesser of the total input bytes and `length`. Defaults to `false`.
155+
- `immutable` — a boolean. If `true`, the result is an immutable ArrayBuffer whose contents cannot be changed, resized, or detached. Defaults to `false`. *This option depends on the [Immutable ArrayBuffer proposal](https://github.com/tc39/proposal-immutable-arraybuffer).*
156+
157+
The `resizable` and `immutable` options are mutually exclusive. A `TypeError` is thrown if both are `true`.
158+
159+
A `TypeError` is thrown for detached buffers or out-of-bounds DataViews. A `RangeError` is thrown if the total byte count exceeds 2<sup>53</sup> - 1.
160+
161+
```js
162+
// Mix of ArrayBuffer, TypedArray, and DataView inputs
163+
const ab = new ArrayBuffer(4);
164+
const u8 = new Uint8Array([1, 2, 3, 4]);
165+
const dv = new DataView(new ArrayBuffer(2));
166+
const result = ArrayBuffer.concat([ab, u8, dv]);
167+
// result.byteLength === 10
168+
169+
// Truncate to 6 bytes
170+
const truncated = ArrayBuffer.concat([ab, u8, dv], { length: 6 });
171+
172+
// Zero-fill to 16 bytes
173+
const padded = ArrayBuffer.concat([ab, u8], { length: 16 });
174+
175+
// Create a resizable result with room to grow
176+
const resizable = ArrayBuffer.concat([ab, u8], { resizable: true, length: 32 });
177+
// resizable.byteLength === 8 (actual data)
178+
// resizable.maxByteLength === 32 (can grow up to 32)
179+
180+
// Create an immutable result (requires Immutable ArrayBuffer proposal)
181+
const immutable = ArrayBuffer.concat([ab, u8], { immutable: true });
182+
// immutable.byteLength === 8
183+
// immutable.immutable === true
184+
```
185+
186+
### `SharedArrayBuffer.concat(items [, options])`
187+
188+
Concatenates the byte contents of multiple ArrayBuffers, SharedArrayBuffers, TypedArrays, or DataViews into a new SharedArrayBuffer.
189+
190+
```js
191+
const sab1 = new SharedArrayBuffer(4);
192+
const sab2 = new SharedArrayBuffer(4);
193+
const sab3 = SharedArrayBuffer.concat([sab1, sab2]);
194+
// sab3.byteLength === 8
195+
```
196+
197+
- `items` — an iterable of ArrayBuffer, SharedArrayBuffer, TypedArray, or DataView instances. For TypedArray and DataView inputs, only the viewed portion of the underlying buffer is included.
198+
- `options` (optional) — an object with the following properties:
199+
- `length` — a non-negative integer specifying the byte length of the result. If less than the total input bytes, the result is truncated. If greater, the result is zero-filled. Defaults to the sum of all input byte lengths.
200+
- `growable` — a boolean. If `true`, the result is a growable SharedArrayBuffer where `length` specifies the maximum byte length (`maxByteLength`). The actual `byteLength` is the lesser of the total input bytes and `length`. Defaults to `false`.
201+
202+
Note: The `immutable` option is not available for SharedArrayBuffers.
203+
204+
A `TypeError` is thrown for detached buffers or out-of-bounds DataViews. A `RangeError` is thrown if the total byte count exceeds 2<sup>53</sup> - 1.
205+
206+
```js
207+
// Mix of SharedArrayBuffer, TypedArray, and DataView inputs
208+
const sab = new SharedArrayBuffer(4);
209+
const u8 = new Uint8Array([1, 2, 3, 4]);
210+
const dv = new DataView(new ArrayBuffer(2));
211+
const result = SharedArrayBuffer.concat([sab, u8, dv]);
212+
// result.byteLength === 10
213+
214+
// Create a growable result with room to grow
215+
const growable = SharedArrayBuffer.concat([sab, u8], { growable: true, length: 32 });
216+
// growable.byteLength === 8 (actual data)
217+
// growable.maxByteLength === 32 (can grow up to 32)
218+
```
219+
220+
### Differences from `set`
221+
222+
Per the current definition of `TypedArray.prototype.set` in the language specification, the user code is responsible for allocating the destination `TypedArray` in advance along with calculating and updating the offset at which each copied segment should go. Allocations can be expensive and the book keeping can be cumbersome, particularly when there are multiple input `TypedArrays`. The `set` algorithm is also written such that each element of the copied `TypedArray` is copied to the destination one element at a time, with no affordance given to allow the implementation to determine an alternative, more optimal copy strategy.
223+
224+
### Why three methods?
225+
226+
`%TypedArray%.concat` operates at the TypedArray level — it is element-oriented, requires same-type inputs, and returns a TypedArray. This is the right level of abstraction when working with typed data (e.g., concatenating `Uint8Array` chunks in a stream).
227+
228+
`ArrayBuffer.concat` and `SharedArrayBuffer.concat` operate at the buffer level — they are byte-oriented, accept heterogeneous inputs (ArrayBuffer/SharedArrayBuffer, TypedArray, DataView), and return the appropriate buffer type. This is the right level of abstraction for controlling buffer properties like resizability/growability and immutability, which are concerns of the buffer, not the TypedArray.
229+
230+
`ArrayBuffer.concat` and `SharedArrayBuffer.concat` are separate methods because the return type differs and the available options differ (`immutable` is only available for ArrayBuffer, `growable` is only available for SharedArrayBuffer). This mirrors the existing separation between the `ArrayBuffer` and `SharedArrayBuffer` constructors in the language.

0 commit comments

Comments
 (0)