Tables with over 128 columns fail due to out of index range.

[shiftcare-ian]

Problem: greenmask dump panics when processing the settings table (129 columns). Greenmask's COPY format decoder uses a fixed [128] array internally. Any table with ≥129 columns causes an index-out-of-range panic. The crash happens because Greenmask still routes it through the COPY decoder even for untransformed tables.

panic: runtime error: index out of range [128] with length 128

goroutine 4819 [running]:
[github.com/greenmaskio/greenmask/internal/db/postgres/pgcopy.(*Row).Decode(0x28c86c1123c0](http://github.com/greenmaskio/greenmask/internal/db/postgres/pgcopy.(*Row).Decode(0x28c86c1123c0), {0x28c88317810f, 0x997, 0x998})
    /home/runner/work/greenmask/greenmask/internal/db/postgres/pgcopy/row.go:103 +0x3c8
[github.com/greenmaskio/greenmask/internal/db/postgres/dumpers.(*TransformationPipeline).Dump(0x28c86b40af60](http://github.com/greenmaskio/greenmask/internal/db/postgres/dumpers.(*TransformationPipeline).Dump(0x28c86b40af60), {0x125c0b8, 0x28c87bf48960}, {0x28c88317810f?, 0x0?, 0x0?})
    /home/runner/work/greenmask/greenmask/internal/db/postgres/dumpers/transformation_pipeline.go:160 +0x73
[github.com/greenmaskio/greenmask/internal/db/postgres/dumpers.(*TableDumper).process(0x28c87bbdd8f0](http://github.com/greenmaskio/greenmask/internal/db/postgres/dumpers.(*TableDumper).process(0x28c87bbdd8f0), {0x125c0b8, 0x28c87bf48960}, {0x12675e0?, 0x28c86beeb920?}, {0x7f76d097fa68?, 0x28c875f0b7e8?}, {0x125c520, 0x28c86b40af60})
    /home/runner/work/greenmask/greenmask/internal/db/postgres/dumpers/table.go:160 +0x466
[github.com/greenmaskio/greenmask/internal/db/postgres/dumpers.(*TableDumper).Execute.(*TableDumper).dumper.func2()](http://github.com/greenmaskio/greenmask/internal/db/postgres/dumpers.(*TableDumper).Execute.(*TableDumper).dumper.func2())
    /home/runner/work/greenmask/greenmask/internal/db/postgres/dumpers/table.go:89 +0x1a5
[golang.org/x/sync/errgroup.(*Group).Go.func1()](http://golang.org/x/sync/errgroup.(*Group).Go.func1())
    /home/runner/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50
created by [golang.org/x/sync/errgroup.(*Group).Go](http://golang.org/x/sync/errgroup.(*Group).Go) in goroutine 27
    /home/runner/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

[wwoytenko]

Thanks for the report!

So far, we haven’t been able to reproduce the issue on our side. Could you please share the config and the DB schema that consistently reproduces this behavior?

You can use the playground for verification.

I tried to reproduce it using the follwing schema and config

CREATE TABLE wide_table (
    col001 INT,
    col002 TEXT,
    col003 INT,
    col004 TEXT,
    col005 INT,
    col006 TEXT,
    col007 INT,
    col008 TEXT,
    col009 INT,
    col010 TEXT,
    col011 INT,
    col012 TEXT,
    col013 INT,
    col014 TEXT,
    col015 INT,
    col016 TEXT,
    col017 INT,
    col018 TEXT,
    col019 INT,
    col020 TEXT,
    col021 INT,
    col022 TEXT,
    col023 INT,
    col024 TEXT,
    col025 INT,
    col026 TEXT,
    col027 INT,
    col028 TEXT,
    col029 INT,
    col030 TEXT,
    col031 INT,
    col032 TEXT,
    col033 INT,
    col034 TEXT,
    col035 INT,
    col036 TEXT,
    col037 INT,
    col038 TEXT,
    col039 INT,
    col040 TEXT,
    col041 INT,
    col042 TEXT,
    col043 INT,
    col044 TEXT,
    col045 INT,
    col046 TEXT,
    col047 INT,
    col048 TEXT,
    col049 INT,
    col050 TEXT,
    col051 INT,
    col052 TEXT,
    col053 INT,
    col054 TEXT,
    col055 INT,
    col056 TEXT,
    col057 INT,
    col058 TEXT,
    col059 INT,
    col060 TEXT,
    col061 INT,
    col062 TEXT,
    col063 INT,
    col064 TEXT,
    col065 INT,
    col066 TEXT,
    col067 INT,
    col068 TEXT,
    col069 INT,
    col070 TEXT,
    col071 INT,
    col072 TEXT,
    col073 INT,
    col074 TEXT,
    col075 INT,
    col076 TEXT,
    col077 INT,
    col078 TEXT,
    col079 INT,
    col080 TEXT,
    col081 INT,
    col082 TEXT,
    col083 INT,
    col084 TEXT,
    col085 INT,
    col086 TEXT,
    col087 INT,
    col088 TEXT,
    col089 INT,
    col090 TEXT,
    col091 INT,
    col092 TEXT,
    col093 INT,
    col094 TEXT,
    col095 INT,
    col096 TEXT,
    col097 INT,
    col098 TEXT,
    col099 INT,
    col100 TEXT,
    col101 INT,
    col102 TEXT,
    col103 INT,
    col104 TEXT,
    col105 INT,
    col106 TEXT,
    col107 INT,
    col108 TEXT,
    col109 INT,
    col110 TEXT,
    col111 INT,
    col112 TEXT,
    col113 INT,
    col114 TEXT,
    col115 INT,
    col116 TEXT,
    col117 INT,
    col118 TEXT,
    col119 INT,
    col120 TEXT,
    col121 INT,
    col122 TEXT,
    col123 INT,
    col124 TEXT,
    col125 INT,
    col126 TEXT,
    col127 INT,
    col128 TEXT,
    col129 INT
);

INSERT INTO wide_table VALUES (
    1, 'text002', 3, 'text004', 5, 'text006', 7, 'text008', 9, 'text010',
    11, 'text012', 13, 'text014', 15, 'text016', 17, 'text018', 19, 'text020',
    21, 'text022', 23, 'text024', 25, 'text026', 27, 'text028', 29, 'text030',
    31, 'text032', 33, 'text034', 35, 'text036', 37, 'text038', 39, 'text040',
    41, 'text042', 43, 'text044', 45, 'text046', 47, 'text048', 49, 'text050',
    51, 'text052', 53, 'text054', 55, 'text056', 57, 'text058', 59, 'text060',
    61, 'text062', 63, 'text064', 65, 'text066', 67, 'text068', 69, 'text070',
    71, 'text072', 73, 'text074', 75, 'text076', 77, 'text078', 79, 'text080',
    81, 'text082', 83, 'text084', 85, 'text086', 87, 'text088', 89, 'text090',
    91, 'text092', 93, 'text094', 95, 'text096', 97, 'text098', 99, 'text100',
    101, 'text102', 103, 'text104', 105, 'text106', 107, 'text108', 109, 'text110',
    111, 'text112', 113, 'text114', 115, 'text116', 117, 'text118', 119, 'text120',
    121, 'text122', 123, 'text124', 125, 'text126', 127, 'text128', 129
);

common:
  pg_bin_path: "/usr/local/pgsql/bin"
  tmp_dir: "/tmp"

log:
  level: "debug"

storage:
  type: "directory"
  directory:
    path: "/tmp"

dump:
  pg_dump_options:
    dbname: "host=playground-db user=postgres password=example dbname=original"
    table: "public.wide_table"
  transformation:
    - schema: "public"
      name: "wide_table"  # Table schema
      subset_conds:
        - "public.wide_table.col001 = 1"
      transformers: # List of transformers to apply
        - name: "RandomInt" # name of transformers
          params: # Transformer parameters
            min: 1
            max: 10
            column: "col129" # Column parameter - this transformer
restore:
  pg_restore_options: # pg_restore option (you can use the same options as pg_restore has)
    jobs: 10
    dbname: "host=playground-db user=postgres password=example dbname=transformed"

And it works fine.

Additionaly I can't confirm that Greenmask routes data through the COPY decoder even for untransformed tables. I see that if remove transformation from the config the table is being dumped via PlainDumpPipeline bypassing the COPY format encoder.

[wwoytenko]

[wwoytenko]

I would appreciate it if you could help by sharing the config and DB schema that reproduce this behavior in your environment.

[shiftcare-ian]

I'll attempt to replicate it again. The config is generated by code and the issue now seems more complex then just 129 columns.

[shiftcare-ian]

Update: We dug into the source code and have a theory on the root cause.

Correction

The original report incorrectly described this as a fixed [128] array. The Row is dynamically allocated via NewRow(len(table.Columns)) in transformation_pipeline.go:86. The 128 in the panic is simply whatever len(table.Columns) returned at runtime — not a hardcoded limit.

What we tried

We created a 129-column table matching our affected table's exact type distribution (mixed boolean, varchar, integer, varchar[], text, timestamp, date, float — 83 nullable, 45 NOT NULL) with a single SetNull transformer. None of these reproduced the panic:

• 1 row, 1001 rows, varied NULL patterns, all-columns-non-NULL
• Memory-constrained Docker (256 MB and 2 GB)

Why we think it happens

The panic is at row.go:103:

p := r.columnPos[idx]

Row.Decode() splits COPY text output by tab. When isDynamic is false (which it is for TransformationPipeline), there's no bounds check — so if COPY ... TO STDOUT produces more fields than len(table.Columns), it panics.

We believe getColumnsConfig() returned fewer columns than COPY produced in our environment. Possible triggers: a DDL migration between introspection and COPY, or a version-specific difference in the introspection query. We're still investigating on our side.

Suggested fix

Regardless of root cause, Row.Decode() should handle the mismatch gracefully. The isDynamic path already does this via appendNewEmptyBuffer() — the non-dynamic path just needs the same safety with a warning:

if idx >= len(r.columnPos) {
    log.Warn().
        Int("expected", len(r.columnPos)).
        Int("actual", idx+1).
        Msg("COPY output has more fields than expected — growing dynamically")
    r.appendNewEmptyBuffer()
}

The dump succeeds (extra columns pass through untransformed), and the warning gives operators visibility into the mismatch.

[shiftcare-ian]

Clarification on the suggested fix:

The fix is at row.go:99. The existing guard only applies to dynamic mode:

if r.isDynamic && idx >= r.tupleSize {
    r.appendNewEmptyBuffer()
}

Remove the r.isDynamic condition so the bounds check applies in all cases, and add a warning:

if idx >= r.tupleSize {
    log.Warn().
        Int("expected", r.tupleSize).
        Int("actual", idx+1).
        Msg("COPY output has more fields than expected — growing dynamically")
    r.appendNewEmptyBuffer()
}

This prevents the panic at row.go:103 (p := r.columnPos[idx]) by growing the buffer before accessing it, regardless of whether the Row was created in dynamic mode or with a fixed size.

[wwoytenko]

Thank you! I will take a look soon.

If you wish you can submit a PR

[wwoytenko]

Most important is to reproduce it somehow with unit test or with config and schema using playground.

Probbaly there might be a dandgerous case. If we have statically allocated NewRow(129) and a table has 129 columns, it's hardly should force columnPos buffer growing. If so it means that Decode(raw []byte) error method might have a bug in decoding logic. For now it decodes data using copy delimiter '\t' and looks like in one of the line this logic divided a row more than 129 times. That's why, I don't think it's related to r.appendNewEmptyBuffer() function call. The row initialization during dump procedure is expected to have a strict size pgcopy.NewRow(len(table.Columns))

[wwoytenko]

I would suggest you to reproduce in the following way.

1. Apply the following patch - it throw a panic with kind of info and real data

diff --git a/internal/db/postgres/pgcopy/row.go b/internal/db/postgres/pgcopy/row.go
index 6b35da3d..e8a7ff1f 100644
--- a/internal/db/postgres/pgcopy/row.go
+++ b/internal/db/postgres/pgcopy/row.go
@@ -16,6 +16,7 @@ package pgcopy
 
 import (
        "errors"
+       "fmt"
        "slices"
 
        "github.com/greenmaskio/greenmask/pkg/toolkit"
@@ -98,6 +99,8 @@ func (r *Row) Decode(raw []byte) error {
                }
                if r.isDynamic && idx >= r.tupleSize {
                        r.appendNewEmptyBuffer()
+               } else if !r.isDynamic && idx >= r.tupleSize {
+                       panic(fmt.Errorf("error encoding row %d: out of range, data '%s'", idx, string(raw)))
                }
 
                p := r.columnPos[idx]

git apply changes.patch

2. Build patched greenmask binary
3. Run greenmask with your config and try to catch this behavior
4. Have a look at the panic and raw record data that was failed
5. If possible try to generate a similar raw record data (tabular) or use the original one if allowed
6. Write a unit test that creates NewRow(129) and call Decode(YOUR_DATA_HERE) and see how it goes. It should fail.
7. If possible try to fix the found issue or explain me how can I reproduce this behaviour, I will do my best

I will be in touch.