Preserve formatting of YAML files

[tuukkamustonen]

Encrypting and decrypting:

array: [element1, element2]

Turns into:

array:
    - elemenet1
    - element2

Also, trailing comments seem to be moved above the line:

field: value  # trailing comment

# trailing comment
field: value

And empty linefeeds are lost:

field: value

field2: value

field: value
field2: value

I would assume there are other examples, too.

Would it be possible to preserve more of the original file's formatting?

[felixfontein]

I don't think you need to create issues for every kind of formatting change that happens. Because there will be a lot.

sops works by using a YAML parser (yaml.v3), transforming the result into its own data structure, working on that one, and then reversing the process. Since sops' internal data structure handles different file formats (JSON, INI, Dotfiles, ...), it does not store much additional information, and without that information it is impossible to restore the original formatting.

So do to what you want:

1. yaml.v3 needs to support that. I've never tested how much survives a round-trip.
2. sops' internal data structures need to be extended to support more YAML-specific details.

Both are potentially a lot of work, and especially 2. will increase the maintenance burden for sops a lot. I'm not sure this is in the scope of sops.

[tuukkamustonen]

Yeah, I don't disagree. So, merged contents of the previous ticket into this one.

If you feel it's not something that SOPS will improve, just close it. I think it's good to have a ticket about the topic, should someone wonder the same, but if you feel there's nothing (sensible) to do here, it can be closed.

Perhaps a note/mention in the docs about it, though? "SOPS will re-format the YAML and lose some of the original formatting, so take that into account." or smth?

[felder]

I just filed #865 and I can't help but wonder if the issues are related.

[felixfontein]

@felder they are unrelated, #865 is clearly a bug.

[felder]

@felixfontein ok thanks!

[dry4ng]

As I'm looking to switch to sops from git-crypt and ansible-vault, I'm also running into this issue. The hardcoded 4 spaces for yaml files is especially annoying. I have to implement additional pre/post commit hooks to reformat configs to our org standard.

[neoakris]

So I remember testing this stuff with sops 3.6.0 and things like

x:
- c: 1
- a: 2
- b: 3

and the items on the list would rearrange themselves in alphabetical order like

x: 
- a: 2
- b: 3
- c: 1

I also remember that sometimes multi line yaml would get condensed to single line with lots of \n's

Kubernetes yaml that had 2 space indentation when saved and reopened could be transformed to 4 space indentation.

Stuff where it'd still be valid yaml, and a computer would evaluate the same way, but it'd be annoying for humans to read, because the formatting wasn't preserved.

I just wanted to point out that I tested a few things on 3.7.2 that I remember used to bork formatting in the past, and now the formatting seems to be preserved. Maybe we got lucky and one of the new versions fixed this :) ?

[arut-plus]

Is there any progress on this?

[felixfontein]

No.

[nourspace]

We at least have the ability to configure indentation now

# .sops.yaml

stores:
  yaml:
    indent: 2

[arut-plus]

Yeah, I know about that.
But I had a different issue - when formatting, sops was splitting long one-line strings containing "\n" (e.g. keys) that were in double quotes into multiple lines (which was problematic for me). I solved it by wrapping them in single quotes instead...
Maybe someone will come across this too, I'm just sharing.

whatever 🥀