This is where I am hung up. Why do we need to read everything into a single property source? The following sentence says because we need to read configmaps and secrets but what about configmap and secrets is different than reading a file?

This is where I am hung up. Why do we need to read everything into a single property source?

because this is they way our code was designed from the beginning. We have this for example:

ConfigMapPropertySourceLocator implements PropertySourceLocator {
    
     public PropertySource<?> locate(Environment environment) { ... }

}

and this method returns a CompositePropertySource composite = new CompositePropertySource("composite-configmap");.

So when we read a configmap, we assume we will compute a single property source, for each source. I mean:

Set<NormalizedSource> sources = new LinkedHashSet<>(this.properties.determineSources(environment));
				LOG.debug("Config Map normalized sources : " + sources);
				sources.forEach(s -> {
					MapPropertySource propertySource = getMapPropertySource(s, env);
					if ("true".equals(propertySource.getProperty(Constants.ERROR_PROPERTY))) {
						LOG.warn("Failed to load source: " + s);
					}
					else {
						LOG.debug("Adding config map property source " + propertySource.getName());
						composite.addFirstPropertySource(propertySource);
					}
				});

so :

• find the sources
• for each of them, create a single PropertySource.

But in case of a yaml multi document, things are not trivial to do. spring-boot does not impose this limitation, but we had it from the very beginning.

This is why I said, I totally agree with the comment from spring-boot maintainer, when he says : "create a single PropertySource just the say we do for each individual yaml document". I agree, but our code is far too involved at this point in time. We would need to revise the entire code and the way we create property sources.

So basically for multi-document yaml files (configmaps/secrets) we have always created a single property source per file instead of separating each document into their own property source?

And when we use Boots YamlPropertiesFactoryBean it creates multiple properties sources and this somehow results in the weird overriding of properties?

So basically for multi-document yaml files (configmaps/secrets) we have always created a single property source per file instead of separating each document into their own property source?

exactly.

And when we use Boots YamlPropertiesFactoryBean it creates multiple properties sources and this somehow results in the weird overriding of properties?

The idea is correct, the details are not. We were already using YamlPropertiesFactoryBean and this one is overriding properties in a "bad way".

We can't use the Boots way, because it creates one PropertySource per yaml document.

Essentially, I think you are spot on with the idea here, yes.

The idea is correct, the details are not. We were already using YamlPropertiesFactoryBean and this one is overriding properties in a "bad way".

Is it because the multiple property sources from the multiple documents results in incorrect ordering of the property sources? Do you have an example yaml document, the property sources Boot would create and the order they would be in as well as what we would expect? This should help me understand the crux of the issue.

yes, an example would make things simpler I guess. I will use what the OP presented to us. I will strip it down to the bare minimum. Suppose you have this yaml:

my-application:
  items:
    - "Item 1"
    - "Item 2"
    - "Item 3"
    - "Item 4"
    - "Item 5"
---
spring:
  config:
    activate:
      on-profile: dev

my-application:
  items:
    - "Item 1"
    - "Item 6"
    - "Item 7"

spring-boot, using YamlPropertySourceLoader, will create two property sources:

• name of the first property source : Config resource 'class path resource [application.yaml]' via location 'optional:classpath:/' (document #0)

• values of the fist property source : my-application.items[0] = Item 1, my-application.items[1] = Item 2 ....

• name of the second property source: Config resource 'class path resource [application.yaml]' via location 'optional:classpath:/' (document #1)

• value of the second property source: my-application.items[0] = Item 1, my-application.items[1] = Item 6, my-application.items[2] = Item 7 ....

Internally, later, when you need to wire things, IndexedElementsBinder will correctly chose the profile based properties (second yaml document from the example).

In our case, we were using YamlPropertiesFactoryBean. So when that one was reading the yaml file, it would do :

    protected Properties createProperties() {
        Properties result = CollectionFactory.createStringAdaptingProperties();
        this.process((properties, map) -> result.putAll(properties));
        return result;
    }

In simpler words, it will "merge" properties from yaml documents, into a single Map, using : result.putAll(properties).

So, it will read the first yaml document, create the properties:

my-application.items[0] = Item 1
my-application.items[1] = Item 2
my-application.items[2] = Item 3
my-application.items[3] = Item 4
my-application.items[4] = Item 5

Then, it will read the second yaml document, create the proeprties:

my-application.items[0] = Item 1
my-application.items[1] = Item 6
my-application.items[2] = Item 7

then, again using result.putAll(properties), merge these. Resulting obviously, in :

my-application.items[0] = Item 1
my-application.items[1] = Item 6
my-application.items[2] = Item 7
my-application.items[3] = Item 4
my-application.items[4] = Item 5

which is not correct.

As we talked above, if we had the option to create separate PropertySources from these two documents, we would and spring-boot injection mechanism would have taken care of the rest. But we are in a situation were this is not an option. And this is why I created the workaround.

Why is this incorrect? The dev profile is active so it is overriding the items in the array from the default profile.

it was supposed to override the entire array, exactly like spring-boot does, not individual items. no?

Ah yes OK, thats where I was all caught up. You are right it should override the property my-application.items. Thanks for being so patient!

if you wire properties using spring-boot only, no kubernetes involved, you will end up with :

List<String> items; // [Item 1, Item 6, Item 7]

but if users use configmap to read proeprties and we have the same yaml file, they will end up with :

List<String> items; // [Item 1, Item 6, Item 7, Item 4, Item 5]

yamlParserGenerator now returns a Map<String, Object> instead of Properties, so we can get rid of one step