Automatically repair lost data partitions by zerolbsony · Pull Request #17279 · apache/iotdb

zerolbsony · 2026-03-09T12:48:49Z

After the ConfigNode restarts, it will automatically start checking whether the data partition tables are complete.

… the DataPartitionTableIntegrityCheckProcedure.

…partition

Adjust method that record the earliest timeslot id for every database

Remove two unit testes only run successful in local environment

…tion Information instead of scanning data directories in the DataNode; Correct the logic that retry after the step failed; Correct skipDataNodes and failedDataNodes serialization and deserialization.

… repair_lost_data_partition

Copilot

Pull request overview

This PR introduces an automated “data partition table integrity check + repair” flow intended to run after ConfigNode restart by adding new DataNode RPCs for (1) earliest timeslot discovery and (2) DataPartitionTable regeneration from local TsFiles, plus a ConfigNode-side procedure to orchestrate detection/repair.

Changes:

Extend the DataNode Thrift RPC API with partition-table integrity check requests/responses.
Add a DataNode-side DataPartitionTableGenerator and supporting utilities (rate limiter, time-partition helpers) to rebuild DataPartitionTable by scanning TsFile resources.
Add a ConfigNode procedure and startup/registration hooks to trigger integrity check and write repaired partition tables back via consensus.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 26 comments.

Show a summary per file

File	Description
iotdb-protocol/thrift-datanode/src/main/thrift/datanode.thrift	Adds new DataNode RPC structs + methods for integrity check and partition-table generation.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/rateLimiter/LeakyBucketRateLimiter.java	Introduces a leaky-bucket throughput limiter used during scanning.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/TimePartitionUtils.java	Adds helpers to convert partitionId ↔ time and a new `satisfyPartitionId` overload.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/partition/SeriesPartitionTable.java	Attempts to prevent duplicate consensus group IDs per time partition.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/enums/DataPartitionTableGeneratorState.java	Adds a status-code enum for generator task state.
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/concurrent/ThreadName.java	Adds thread names for the new scan/recovery pools.
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template	Adds new recovery-related configuration parameters.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/timeindex/ITimeIndex.java	Adds a limiter-aware `getDevices` method signature.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/timeindex/FileTimeIndex.java	Implements limiter-aware device scanning from `.resource` files.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/timeindex/ArrayDeviceTimeIndex.java	Implements limiter-aware `getDevices` (no-op for in-memory index).
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/TsFileResource.java	Adds `getDevices(limiter)` delegating to the time index.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java	Implements new RPC endpoints and local filesystem scanning + DataPartitionTable serialization.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/OperationType.java	Adds operation types for new RPC error handling.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/partition/DataPartitionTableGenerator.java	Adds generator that rebuilds a DataPartitionTable from local TsFiles.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java	Loads new recovery properties into DataNode config.
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java	Adds new config fields + accessors for recovery tuning.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java	Adds restart-time background task intended to trigger integrity check.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/store/ProcedureType.java	Registers a new procedure type for integrity check.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/store/ProcedureFactory.java	Adds procedure deserialization/dispatch for the new procedure.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/state/DataPartitionTableIntegrityCheckProcedureState.java	Adds state-machine states for the integrity check procedure.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java	Implements the integrity check and repair orchestration procedure.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/ConfigNodeProcedureEnv.java	Adds a new env class (currently appears redundant/unreferenced).
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/node/NodeManager.java	Adds registration-time trigger that can launch the integrity check.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java	Adds a `dataPartitionTableIntegrityCheck()` convenience API.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/conf/ConfigNodeDescriptor.java	Loads the new wait-timeout property into ConfigNode config.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/conf/ConfigNodeConfig.java	Adds ConfigNode config field for “wait all DataNodes up” timeout.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/client/sync/SyncDataNodeClientPool.java	Wires new sync actions to the new DataNode RPC calls.
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/client/sync/CnToDnSyncRequestType.java	Adds new sync request types for integrity check and generation.
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/service/DataNodeInternalRPCServiceImplTest.java	Adds logging members (no new test coverage for the new RPCs).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

...e/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java

+                                        if (timeslot
+                                            < databaseEarliestRegionMap.get(databaseName)) {
+                                          databaseEarliestRegionMap.put(databaseName, timeslot);
+                                        }


...che/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java

+        assignedDataPartition.put(
+            lostDataPartitionsOfDatabases.stream().findFirst().get(), finalDataPartitionTable);


zerolbsony · 2026-03-17T03:32:35Z

...b-core/datanode/src/main/java/org/apache/iotdb/db/partition/DataPartitionTableGenerator.java

+    }
+
+    status = TaskStatus.IN_PROGRESS;
+    return CompletableFuture.runAsync(this::generateDataPartitionTableByMemory);


Changed yet

zerolbsony · 2026-03-17T03:32:21Z

...c/main/java/org/apache/iotdb/confignode/procedure/impl/partition/ConfigNodeProcedureEnv.java

+ * Environment object for ConfigNode procedures. Provides access to ConfigManager and other
+ * necessary components.
+ */
+public class ConfigNodeProcedureEnv {
+
+  private final ConfigManager configManager;
+
+  public ConfigNodeProcedureEnv(ConfigManager configManager) {
+    this.configManager = configManager;
+  }
+
+  public ConfigManager getConfigManager() {
+    return configManager;
+  }


zerolbsony · 2026-03-17T03:31:34Z

iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java

+    try {
+      dataPartitionTableCheckFuture.get();
+    } catch (ExecutionException | InterruptedException e) {
+      LOGGER.error("Data partition table check task execute failed", e);


Changed yet

zerolbsony · 2026-03-17T03:06:21Z

...che/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java

+    int earliestTimeslotsSize = byteBuffer.getInt();
+    earliestTimeslots = new ConcurrentHashMap<>();
+    for (int i = 0; i < earliestTimeslotsSize; i++) {
+      String database = String.valueOf(byteBuffer.getChar());


Changed yet.

zerolbsony · 2026-03-17T03:04:32Z

...e/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java

+    try {
+      Files.list(dataDir.toPath())
+          .filter(Files::isDirectory)
+          .forEach(
+              sequenceTypePath -> {
+                try {
+                  Files.list(sequenceTypePath)


Changed yet

...e/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java

zerolbsony · 2026-03-17T06:29:12Z

iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/node/NodeManager.java

+      // Trigger integrity check asynchronously
+      try {
+        configManager.getProcedureManager().dataPartitionTableIntegrityCheck();
+        LOGGER.info("Data partition table integrity check procedure submitted successfully");


These codes have some problems, can't pass tests.

zerolbsony · 2026-03-17T06:29:32Z

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java


+  /* Need use these parameters when repair data partition table */
+  private int partitionTableRecoverWorkerNum = 10;
+  private int partitionTableRecoverMaxReadBytesPerSecond = 1000;


Changed yet

CRZbulabula

PTAL.

CRZbulabula · 2026-03-13T06:19:00Z

iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/conf/ConfigNodeConfig.java


  private long forceWalPeriodForConfigNodeSimpleInMs = 100;

+  private long partitionTableRecoverWaitAllDnUpTimeout = 60000;


Refactor this parameter

Suggested change

private long partitionTableRecoverWaitAllDnUpTimeout = 60000;

private long partitionTableRecoverWaitAllDnUpTimeoutInMs = 60000;

CRZbulabula · 2026-03-13T06:29:55Z

iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java

+            dataPartitionTableCheckExecutor.submit(
+                () -> {
+                  LOGGER.info(
+                      "Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");


Better to append a prefix for all of your logs appended, e.g.,

Suggested change

"Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");

"[DataPartitionIntegrity] Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");

CRZbulabula · 2026-03-13T06:30:05Z

iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java

+                () -> {
+                  LOGGER.info(
+                      "Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
+                  //          Thread.sleep(CONF.getPartitionTableRecoverWaitAllDnUpTimeout());


Remove this.

CRZbulabula · 2026-03-13T06:30:31Z

iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java

+                      latch.countDown();
+                    } else {
+                      LOGGER.info("No running datanodes found, waiting...");
+                      Thread.sleep(5000); // 等待5秒后重新检查


Suggested change

Thread.sleep(5000); // 等待5秒后重新检查

Thread.sleep(5000);

iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java

CRZbulabula · 2026-03-13T07:10:42Z

...che/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java

+            .forEach(
+                slot -> {
+                  if (!TimePartitionUtils.satisfyPartitionId(
+                      slot.getStartTime(), earliestTimeslot)) {
+                    lostDataPartitionsOfDatabases.add(database);
+                    LOG.warn(
+                        "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+                        database,
+                        earliestTimeslot);
+                  }


Rectify these codes. The only thing u need to check is whether the database's earlist timeSlot greater than the DataNode's.

...che/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java

CRZbulabula · 2026-03-13T07:23:46Z

iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/node/NodeManager.java

    getClusterSchemaManager().adjustMaxRegionGroupNum();

+    // Check if all DataNodes are registered and trigger integrity check if needed
+    checkAndTriggerIntegrityCheck();


Remove this function.

CRZbulabula · 2026-03-13T07:54:37Z

...e/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java

+          ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+          new ThreadPoolExecutor.CallerRunsPolicy());
+
+  private Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();


This could be set to a local variable.

The executor is no longer a global parameter of class.

Change this map into a local variable.

Change this map into a local variable.

Changed yet

...che/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java

…eted in the progress caused by rpc timeout

…is from ConfigNode, the data partition table lost.

CRZbulabula

PTAL.

CRZbulabula · 2026-03-17T06:36:46Z

...che/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java

+                .min(Comparator.comparingLong(TTimePartitionSlot::getStartTime))
+                .orElse(null);
+
+        if (!TimePartitionUtils.satisfyPartitionId(localEarliestSlot.getStartTime(), earliestTimeslot)) {


Judge whether earliestTimeSlot.startTime < localEarliestSlot.startTime

CRZbulabula · 2026-03-17T06:44:27Z

...che/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java

+    setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+    return Flow.HAS_MORE_STATE;


Suggested change

setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);

return Flow.HAS_MORE_STATE;

setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);

return Flow.HAS_MORE_STATE;

...che/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java

CRZbulabula · 2026-03-17T06:51:19Z

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java

+  /* Need use these parameters when repair data partition table */
+  private int partitionTableRecoverWorkerNum = 10;
+  // Rate limit set to 10 MB/s
+  private int partitionTableRecoverMaxReadBytesPerSecond = 10;


MaxReadMBs instead of Bytes

CRZbulabula · 2026-03-17T06:53:37Z

...e/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java

+          ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+          new ThreadPoolExecutor.CallerRunsPolicy());
+
+  private Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();


Change this map into a local variable.

CRZbulabula · 2026-03-17T06:54:19Z

...e/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java

+  private Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();
+
+  // Must be lower than the RPC request timeout, in milliseconds
+  private static final long timeoutMs = 50000;


DO NOT append member variables easily.

...e/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java

CRZbulabula · 2026-03-17T07:09:44Z

...e/datanode/src/test/java/org/apache/iotdb/db/service/DataNodeInternalRPCServiceImplTest.java


 public class DataNodeInternalRPCServiceImplTest {

+  private static final Logger LOG =


Why change this file?

I forgot to clean it. I coded some unit tests for test rpc interfaces on the local ago.

CRZbulabula · 2026-03-17T07:11:07Z

iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/TimePartitionUtils.java

    return startPartition <= partitionId && endPartition >= partitionId;
  }

+  public static boolean satisfyPartitionId(long startTime, long partitionId) {


No need to update this class in our current implementation.

The overflow problem still exist, that's no way to resolve, previously, the user set timePartitionOrigin to Long.MIN_VALUE. In this case, adding it to partitionId = -1 will indeed cause an overflow. However, the partition table in the system only accepts timestamps of the long type and does not support bigint timestamps. Therefore, if an overflow actually occurs, we have to accept the outcome where the program is interrupted by an exception being thrown.

CRZbulabula

LGTM!

...che/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java

iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template

...commons/src/main/java/org/apache/iotdb/commons/utils/rateLimiter/LeakyBucketRateLimiter.java

iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/TimePartitionUtils.java

…q directory

...che/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java

...b-core/datanode/src/main/java/org/apache/iotdb/db/partition/DataPartitionTableGenerator.java

...e/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java

iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template

...commons/src/main/java/org/apache/iotdb/commons/utils/rateLimiter/LeakyBucketRateLimiter.java

Adjust new method to compute progress of data partition table generation

zerolbsony added 15 commits March 1, 2026 23:58

Come true all rpc interfaces in the DataNode, and partial features in…

55665f7

… the DataPartitionTableIntegrityCheckProcedure.

Debugged and verified all key logic in the procedure.

1fb3ab5

Merge remote-tracking branch 'upstream/master' into repair_lost_data_…

cb2fd58

…partition

Correct dataPartitionTables and finalDataPartitionTable serialization;

cb5f601

Adjust method that record the earliest timeslot id for every database

Correct heartbeat logic when data partition table is generating;

f19d1d3

Remove two unit testes only run successful in local environment

Use StorageEngine.getInstance().getAllDataRegions() to get Data Parti…

49ef823

…tion Information instead of scanning data directories in the DataNode; Correct the logic that retry after the step failed; Correct skipDataNodes and failedDataNodes serialization and deserialization.

Adjust the default value is 1 min

3960a07

Adjust the default value is 1 min

98a18c3

Merge remote-tracking branch 'origin/repair_lost_data_partition' into…

129610a

… repair_lost_data_partition

Append a description about the unit

d0882ee

use the spotless command to format code

9a53bd2

Avoid writing duplicate values

0448cb0

Fix bug when get no data partition table in the ConfigNode.

2468cf7

Add a license description.

52e3fde

mvn spotless:apply

7c7693d

CRZbulabula requested a review from Copilot March 13, 2026 06:18

Copilot started reviewing on behalf of CRZbulabula March 13, 2026 06:19 View session

Copilot AI reviewed Mar 13, 2026

View reviewed changes

CRZbulabula requested changes Mar 13, 2026

View reviewed changes

zerolbsony added 11 commits March 13, 2026 17:32

Fix problems based on review opinions

5092137

Remove some unuseful functions

631a1ea

Find the earliest time slot in the ConfigNode

500a0ae

Resolve the problem that data partition table generation is not compl…

9355430

…eted in the progress caused by rpc timeout

Correct need to redirect the target step.

328d85c

Integrate the merge logic into the DataPartitionTable class

c6b77a4

Support multiple databases data partition tables are repaired

0956e07

Fix bugs

62fb277

Fix

b47f0fb

Compare startTime between ConfigNode and DataNode, if the larger one …

4929a91

…is from ConfigNode, the data partition table lost.

Change to 10 seconds.

6dd58ab

Correct the input parameter name

5a652a9

CRZbulabula requested changes Mar 17, 2026

View reviewed changes

zerolbsony added 5 commits March 17, 2026 15:23

Changed to the local variable.

83cb061

Fix all opinions

57e8044

spotless code

617b9ce

license

b90105b

CRZbulabula approved these changes Mar 18, 2026

View reviewed changes

jt2594838 reviewed Mar 18, 2026

View reviewed changes

zerolbsony added 5 commits March 18, 2026 23:31

Fix some opinions

2081359

Fix another lots of opinions

e923176

Optimize codes

0728ca3

Skip current loop when no time slot info is found in the seq and unse…

35546a6

…q directory

Correct the logic that get the earliest time slot

dc0b5f9

jt2594838 reviewed Mar 20, 2026

View reviewed changes

zerolbsony added 4 commits March 20, 2026 14:37

Fix some opinions

61a1b7b

Use the Google guava RateLimiter instead of LeakyBucketRateLimiter;

230068e

Adjust new method to compute progress of data partition table generation

Fix

899d134

Fix bug

14a71ab

jt2594838 approved these changes Mar 20, 2026

View reviewed changes

Fix serialize bugs

fd43f1e

		assignedDataPartition.put(
		lostDataPartitionsOfDatabases.stream().findFirst().get(), finalDataPartitionTable);


		private long forceWalPeriodForConfigNodeSimpleInMs = 100;

		private long partitionTableRecoverWaitAllDnUpTimeout = 60000;

	"Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
	"[DataPartitionIntegrity] Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");

	Thread.sleep(5000); // 等待5秒后重新检查
	Thread.sleep(5000);

		setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
		return Flow.HAS_MORE_STATE;


		public class DataNodeInternalRPCServiceImplTest {

		private static final Logger LOG =

Conversation

zerolbsony commented Mar 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CRZbulabula left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CRZbulabula left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CRZbulabula left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!