[HDDS-1201] Reporting corrupted containers info to SCM

[hgadre]

The previous patch for this jira did not implement the logic completely. Also there was no integration test to verify end-to-end functionality. Both these issues have been fixed as part of this change.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	1328	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	74	Maven dependency ordering for branch
+1	mvninstall	478	trunk passed
+1	compile	247	trunk passed
+1	checkstyle	68	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	948	branch has no errors when building and testing our client artifacts.
+1	javadoc	169	trunk passed
0	spotbugs	326	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	532	trunk passed
		_ Patch Compile Tests _
0	mvndep	31	Maven dependency ordering for patch
+1	mvninstall	437	the patch passed
+1	compile	301	the patch passed
+1	javac	301	the patch passed
-0	checkstyle	39	hadoop-hdds: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0)
-0	checkstyle	42	hadoop-ozone: The patch generated 101 new + 0 unchanged - 0 fixed = 101 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	748	patch has no errors when building and testing our client artifacts.
+1	javadoc	170	the patch passed
-1	findbugs	225	hadoop-hdds generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1	findbugs	265	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	170	hadoop-hdds in the patch failed.
-1	unit	273	hadoop-ozone in the patch failed.
+1	asflicense	39	The patch does not generate ASF License warnings.
		6750

Reason	Tests
FindBugs	module:hadoop-hdds
	Switch statement found in org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.check() where default case is missing At KeyValueContainer.java:where default case is missing At KeyValueContainer.java:[lines 655-669]
Failed junit tests	hadoop.ozone.container.keyvalue.TestKeyValueContainerCheck

Subsystem	Report/Notes
Docker	Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/1/artifact/out/Dockerfile
GITHUB PR	#1032
JIRA Issue	HDDS-1201
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux a8e6cfd7c2b2 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / f02b0e1
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/1/artifact/out/diff-checkstyle-hadoop-hdds.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/1/artifact/out/diff-checkstyle-hadoop-ozone.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/1/artifact/out/new-findbugs-hadoop-hdds.html
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/1/artifact/out/patch-findbugs-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/1/testReport/
Max. process+thread count	1287 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	85	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	311	Maven dependency ordering for branch
+1	mvninstall	638	trunk passed
+1	compile	277	trunk passed
+1	checkstyle	76	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	1001	branch has no errors when building and testing our client artifacts.
+1	javadoc	229	trunk passed
0	spotbugs	417	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	695	trunk passed
		_ Patch Compile Tests _
0	mvndep	46	Maven dependency ordering for patch
+1	mvninstall	593	the patch passed
+1	compile	338	the patch passed
+1	javac	338	the patch passed
-0	checkstyle	50	hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-0	checkstyle	51	hadoop-ozone: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	867	patch has no errors when building and testing our client artifacts.
+1	javadoc	204	the patch passed
+1	findbugs	652	the patch passed
		_ Other Tests _
+1	unit	389	hadoop-hdds in the patch passed.
-1	unit	2198	hadoop-ozone in the patch failed.
+1	asflicense	61	The patch does not generate ASF License warnings.
		8980

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.ozone.client.rpc.TestBCSID
	hadoop.ozone.TestSecureOzoneCluster
	hadoop.hdds.scm.pipeline.TestSCMPipelineManager
	hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
	hadoop.ozone.TestMiniOzoneCluster
	hadoop.ozone.om.TestOzoneManager
	hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException

Subsystem	Report/Notes
Docker	Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/2/artifact/out/Dockerfile
GITHUB PR	#1032
JIRA Issue	HDDS-1201
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 977bead8f735 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / d8bac50
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/2/artifact/out/diff-checkstyle-hadoop-hdds.txt
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/2/artifact/out/diff-checkstyle-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/2/testReport/
Max. process+thread count	3677 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	32	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	64	Maven dependency ordering for branch
+1	mvninstall	473	trunk passed
+1	compile	263	trunk passed
+1	checkstyle	75	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	865	branch has no errors when building and testing our client artifacts.
+1	javadoc	149	trunk passed
0	spotbugs	320	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	504	trunk passed
		_ Patch Compile Tests _
0	mvndep	27	Maven dependency ordering for patch
+1	mvninstall	508	the patch passed
+1	compile	246	the patch passed
+1	javac	246	the patch passed
-0	checkstyle	40	hadoop-ozone: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
+1	mvnsite	0	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	633	patch has no errors when building and testing our client artifacts.
+1	javadoc	147	the patch passed
+1	findbugs	517	the patch passed
		_ Other Tests _
+1	unit	249	hadoop-hdds in the patch passed.
-1	unit	1812	hadoop-ozone in the patch failed.
+1	asflicense	39	The patch does not generate ASF License warnings.
		6833

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
	hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
	hadoop.ozone.client.rpc.TestOzoneRpcClient
	hadoop.ozone.om.TestScmSafeMode
	hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
	hadoop.ozone.client.rpc.TestSecureOzoneRpcClient

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/3/artifact/out/Dockerfile
GITHUB PR	#1032
JIRA Issue	HDDS-1201
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux b131bec4f483 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / de6b7bc
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/3/artifact/out/diff-checkstyle-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/3/testReport/
Max. process+thread count	5124 (vs. ulimit of 5500)
modules	C: hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1032/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[nandakumar131]

+1, LGTM

[elek]

This patch introduced new checkstyle errors.

I recommend to add ozone label to the jira to get clear reports from the supplementary jenkins instance.

Contributors (who has no write access to the repository) can add ozone label with a new comment with the "/label ozone" message.

[hgadre]

@elek sorry about that. I have filed HDDS-1794 to fix this.

[elek]

Oh, no worries at all. It's a problem with the build system if these minor problems are not clearly visible immediately on the PR.

I just tried to share how can it be avoid, but I am also thinking to document it better on the wiki.

(ps: I am also experimenting with helper scripts such as ./hadoop-ozone/dev-support/checks/checkstyle.sh to make it easier to run the checks locally. For me it helps a lot, but checkstyle report will be more usable after HDDS-1735...)