build!: cleanup dependency tree, shaded JAR, and deploy properly (#632)

* fix: exclude maven metadata and jacoco from shaded JAR

Excluding maven metadata fixes the following warning from
`deployDistribution.sh`:
```
[WARNING] Artifact org.jlab.coat:coat-libs JAR (about to be
installed/deployed) contains Maven Plugin metadata for conflicting
coordinates: org.jacoco:jacoco-maven-plugin. Your JAR contains rogue
Maven Plugin metadata. Possible causes may be: shaded into this JAR some
Maven Plugin or some rogue resource.
```

Excluding `jacoco` since only used here for coverage testing

* fix: now we can use `mvn deploy`

* build: work in progress...

* doc: deployment notes

* fix: mermaid

* fix: mermaid

* fix: mermaid

* fix: mermaid

* fix: mermaid

* fix: mermaid

* fix: more progress and an unhelpful git commit message

* fix: flow control

* doc: tersify

* build: cleanup

* fix: clear `ejml` warnings

* build: untangling dependencies

* build: openjfx

* fix: more untangling

* fix: classifiers

* fix: only include `junit` where actually needed

* fix: even more untangling

* fix: resolved almost all the transitive dependencies

* fix: the knot is untangled!!!

* doc: mermaid

* doc: mermaid

* doc: mermaid

* fix: remove test-scoped dependency `org.jlab.jnp:jnp-hipo4` from reconstruction tests

This was strange. If I scope `jnp-hipo4` to `test` in the POM file, as
suggested by `mvn dependency:analyze`, I would get build errors in very
specific methods, for example, `ECCommon.readStripsHipo`:
- my IDE (Eclipse JDT LSP) correctly sees `DataEvent` as `org.jlab.io.base.DataEvent`
- the Maven build (`maven-compiler-plugin`) would get confused and think
  that `DataEvent` comes from somewhere in `jnp-hipo4`, even though this
  package was not being imported

With the help of @raffaelladevita we found that `jnp-hipo4` dependency
could actually be removed from all of the tests, so we can avoid the
issue altogether.

* fix: need `CLAS12DIR` in the test

* ci: tell dependabot to not update `ejml-simple`

* refactor: clarify which deps are for conflict resolution

* ci: dependency analysis

* fix(ci): dependence

* fix: no progress printout

* fix: stop screaming about unused shade deps

* ci: release with version number in name; open issue if fail

* fix: syntax and issue template

* test: dump classpath for tests

* perf(ci): move unit tests to their own job

* ci: update needs for `final`

* fix: exclude `coat-libs` from javadoc aggregation

* fix: DRY violation of plugin version

* fix: `coat-lib` -> `coat-libs`

* fix: exclude `coat-libs` from `javadoc` in parent POM

* fix: use `skip` configuration to self-exclude `coat-libs`

* fix: `execution`

* fix: explicit control in CI config

* fix: install all modules' JARs

* fix: env

* feat: warn if a `target` directory persists after `mvn clean`

* fix: improve installation

* fix: claspathology

* fix: source after clara

* doc: choose version 1

* test: disable fail-fast

* fix: appease clara and install `common-tools` to `lib/services`

* fix: add `cnuphys/snr` and `clas-math` to `coat-libs`, since...

...they were in the original `coat-libs` somehow

* doc: general deployment procedure

* fix: small stuff

* fix(coat-libs): `org.jlab.clas` -> `org.jlab.coat`

* fix: actually deploy

* fix(ci): `-pl` args

* fix: release permission

Author: c-dilks

.github/dependabot.yml

@@ -18,6 +18,7 @@ updates:
       - jhep
     ignore:
       - dependency-name: "org.jlab:groot" # since version numbers are not in order
+      - dependency-name: "org.ejml:ejml-simple" # keep version the same as `j4ml:j4ml-clas12:jar:0.9-SNAPSHOT`; see pull requests #636 and #632
   - package-ecosystem: "github-actions"
     directory: "/"
     schedule:

.github/issue_release.md

@@ -0,0 +1,4 @@
+---
+title: Release build failed for tag ${{ env.TAG_NAME }}
+---
+Workflow Run: {{ env.REPO_URL }}/actions/runs/{{ env.RUN_ID }}

.github/workflows/ci.yml

@@ -26,6 +26,7 @@ env:
   javadoc_version: 24 # newer than `java_version` for better javadoc
   groovy_version: 4.x
   CCDB_CONNECTION: 'sqlite:////cvmfs/oasis.opensciencegrid.org/jlab/hallb/clas12/sw/noarch/data/ccdb/ccdb_latest.sqlite'
+  nthreads: 4
 
 jobs:
 
@@ -51,35 +52,48 @@ jobs:
           cvmfs_repositories: 'oasis.opensciencegrid.org'
       - name: cvmfs
         run: ls /cvmfs/oasis.opensciencegrid.org/jlab/hallb/clas12/sw/noarch/data/ccdb/ccdb_latest.sqlite
-      - name: echo CCDB_CONNECTION
-        run: echo $CCDB_CONNECTION
+      - name: bump version to tag if tag trigger
+        if: ${{ github.ref_type == 'tag' }}
+        run: libexec/version-bump.sh ${{ github.ref_name }}
       - name: build
-        run: ./build-coatjava.sh --spotbugs --unittests --no-progress -T4
+        run: ./build-coatjava.sh --no-progress -T${{ env.nthreads }}
       - name: tar # tarball to preserve permissions
         run: tar czvf coatjava.tar.gz coatjava
       - uses: actions/upload-artifact@v4
         with:
           name: build_${{ matrix.runner }}
           retention-days: 1
           path: coatjava.tar.gz
+
+  # tests
+  #############################################################################
+
+  unit_tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-java@v4
+        with:
+          java-version: ${{ env.java_version }}
+          distribution: ${{ env.java_distribution }}
+      - uses: cvmfs-contrib/github-action-cvmfs@v5
+        with:
+          cvmfs_repositories: 'oasis.opensciencegrid.org'
+      - name: unit tests and spotbugs
+        run: ./build-coatjava.sh --spotbugs --unittests --no-progress -T${{ env.nthreads }}
       - name: collect jacoco report
-        if: ${{ matrix.runner == 'ubuntu-latest' }}
         run: validation/jacoco-aggregate.sh
       - name: publish jacoco report
-        if: ${{ matrix.runner == 'ubuntu-latest' }}
         uses: actions/upload-artifact@v4
         with:
           name: jacoco_report
           path: publish/
           retention-days: 1
 
-  # tests
-  #############################################################################
-
   test_coatjava:
     needs: [ build ]
     strategy:
-      fail-fast: true
+      fail-fast: false
       matrix:
         runner:
           - ubuntu-latest
@@ -116,8 +130,6 @@ jobs:
           cvmfs_repositories: 'oasis.opensciencegrid.org'
       - name: untar build
         run: tar xzvf coatjava.tar.gz
-      - name: echo CCDB_CONNECTION
-        run: echo $CCDB_CONNECTION
       - name: run test
         run: |
           cd validation/advanced-tests
@@ -146,39 +158,32 @@ jobs:
       - name: test run-groovy
         run: coatjava/bin/run-groovy validation/advanced-tests/test-run-groovy.groovy
 
-  release:
+  dependency_analysis:
     needs: [ build ]
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+      - name: Set up JDK
+        uses: actions/setup-java@v4
+        with:
+          java-version: ${{ env.java_version }}
+          distribution: ${{ env.java_distribution }}
       - uses: actions/download-artifact@v4
         with:
           name: build_ubuntu-latest
-      - name: Release
-        uses: softprops/action-gh-release@v2
-        if: startsWith(github.ref, 'refs/tags/')
-        with:
-          files: coatjava.tar.gz
-        env:
-          GITHUB_TOKEN:  ${{ secrets.GITHUB_TOKEN }}
-
-  final:
-    needs:
-      - build
-      - test_coatjava
-      - test_run-groovy
-      - release
-    runs-on: ubuntu-latest
-    steps:
-      - name: pass
-        run: exit 0
+      - name: untar build
+        run: tar xzvf coatjava.tar.gz
+      - name: print dependency tree
+        run: mvn dependency:tree -Ddetail=true --no-transfer-progress
+      - name: dependency analysis # note: skips `coat-libs`, since shaded JAR dependencies are "unused" according to `dependency:analyze`
+        run: mvn dependency:analyze -DfailOnWarning=true -pl '!org.jlab.coat:coat-libs' --no-transfer-progress
 
   # documentation
   #############################################################################
 
   generate_documentation:
     runs-on: ubuntu-latest
-    needs: [ build ]
+    needs: [ unit_tests ]
     steps:
       ### mkdocs
       - uses: actions/checkout@v4
@@ -198,10 +203,10 @@ jobs:
         with:
           java-version: ${{ env.javadoc_version }}
           distribution: ${{ env.java_distribution }}
-      - name: build coatjava
+      - name: build coatjava javadocs # javadoc:aggregate output dir cannot be controlled, so assume the latest "standard" path and `mv` it
         run: |
-          ./build-coatjava.sh --no-progress --docs
-          mv coatjava/share/doc/coatjava/html pages/javadoc
+          mvn javadoc:aggregate -pl '!org.jlab.coat:coat-libs' --no-transfer-progress
+          mv target/reports/apidocs pages/javadoc
       ### upload artifacts
       - uses: actions/upload-pages-artifact@v3
         with:
@@ -222,3 +227,49 @@ jobs:
       - name: deployment
         id: deployment
         uses: actions/deploy-pages@v4
+
+  # finalize
+  #############################################################################
+
+  release_build:
+    if: ${{ github.ref_type == 'tag' }}
+    needs: [ build ]
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      issues: write
+    env:
+      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      REPO_URL: ${{ github.event.repository.html_url }}
+      RUN_ID: ${{ github.run_id }}
+      TAG_NAME: ${{ github.ref_name }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/download-artifact@v4
+        with:
+          name: build_ubuntu-latest
+      - name: rename artifact
+        run: |
+          tar xzf coatjava.tar.gz
+          mv coatjava{,-${{ env.TAG_NAME }}}
+          tar czf coatjava-${{ env.TAG_NAME }}{.tar.gz,}
+      - name: release
+        uses: softprops/action-gh-release@v2
+        with:
+          files: coatjava-${{ env.TAG_NAME }}.tar.gz
+      - name: open issue if failed
+        if: ${{ cancelled() || failure() }}
+        uses: JasonEtco/create-an-issue@v2
+        with:
+          filename: .github/issue_release.md
+
+  final:
+    needs:
+      - test_coatjava
+      - dependency_analysis
+      - test_run-groovy
+      - generate_documentation
+    runs-on: ubuntu-latest
+    steps:
+      - name: pass
+        run: exit 0

.gitignore

@@ -1,12 +1,16 @@
 # installed:
 /coatjava
-/myLocalMvnRepo
 target
 /publish
 /docs/javadoc/*
 /clara-home
 tmp
 
+# deployment
+/myLocalMvnRepo
+/coatjava-*.tar.gz
+dependency-reduced-pom.xml
+
 *.class
 *lundfiles*.dat
 .DS_Store

README.md

@@ -5,6 +5,7 @@
 
 - [**Documentation Homepage**](https://jeffersonlab.github.io/coatjava)
 - [API Documentation (Javadoc)](https://jeffersonlab.github.io/coatjava/javadoc)
+- [Developer notes](/docs/dev_notes.md)
 
 ----
 

bin/bg-merger

@@ -7,7 +7,7 @@ export MALLOC_ARENA_MAX=1
 split_cli $@
 
 java -Xmx1536m -Xms1024m -XX:+UseSerialGC ${jvm_options[@]} \
-    -cp "$CLAS12DIR/lib/clas/*:$CLAS12DIR/lib/services/*:$CLAS12DIR/lib/utils/*" \
+    -cp ${COATJAVA_CLASSPATH:-''} \
     org.jlab.analysis.eventmerger.EventMerger \
     ${class_options[@]}
 

bin/bos2hipo

@@ -3,6 +3,6 @@
 . `dirname $0`/../libexec/env.sh
 
 java -Xms1024m -XX:+UseSerialGC \
-    -cp "$CLAS12DIR/lib/clas/*:$CLAS12DIR/lib/plugins/*" \
+    -cp ${COATJAVA_CLASSPATH:-''} \
     org.jlab.io.utils.Bos2HipoEventBank \
     $*

bin/cvt-compare

@@ -5,6 +5,6 @@
 export MALLOC_ARENA_MAX=1
 
 java -Xmx1536m -Xms1024m -XX:+UseSerialGC \
-    -cp "$CLAS12DIR/lib/clas/*:$CLAS12DIR/lib/services/*:$CLAS12DIR/lib/utils/*" \
+    -cp ${COATJAVA_CLASSPATH:-''} \
     org.jlab.rec.cvt.Geometry \
     $*

bin/daqEventViewer

@@ -3,6 +3,6 @@
 . `dirname $0`/../libexec/env.sh
 
 java -Xms1024m \
-    -cp "$CLAS12DIR/lib/clas/*:$CLAS12DIR/lib/services/*:$CLAS12DIR/lib/utils/*" \
+    -cp ${COATJAVA_CLASSPATH:-''} \
     org.jlab.detector.examples.RawEventViewer \
     $*

bin/dclayereffs-ana

@@ -3,6 +3,6 @@
 . `dirname $0`/../libexec/env.sh
 
 java -Dsun.java2d.pmoffscreen=false -Xmx2048m -Xms1024m \
-    -cp "$CLAS12DIR/lib/clas/*:$CLAS12DIR/lib/utils/*:$CLAS12DIR/lib/services/*" \
+    -cp ${COATJAVA_CLASSPATH:-''} \
     org.jlab.service.dc.LayerEfficiencyAnalyzer \
     $*