Show of hands: how many of us truly understand how your build automation tool builds its dependency tree? Now lower your hand if you understand because you work on build automation tools. Thought so!
One frustrating responsibility of software engineers is understanding your project’s dependencies: what transitive dependencies were brought in and by whom; why v1.3.1 is used when v1.2.10 was declared; what resulted when the transitive dependencies changed; how did multiple versions of the same artifact occur?
Every software engineer has piped a dependency tree into a text file, searched for specific artifacts, and then worked their way back up identify its origin. For anything other than trivial projects, creating a mental map of the dependencies is extremely difficult, if not impossible.
I faced this problem when starting a new job with a mature code base, presenting a challenge to assemble the puzzle pieces. I’ve previously worked with graph databases and thought a graphical view of the dependency artifacts could be created using Neo4J, which resulted in DependencyLoader.
Note: this is not a tutorial on graph databases, nor does this tutorial require a background in graph databases. If interested, Neo4J has tutorials and white papers to help you get started.
Set Up Environment
Install Java
Java 11 or later is required, if not already available install your favorite OpenJDK flavor.
Install Neo4J
The tutorial requires a Neo4J database into which the dependency information is loaded, preferably unshared as the loader purges the database before each run. You have been warned!
Neo4J provides personal sandboxes, ideal for short-term projects like this tutorial.
Alternatively, install Neo4J locally on your desktop or laptop. Homebrew simplifies MacOS installations: brew install neo4j && brew services start neo4j
Before continuing, confirm access to your Neo4J database using the browser, using either the link and credentials for the Neo4J sandbox or locally at http://localhost:7474. The default credentials for a local install is neo4j/neo4j; upon successful login you are forced to change the password.
Clone Repositories
The neo4j-gradle-dependencies repository contains the for loading the dependencies into Neo4J. This tutorial will generate a dependency graph for spring-boot. You must clone these two repositories.
Scott.Sosna@mymachine src% git clone git@github.com:scsosna99/neo4j-gradle-dependencies.git
Scott.Sosna@mymachine src% git clone git@github.com:spring-projects/spring-boot.git
Note: local Gradle is not required as both repositories use the Gradle Wrapper, which download all necessary components the first time the wrapper is used.
Generate Dependencies
DependencyLoader takes the dependency tree generated by Gradle as input Though multiple configurations may be loaded together – i.e. compileClasspath, runtimeClasspath, testCompileClasspath, testRuntimeClasspath – starting with a single configuration is simpler to navigate, especially for a tutorial.
To generate dependencies for all configurations:
gradle dependencies./gradlew dependencies
To generate dependencies for a single configuration
gradle dependencies --configuration <configuration>./gradlew dependencies --configuration <configuration>
Generate Spring Boot Dependencies
This tutorial creates a dependency graph in Neo4J using the compileClasspath dependencies of Spring Boot. From the directory where the repositories were cloned, execute the following commands:
Scott.Sosna@mymachine src% cd spring-boot/spring-boot-project/spring-boot
Scott.Sosna@mymachine spring-boot% ./gradlew dependencies --configuration compileClasspath > dependencies.out
The file dependencies.out contains the compile-time classpath dependencies for Spring Boot.
Load Dependencies
First, confirm the connection URL and authentication credentials in DependencyLoader.java and modify if necessary.
Execute the following commands to load the Spring Boot dependencies into Neo4j:
Scott.Sosna@mymachine spring-boot% cd ../../../neo4j-gradle-dependencies
Scott.Sosna@mymachine neo4j-gradle-dependencies% ./gradlew clean run --args="../spring-boot/spring-boot-project/spring-boot/dependencies.out"
When successful, the final output lines from gradle are:
BUILD SUCCESSFUL in 2s
3 actionable tasks: 2 executed, 1 up-to-date
View Dependencies
Multiple tools are available for displaying Neo4J graphs, but the built-in browser tool is adequate for this tutorial.
Show the Complete Tree
The query MATCH(a) RETURN a is the relational-equivalent of SELECT * FROM <table>
View Details of an Artifact
Each artifact found creates a node whose properties identify the artifact (groupId/artifactId) and its type, shown on the right-side pane.
View Details of a Dependency
Each dependency is created as a relationship whose properties identify the specifics about the dependency: configuration, specified/version, configuration. The dependency selected below shows spring-security:spring-web depends on io.micormeter:micrometer-observation, but the spring-web specified version 1.10.7 was resolved as version 1.11.0.
Traverse Dependencies
Neo4J allows you to explore the graph node-by-node, allowing you to manually expand the graph, node by node, providing a way to explore specific areas of the dependency tree.
Assume that you want to understand the dependencies for the artifact io.projectreactor.netty:reactor-netty-http. First, we’ll query Neo4J for that specific node.
MATCH(a:Artifact {groupId: 'io.projectreactor.netty', artifactId: 'reactor-netty-http'}) RETURN a
Double-clicking on the node shows its neighboring nodes – the artifact(s) depended on it and the artifact(s) it depends on.
This expanded graph shows one artifact that is depended on it – the root of the project with an artifact type PROJECT and six other dependencies on which it’s depended.
Next, double-click on io.netty:netty-codehttps://github.com/netty/netty/tree/4.1/codec-httpc-http to show the next level of dependencies. Note that besides the relationships (dependencies) of the selected node, additional relationships for nodes already on the graph may be shown.
Identify Version Mismatch
Gradle’s dependency output indicates where the specified version was not the version resolved by Gradle. The properties on the dependency (relationship) can be used in a Neo4J query, restricting the relationships shown and the attached artifacts (nodes).
MATCH (a:Artifact)-[d:DEPENDS_ON]->(b:Artifact) WHERE d.specifiedVersion<>d.resolvedVersion RETURN a,b,d
Neo4J can return results in a tabular format for easier review, if necessary.
MATCH (a:Artifact)-[d:DEPENDS_ON]->(b:Artifact) WHERE d.specifiedVersion<>d.resolvedVersion RETURNa.name AS source, b.name AS dependency, d.specifiedVersion AS specified, d.resolvedVersion AS resolved
Additional Information
mappings.out
The mappings.out file allows you to customize artifact type assigned to a node based on artifacts groupId, most commonly to specifically identify artifacts created by your organization.
Input Directory
The command line argument for DependencyLoader may be a directory containing multiple Gradle dependency trees loaded into the same Neo4J database. This helps in understanding dependencies of related projects with separate build.gradle files.
Constrained and Omitted
Gradle identifies certain dependencies as Constrained and Omitted. Currently those are not loaded, but would be easy to include, likely by creating additional properties for the relationships.
