Climbing FFLib

#apex#salesforce#optimisation#hill-climbing#case-study

When testing a tool, it is useful to test it at the extremes. For Climber, one extreme is a mature code base that is unlikely to carry much technical debt. Testing at this level finds problems in the tool as much as in the code.

We looked for a public code base we could climb and share without any IP risk. FFLib is a great candidate because it is very mature and fully public.

This post follows on from Hill Climbing Apex Code, where we introduced Climber. The full results are in the case study.

How we climbed it

We climbed FFLib twice.

  1. Out of the box. We ran Climber with its default configuration. Governor limit use is one of the dimensions Climber scores, so we first added tests that measure limit headroom. FFLib does not include tests like these.
  2. Adjusted for style. The first climb showed that many low scores came from PMD rules that require braces. FFLib commonly leaves out the braces on single-line conditionals and loops, so 198 of its 229 PMD findings came from these rules. We turned off the four brace rules and climbed again.

We did no anchoring before either climb. Anchoring is generally a good idea, but it takes domain knowledge to define correct behaviour and the right tests. We did not want to presume that depth of knowledge of FFLib.

This is the scoring map before the adjusted climbs. Each climb ran an ascent on each group in the map. Hover/tap on a group to see its scores for each dimension.

Figure 1. FFLib map adjusted for style.

The climbs

The out of the box climb used Climber’s default PMD rules.

Figure 2. Out of the box ascents.

The adjusted climb left the four brace rules out of the score.

Figure 3. Ascents adjusted for style.

In the adjusted climb, the average score rose from 83.7 to 90.0. This small gain confirms that FFLib is a mature library. The gain is not zero, though, and the retrospectives from both climbs surfaced a few observations that may be useful to the FFLib authors.

What we learned

We learned how to describe Climber in terms of what it is good for and what it is not good for. It is good for finding and fixing technical debt, in both greenfield and brownfield code. It is not a replacement for a coding agent writing new code.

Configuring the scoring tools to your own standards matters. Doing it up front would have saved us a whole climb. Tuning Climber also reduces the iterations and resources a climb needs. The adjusted climb took 75 iterations, against 146 for the out of the box climb.

Climbing explores the small gaps that an agent asked to scan the whole system in one pass would probably miss. Big gains are made separately from small gains, and the small gains are tried repeatedly until there are none left to find. Each step also runs with a fresh, small context, which makes agents more capable. In theory, this makes climbing better at finding improvements than an agent asked to find them in one or more passes. That remains to be proven with rigorous testing, which is a job for the future.

Most big gains in software come from architecture decisions, and climbing will not make those for you. Architecture and design remain a separate process. Climbing can still inform that process. Just as anchoring is useful after a first ascent, the retrospectives give you knowledge for making better architecture decisions. You can climb a prototype or existing code to learn what the design needs.

Summary

This exercise confirmed that climbing a mature library is unlikely to produce large gains, because public use has already cleaned it up.

Many private code bases carry much more technical debt, often because of time pressure and other factors. Climbing those code bases can produce much larger gains.

Climbing FFLib also produced six improvements to Climber itself. We are climbing the climber.