Preference Optimization and Multi-Candidate Solution Sets

In multi-objective evolution,preference optimizationtreats the decision-maker's preferences as a guiding signal for search and selection, rather than applying a one-off scoring pass after the search ends. It brings two structural changes: first, it does not compress multiple objectives into a single scalar score in advance; second, it does not hand over just one "optimal solution," but outputs a set of candidate solutions that differ from one another in their trade-off structure, leaving the final decision to domain experts.

Why a Candidate Solution Set Is Needed

In multi-objective problems, there is no naturally unique "optimal solution." When objectives conflict, improving one objective for a solution often comes at the cost of worsening another, so comparable solutions line up along anIdeal Trade-off Frontier. Merging the objectives into a weighted sum in advance is equivalent to picking an implicit trade-off point on the decision-maker's behalf without knowing their true preferences; once real-world constraints or preferences change, that solution may instantly lose its value. What preference optimization must do is encode "which direction the human wants to look" into the search process, focusing search resources on the regions that are truly of concern.

How Preferences Enter the Search

Reference point: provides an ideal combination of objective values to guide the population toward that direction.
Objective priority: declares which objective is less expendable, used for non-dominated sorting or elimination decisions.
Weight interval: not a fixed weight, but an acceptable range of weights that allows multiple trade-offs to hold at the same time.
Interactive feedback: experts provide preferences progressively over multiple iterations, and the search narrows accordingly.

These forms all serve the same function:narrowing the region of interest and improving search efficiency, rather than collapsing the entire solution space into a single point in advance.

The Core of a Candidate Solution Set: Preserving Diversity

The value of a candidate solution set lies not in "having many" but insolutions within the set being spread out across objective trade-offs: the conservative end, the aggressive end, and the balanced middle each have representatives, covering different trade-off patterns. If you merely offer a few perturbed copies near the same solution, they cannot support any meaningful trade-off decision.

Diversity also does not maintain itself. Without mechanism intervention, the evolutionary process will spontaneously converge toward the currently optimal region, and the candidate set will eventually degenerate into repeated expressions of a single answer. Therefore, mechanisms such as distribution maintenance, crowding-distance control, or archive strategies are needed, treating "spread" as one of the explicit protection goals.

Boundary Distinction

Preferences do not mean deciding on someone's behalf, and diversity does not mean random noise. Differences among candidate solutions must be explainable by the objective trade-off structure—otherwise it is merely unusable, ineffective diversity.

The output form thus shifts from a "single optimal solution" to an "explainable, comparable candidate solution set": experts can combine domain knowledge and real-world constraints to make a choice, and feed the selection result back into the search process as a new round of feedback.

Candidate Solution Distribution Scatter Plot

The diagram uses scattered candidate points to represent the search output after preference optimization: reference points, objective priorities, weight intervals, or interactive feedback guide the search to narrow, while distribution maintenance and crowding control preserve diversity. Solutions within the set should cover the conservative end, the aggressive end, and the balanced middle, allowing experts to make the final trade-off in light of real-world constraints, rather than collapsing into a single solution in advance.

Question 1

What is the most fundamental difference between preference optimization and the approach of "search first, then score uniformly at the end"?

Preference optimization encodes the decision-maker's preferences into the search and selection process, rather than applying a one-off scoring pass after the search ends.

Preference optimization hands the final decision entirely to the algorithm to complete automatically.

Preference optimization eliminates conflicts between objectives, reducing the problem to a single-objective one.

Preference optimization merely changes weighting coefficients from decimals to integers, with no change in the computation method.

Question 2

In a candidate solution set, what makes a set of solutions "effectively diverse"?

All solutions in the set are copies obtained by making random perturbations near the same solution.

Solutions in the set are spread out across objective trade-offs, and every difference can be explained by the trade-off structure.

The greater the differences among solutions in the set, the better, even if such differences cannot be explained.

The more solutions in the set, the better; whether they resemble one another does not matter.

Question 3

What is the main role of preference forms such as "reference points" or "weight intervals" in the search?

Replacing all objective functions so that objective values no longer need to be computed.

Directly giving the unique global optimum so the search completes in one step.

Narrowing the search's region of interest and improving search efficiency, without collapsing the solution space into a single point in advance.

Guaranteeing that candidate solutions will never be duplicated.

Question 4

If distribution maintenance, crowding control, or archive strategies are missing, how does a candidate solution set typically degenerate?

The preference signal automatically becomes invalid, and the search process stops immediately.

The number of candidate solutions automatically increases, and diversity keeps improving.

Conflicts between objectives automatically disappear, and the problem automatically becomes single-objective.

Evolution naturally converges to the currently optimal region, and the candidate set becomes repeated copies near the same solution.

Question 5

When the output form shifts from a "single optimal solution" to an "explainable, comparable candidate solution set," what is the most direct change?

Differences among candidate solutions should be randomized as much as possible to increase the space of choices.

The problem is simplified to a single objective, and weight settings no longer matter.

The final trade-off is made by domain experts in light of real-world constraints, and the selection result can be fed back for further iteration.

The algorithm no longer requires any objective function evaluation.

Case Study: Scheduling and Routing Decisions for an Urban Delivery Fleet

Read the scenario below and answer the two questions.

The dispatch team of an urban delivery platform needs to generate daily scheduling and routing plans for its fleet. It must simultaneously weigh two conflicting objectives: delivery cost (lower is better) and average delivery latency (shorter is better); hard constraints also exist, such as load limits, restricted-traffic hours, and driver working hours. Previously, the team always used a weighted sum of the two objectives into a single score and output only one "optimal schedule." This time, they decided to introduce preference optimization and want the system to output a set of candidate solutions for dispatch experts to choose from.

Compared with directly weighting the two objectives into a single score, what does introducing preference optimization change in the "search process" and the "output form" respectively?

Answer:
Search process: preferences (reference points, objective priorities, weight intervals, or interactive feedback) act as selection and screening signals during iterations, concentrating search resources on the trade-off segments of concern, rather than first fixing an implicit trade-off point and then searching. Output form: the result changes from a single optimum to a set of candidate solutions located in different segments of the ideal trade-off frontier, such as representative solutions for three routes—conservative (cost-first), balanced (cost and latency in equilibrium), and aggressive (latency-first)—returning the final trade-off decision to the dispatch experts.

After the run ends, if all candidate solutions have very similar cost and latency values, which type of mechanism would you suspect is missing? And how would you judge whether the differences among these solutions are "effective"?

Answer:
Mechanism level: suspect that diversity-maintenance mechanisms are missing, such as distribution maintenance, crowding control, or archive strategies, causing the population to converge to a single point and the candidate set to degenerate into repeated copies near the same solution. Effectiveness judgment: project the candidate solutions onto the "cost–latency" plane. If the points are clearly spread out along the ideal trade-off frontier and the trade-off direction of each point can be explained by the objective trade-off structure (for example, one solution has lower cost but higher latency, while another is the opposite), the differences are effective; if the points cluster together, or the differences cannot be explained by the trade-off structure and are merely random jitter, this is ineffective diversity, and you need to adjust the selection pressure and distribution-maintenance strategy before running again.