SPV and Modeling

SPV may be used as a platform for comparison between a reference model and an HDL design. DPF (Digital Processing Framework), an SPV companion library, provides a framework for using Matlab [1] or C++ based reference models to verify signal processing designs. This document discusses the operational concept of DPF and the rationale behind it.

Reference Model Fidelity

The principle of the system is simple. Generate and drive stimulus, while recording the input of the device under test (DUT) and the signals to be checked. Then call the reference model with the recorded inputs to predict the expected outcome and compare the results.

In practice, wedding the reference model and the HDL is more involved. Much of the difficulty lies in the weaknesses of the reference model.

The reference model must be at least bit accurate [2] to check signal values and clock accurate if the values are time critical as well. However, it is difficult to keep the fidelity of the reference model high (exact) at all times over the course of development.

· Reference models often start their existence as feasibility studies and performance simulators. As such, they won’t be closely coupled to the design implementation. Adaptation will be necessary, but we’d want to keep the effort to a minimum.

· It is not desirable to keep the reference model implementation closely coupled to the HDL design, unless absolutely necessary. After all, the reference model’s advantage lies in its being easier to implement (think of calculating an FFT in Matlab versus a Verilog/VHDL implementation) and in being an independent interpretation of the design specification.

· Some periods of the simulation are “don’t care”, such as the idle times between packets in a communications system. It is not worthwhile to invest in the reference model for these periods.
In addition, some values may be “don’t care” under certain conditions by design.

DPF provides several ways to ease the accuracy requirement on the reference model.

· Recording envelope – The checks may optionally only be applied of the course of the recorded input. Idle times need not be recorded or checked.

· Quantitative strictness – The total number of values from each realm may optionally not be exact. Alternatively, the last reference value for a signal may be reused for comparison until the next reference model execution.

· Abort events – Simulation events can be used to suspend checking.

· Masking – Custom signal collectors may be programmed to mark values (or bits of values) as “don’t care”. These masked values will not be considered in error if different from the reference model’s prediction.

· Deviation tolerance – Test points can be configured to allow for a deviation (+/-) from the predicted results. This is useful when the reference model is inexact, such as when floating point calculations predict the values of fixed point hardware.[3]

Online versus Offline

Offline schemes generally take the following steps. 1) Test vector generation (input and predicted output), 2) simulation run, 3) vector comparison. The DPF concept is essentially online. [4] That is, randomized stimulus generation and reference model prediction are executed during the simulation. This provides the following advantages:

· Flexibility – Stimulus can be programmed to react to simulation events. (e.g. resend packet after collision) This results in much more lifelike scenarios, including situations which may not have even been thought of.

· Robustness – A statically planned test vector may become useless due to timing changes or marginal scenarios (e.g. high SNR). An online, programmed, response will result in less test maintenance.

· Faster test termination – There is no need to run a sequence of 100 packets just to discover that packet #2 is corrupt.

There are also disadvantages:

· Memory consumption – Both the simulator, the signal recording, and the reference model will at some point be running at one time.

· Redundant calls – The reference model is called for each recording no matter how many times the same exact test runs.[5]

Despite these disadvantages, DPF’s concept is online. This is due to the relatively low (and falling) price of computer memory and the risk one takes

in assuming that the same test will run in exactly the same fashion, no matter what changes take place in the design or reference model. Accordingly,

the advantages far outweigh the disadvantages.

Intended versus Recorded

The reference model input can be constructed in two ways. It can be fabricated based on what we intend to do. For example, we can tell the reference model to compute the output for a packet with a given length, modulation, and bit rate. Or, we can record the actual DUT inputs and feed that to the reference model. Each has advantages and disadvantages.

For the “intended” method, we have the ability to call the reference model before driving the stimulus.

Ö This results in better performance and memory consumption for two reasons.

o There is no need to record reference model input.

o There is no need to record signal values and time tags at the test points. When the reference model is called after the drive, the test point data must be recorded, because the comparison necessarily occurs after the reference model call. In order to place any errors that may be detected for user analysis, a time tag will have to be recorded for each value as well. When the reference model is called before the drive, comparison can occur as the drive progresses – so there is no need to save signal values and their time tags.[6]

Ö Missing HDL modules – When we need to fabricate inputs from (missing) sibling modules with interfaces to the DUT, there may be a bi-directional flow of information between them. If the reference model is called before the drive, we can use the same call both for output prediction and to help us fabricate the input to the DUT.

For the “recorded” method, we must record both the reference model input (DUT inputs) and the test point signal values (+ time tags).

Ö Reliability – The reference model will see exactly what the DUT sees. No more, no less, no different.

Ö Timing – Sometimes the exact timing of the inputs, or even the DUTs internals, can throw the reference model off. (This is especially true for clock accurate models). With this method, the exact timing of the inputs is preserved and it is easy to add an internal signal to the recording to help the reference model synchronize its calculations with the DUT.

Ö Missing reference model modules – When testing a DUT within the confines of a larger system, there may not be a reference model available for all of the system parts. In this case, it may not be possible to feed the generated “intent” to the reference model, as its concepts do not include the larger system. In this case, the “recorded” method is the obvious solution, since it records signals on and within the DUT.

Ö Lab recording injection – In the prototype laboratory, a far greater volume of testing can take place, at the expense of observability. However, if we can record enough inputs and state information in the lab (possibly by running the same scenario multiple times), we can replay a problematic scenario in simulation. There, we’ll have full observability into the depths of the DUT, but to fully leverage the simulation we will require comparison to the reference model. As in the previous paragraph, there may be no simple way to reliably feed the “intent” to the reference model.

The design of DPF supports the “recorded” method with comparison taking place after the drive. Despite the disadvantages, we found that “Timing” and “Missing reference model modules” were the most intractable problems facing our users. [7]

In practice, as a workaround for the performance and memory problems in large scale designs, we can shut off most test points, leaving only test points at major block outputs. If a problem was detected with the block, then the test would be rerun, with all the test points enabled for the problem block only.

A more complete solution is to break up the recording period into smaller chunks and call the reference model more often on less data. This may, however, be unnatural to systems with an obvious unit of atomic data (e.g. a packet).

As regards the “Missing HDL modules” problem; if absolutely necessary, it is still possible to call the reference model as in the “intended” method, but save the output data until the end of the drive for comparison. Alternatively, it may be possible to generate the missing module’s data on a clock-by-clock basis as part of the driver’s logic, using the DUT intermediate output as input to a specialized function which will model the behavior of the missing module, instead of the reference model. [8]

Post Simulation Analysis

Ultimately, the algorithm developer and the HDL designer will have to combine efforts in order to debug the DUT. The entire effort described up to here is useless if there is no easy way to analyze the DUT simulation results together with those of the reference model.

The system must be capable of reporting errors, including identification of the errant signals, the expected and actual values, and the (simulation) time of the error. This, however, is just the beginning of the task faced by those debugging the DUT. At this point, it is necessary to analyze the behavior of both the DUT and the reference model, not just when the error occurred, but up to that point (and sometimes beyond), as well.

To address these issues, DPF defines the concept of a Test Point. A Test Point defines the mapping between HDL signals and reference model variables. It also defines the timing (when the signals/variables are “valid”) for comparison as well as other parameters, like the tolerance mentioned above.

With this information in hand, DPF is capable of making the comparisons – and logging comparison failures, including terminating the simulation. Furthermore, as an aid to debugging, the reference model data is written to an “instrumentation” file in the vcd standard format, using the same timing as the comparison. In other words, the simulator’s waveform viewer can be used to view both the DUT’s trace and the reference model output, synchronized together on the same time axis.

Summary

DPF is SPV’s extension library for verification of signal processing applications. It supplies a framework and methodology for comparing a reference model’s predictions to the hardware simulation. It eases the task by taking into account the difficulties commonly met in such environments while providing an online, reactive, test environment.

---

[1] Matlab integration is accomplished with either: using the Matlab compiler to create an executable library which can then be used without a Matlab license or using the Matlab engine interface which launches a Matlab instance. The differences between these two methods are relatively minor in terms of the C++ code required. The more significant difference is in licensing.

[2] Or close to bit accurate – See later on.

[3] In practice, a deviation in one block’s output will often be propagated, and possibly amplified, through other blocks. Thus, the deviation tolerance is most useful at the block level test and a true bit accurate model will have to be created for larger scale testing.

[4] Though, if desired, it would certainly be possible to read prepared input and expected results from files.

[5] Conceivably, a cache scheme could be used where the results of a reference model call are stored on disk and retrieved when identical inputs are seen in the simulation.

[6] Though there is a need to save the reference model output for the comparison. However, peak memory consumption should still be lower as it is possible to progressively release the memory of the reference model output as the drive (and comparison) progresses. And, there is no need for time tags at all.

[7] At one point in the development of DPF, both methods were supported. However, this caused the users (and us!) much confusion, so it was decided to support the more robust of the two.

[8] Note that the “specialized function” could be a part of the reference model’s code.