Troubleshooting
Simulation Hangs
Did you directly call an async def
function without using await
;
or a coroutine
without using yield
?
If you want to exit cocotb and the simulator using Control-C (the Unix signal SIGINT
) but this doesn’t work,
you can try Control-\ (the Unix signal SIGQUIT
).
Increasing Verbosity
If things fail in the VPI/VHPI/FLI area, check your simulator’s documentation to see if it has options to
increase its verbosity about what may be wrong. You can then set these options on the make command line
as COMPILE_ARGS
, SIM_ARGS
or EXTRA_ARGS
(see Build options and Environment Variables for details).
If things fail from within Python, or coroutines aren’t being called when you expect, the
COCOTB_SCHEDULER_DEBUG
variable can be used to (greatly) increase the verbosity of the scheduler.
Building cocotb In Development Mode
By default cocotb binaries installed from PyPi are stripped, i.e. they do not contain debug symbols. Rebuilding cocotb from source can add this information back, making it significantly easier to debug cocotb code. In the following, we’ll assume the use of a Linux machine for debugging cocotb, which simplifies the process significantly.
First, install all build requirements as listed at Installing the Development Version.
Then execute the following commands to download a development version of cocotb and prepare a shell environment with this a development build of cocotb available:
$ # Obtain the latest development version of cocotb through git
$ git clone https://github.com/cocotb/cocotb.git
$ # Build cocotb in debug mode, and enter a bash shell
$ cd cocotb
$ nox -s dev -- /bin/bash
Attaching a Debugger
C and C++
The most convenient way to debug the cocotb C code and the interaction between cocotb and the simulator is using GDB. This is a two-step process:
Run the simulation with
COCOTB_ATTACH
set.Use
gdb -p
to attach to the simulator process.
Have a look at Build options and Environment Variables for various useful variables related to debugging.
Example:
Debug the test test_array_simple
with Questa, using the VHDL toplevel and the VHPI.
Run the simulation and take note of the process identifier (PID) displayed after the simulator starts up.
$ make -C tests/test_cases/test_array_simple SIM=questa TOPLEVEL_LANG=vhdl VHDL_GPI_INTERFACE=vhpi COCOTB_ATTACH=300 COCOTB_LOG_LEVEL=trace ... # -.--ns ERROR gpi ..mbed/gpi_embed.cpp:154 in _embed_init_python Waiting for 300 seconds - attach to PID 9583 with your debugger
Open a new terminal window or tab, and attach GDB to the running process.
$ gdb -p 9583 ... 48 r = INTERNAL_SYSCALL_CANCEL (clock_nanosleep_time64, clock_id, flags, req, (gdb) # Set breakpoints or do anything else you'd like to do. Finally, let the simulation run: (gdb) continue Continuing. [Inferior 1 (process 9583) exited normally] (gdb) quit
Python
When executing the Makefile to run a cocotb test, a Python shell interpreter is called from within the
VPI/VHPI/FLI library.
Hence it is not possible to directly attach a Python debugger to the Python process being part of the simulator that uses the aforementioned library.
Using import pdb; pdb.set_trace()
directly is also frequently not possible,
due to the way that simulators interfere with stdin
.
To successfully debug your Python code use the remote_pdb Python package to create a pdb instance accessible via a TCP socket:
In your code insert the line:
from remote_pdb import RemotePdb; rpdb = RemotePdb("127.0.0.1", 4000)
Then before the line of code you want the debugger to stop the execution, add a breakpoint:
rpdb.set_trace() # <-- debugger stops execution after this line <your code line> # <-- next statement being evaluated by the interpreter
Run the Makefile so that the interpreter hits the breakpoint line and hangs.
Connect to the freshly created socket, for instance through telnet:
telnet 127.0.0.1 4000
Embedding an IPython shell
Added in version 1.4.
A prebuilt test is included to easily launch an IPython shell in an existing design.
- cocotb.ipython_support.run_ipython(dut)[source]
A test that launches an interactive Python shell.
Do not call this directly - use this as
make MODULE=cocotb.ipython_support
.Within the shell, a global
dut
variable pointing to the design will be present.
To embed a shell within an existing test, where it can inspect local variables, the embed()
function can be used.
- async cocotb.ipython_support.embed(user_ns={})[source]
Start an IPython shell in the current coroutine.
Unlike using
IPython.embed()
directly, theawait
keyword can be used directly from the shell to wait for triggers. Theyield
keyword from the legacy Generator-based coroutines is not supported.This coroutine will complete only when the user exits the interactive session.
- Parameters:
user_ns (dict) – The variables to have made available in the shell. Passing
locals()
is often a good idea.cocotb
will automatically be included.
Notes
If your simulator does not provide an appropriate
stdin
, you may find you cannot type in the resulting shell. Using simulators in batch or non-GUI mode may resolve this. This feature is experimental, and not all simulators are supported.
Setting make variables on the command line
When trying to set one of the make variables listed in Build options and Environment Variables from the command line,
it is strongly recommended to use an environment variable, i.e.
EXTRA_ARGS="..." make
(for the fish
and csh
shells: env EXTRA_ARGS="..." make
)
and not make EXTRA_ARGS=...
.
This is because in the case of the discouraged make EXTRA_ARGS=...
,
if one of the involved Makefiles contains lines to assign (=
) or append (+=
) to EXTRA_ARGS
internally,
such lines will be ignored.
These lines are needed for the operation of cocotb however,
and having them ignored is likely to lead to strange errors.
As a side note,
when you need to clear a Makefile variable from the command line,
use the syntax make EXTRA_ARGS=
.
GLIBCXX_3.4.XX
not found
This error can occur on Linux, and will raise ImportError: /some/libstdc++.so.6: version `GLIBCXX_3.4.XX' not found
.
This occurs because an older non-C++11 version of libstdc++
is being loaded by the simulator or cocotb.
It is usually an issue with your environment, but sometimes can occur when using a very old version of certain simulators.
Check your environment
To see if your environment is the issue, look at the value of the LD_LIBRARY_PATH
environment variable.
Ensure the first path in the colon-delimited list is the path to the libstdc++
that shipped with the compiler you used to build cocotb.
echo $LD_LIBRARY_PATH
This variable might be empty, in which case the loader looks in the system’s libraries. If the library you built cocotb with is not first, prepend that path to the list.
export LD_LIBRARY_PATH=/path/to/newer/libraries/:$LD_LIBRARY_PATH
Check your simulator
Sometimes, simulators modify the LD_LIBRARY_PATH
so they point to the libraries that are shipped with instead of the system libraries.
If you are running an old simulator, the packaged libraries may include a pre-C++11 libstdc++
.
To see if your simulator is modifying the LD_LIBRARY_PATH
, open the simulator up to an internal console and obtain the environment variable.
For example, with Mentor Questa and Cadence Xcelium, one could open a Tcl console and run the env command to list the current environment.
The LD_LIBRARY_PATH
should appear in the list.
If the simulator does modify the LD_LIBRARY_PATH
, refer to the simulator documentation on how to prevent or work around this issue.
For example, Questa ships with GCC.
Sometimes that version of GCC is old enough to not support C++11 (<4.8).
When you install cocotb, pip uses the system (or some other) compiler that supports C++11.
But when you try to run cocotb with the older Questa, it prepends the older libraries Questa ships with to LD_LIBRARY_PATH
.
This causes the older libstdc++
Questa ships with to be loaded, resulting in the error message.
For Questa, you can use the -noautoldlibpath
option to turn off the LD_LIBRARY_PATH
prepend to resolve this issue.
Using cocotb with more than one Python installation
Users of cocotb with more than one installation of a single Python version (including conda env
users)
must take care not to re-use cached versions of the installed cocotb package.
If this isn’t done, running simulations fails with errors like libpython3.7m.so.1.0: cannot open shared object file: No such file or directory
.
Cocotb builds binary libraries during its installation process.
These libraries are tailored to the installation of Python used when installing cocotb.
When switching between Python installations, cocotb needs to be re-installed without using cached build artifacts, e.g. with pip install --no-cache-dir cocotb
.
On Linux distributions, setting LD_DEBUG=libs
(example: LD_DEBUG=libs make SIM=verilator
) prints detailed output about which libraries are loaded from where.
On Mac OS, you can use DYLD_PRINT_LIBRARIES=1
instead of LD_DEBUG=libs
to get similar information.
On Windows, use Process Explorer.
Further details are available in #1943.