DAP protocol library

Since about end of September 2020 I’m working on an implementation of DAP (Debug Adapter protocol), which allows tools like Visual Code to talk with various debuggers. Of course there is easy to access specification, but there is no lightweight, non-proprietary, easy to integrate solution, which could be used on real, less capable machines, emulators or native debuggers. Protocol itself is pretty straightforward, just JSON messages in plain text flying from one side to the other.

As long time Atari developer (well, don’t ask me why) I find debugging process quite tedious. We have modern compilers (like gcc 11, which is great), no standard IDE’s, you have to assemble proper toolchain yourself and know how to. Writing code and building it is an easy part. Problems arise when there are errors and you have to investigate them on a real hardware. When you write a code with c/c++ you have to debug in m68k assembly, source level debugging doesn’t exist. Operating system on Atari isn’t bullet proof either – there is no memory protection and additionally Atari first rule of optimisation is “kill the whole operating system”. That’s why creating hard to kill, native debugger is quite challenging.

In most debugging cases after building your program on PC, you have to manually (or by scripts) copy the binary somewhere on drive, launch emulator, launch proper debugger – choose one which doesn’t crash or hang under special circumstances, sometimes manually typing in parameters for binary name and it’s parameters, set up breakpoints by looking them up in debugger and typing them in. And each time you have adjustment to a program you have to repeat those steps (if you will not forget in the middle what were you doing).

Things now are little better with Tat / Avena HRDB, which is a frontend for m68k Hatari emulator debugger using custom integration protocol. Still it has flaws of native debuggers, although it doesn’t depend directly on hardware anymore, works for plain mc68000 only and if you are using higher level languages you will not find any source level debugging, no human readable call stacks. Only a wall of m68k assembly with symbols (if you know how to generate them), which isn’t show stopper, but nowadays it’s just not convenient. Even native Pure C in 90’s of last century had source level debugging. I think it was best Atari IDE for writing C programs, if you like old ANSI C syntax. Personally I don’t, because newer C standards came along and are more pleasant to work with. For various reasons we don’t have source level debugging anymore since time when modern gcc compilers jumped in and not everyone wants to use native tools. At last I’ve stopped using them at some point, because modern computers can compile everything much faster, code editors are much better and there is nearly no possibility to lose or corrupt sources by accident, when playing around with hardware and interrupts, which could happen in the past on real hardware. GDB still isn’t functional with TOS binaries, no one was able to change it since a long time.

If you have non-standard hardware, not supported by emulators (for example “frankenfalcon” CT60 with SuperVidel or CTPCI like me or new hardware), things look much more bleak. There is additional problem involved of transferring data to target machine. If you have an hardware like Netusbee, CosmosEx, Ultrasatan, ParCP, Ghostlink, second, PC formatted SD/CF card then you are covered, otherwise the only thing left are good, old floppies or transfer via serial null modem cable. But both options mean some effort on developer side. Most debuggers don’t work or break in various ways on hardware like above, that’s why new releases are not frequent and sometimes aren’t very good. Simple things take more time than they should and time is limited resource. And you are left with printf debugging, which isn’t covering you in all cases like interrupts or DSP related code or dump everything to external terminal via serial port.

All of this could be eliminated with proper tools. Better tools, better automation of tasks equals more iterations of changes, you build and test things more quickly. Here is an example how looks VSCode Amiga toolchain integration by Bartman/Abyss. And here is short seminar about it. Or take look at DeZog Zx Spectrum debugger. As I would like to focus more on writing programs and less on handling everything else I thought that I could make something about it and maybe help with easier VSCode integration on other platforms too.

Using Debug Adapter Protocol (or ‘DAP’ as I will call it later) will eliminate the need for building IDE as long it respects protocol. So, for example building Atari TOS programs could be done entirely from VSCode and then they could be launched directly under emulator or submitted and launched to real hardware via one button. Of course if both ends can understand the protocol, which is the part that I currently work on.

There is also an option of creating native debugger frontend from scratch (like in HRDB), long time ago I knocked up, fairly easily prototype of debugger frontend on top of Vulkan / OpenGL with Dear ImGui and intended to integrate it with gdb stub (which talks with it’s own protocol):

Probably I will return to this idea after finishing DAP.

Second part of task will be DAP integration into existing emulator (Hatari), native debugger or creating proxy for integration with existing Atari native debuggers (most of them can be connected via serial port like ADebug). There are rough plans to incorporate DAP based native debugger into Sqward’s Uip-tools and / or make it standalone similarly to gdb stub.

Integration is trivial now, because DAP library user will have to configure features supported by toggling some flags in debug adapter backend, provide and fill in well defined callbacks for DAP requests, which will translate given machine / emulator internal state to structures understood by protocol library when requested. Translating it to JSON and message sending / receiving is handled by library with use of communication drivers, which are the only hardware / platform dependent part. Currently, I have builds for Windows (amd64), Linux (armv7 32-bit), Atari TOS (m68k) and even Amiga (m68k), but right now I’m focusing on Windows and Atari TOS targets. For each target there are TCP/UDP/IP and serial drivers, adding more is quite easy. There are also other possibilities like adding communication via parallel / LPT port, which should be faster than via serial port, but it will be implemented at later stages.

My client / host prototype is written in C99, the only dependency is cJSON library and libc, it can connect via network with TCP and UDP protocol (Windows / Linux), serial port (Windows, Atari) for other target platforms there are no drivers yet. Client / host are passing DAP initialisation process and client can issue valid, although meaningless DAP requests on which host can reply with DAP compliant responses or events.

So, right now I am focusing on missing DAP parts like filling JSON responses with test content and finishing missing DAP client requests (which actually isn’t required for host or VSCode integration), checking error handling and so on. After that the rest will be writing missing drivers, bugfixing and stability improvements. Probably cJSON will be removed in the future and libc will be removed on Atari / Amiga targets with something more lightweight.

Making of… (read at your own risk)

Much time was spent on setting up the whole project to make it easy to compile on various targets / platforms (AmigaOS / Atari TOS compilers aren’t officialy supported / recognised anywhere), I ended up using CMake (many developers hate it due to syntax, me too) with custom toolchain templates (gcc, clang, msvc) for several operating systems (Windows / Linux / Atari TOS / AmigaOS). Latest msvc versions (VS2019) started to support C99 and it could be easily enabled in CMake (also in recent versions). Additionally on Atari I had to switch from Vincent Riviere gcc 4 to Thorsten Otto’s latest gcc (10.x), because gcc 4.x didn’t have full C99 support. For amiga builds I’ve used amiga-gcc. Some parts were borrowed from other secret projects I’m working on, so I didn’t start completely from scratch.

When I started first thing was figuring out TCP / IP communication on Windows and building client / server application and make them talk to each other.

After that, I had to decide how to handle json, as I wanted to start everything as quickly as possible and decided to use cJSON C89 library. Which has bitten me recently on armv7 32-bit Linux target because parser crashed on unaligned accesses, which will probably bite me again on plain m68000 cpu too (this one cannot access odd addresses). I also setup custom linear temporal allocator and various development support things like logging system with log levels and asserts, which both can be turned on/off depending on build configuration.

DAP specification had to be translated by hand into C equivalent, probably I will get rid this step in the future and automate C headers and structures generation from official specification provided as JSON schema. Which step, by the way, could be provided by official DAP project. I had to create json templates for each DAP event / response / request and embed them in a library. I just parse them from memory and update them instead of creating them from scratch in runtime. This probably will change in the future, because some fields aren’t required and could be not present in character stream. I really underestimated amount of work with this part and if I knew it how everything ends I would automate header generation earlier. One thing I didn’t want to alter was general DAP nomenclature and field order to help my future self with protocol changes.

Next thing was create general api and naming patterns. Main aim was to easy setup client / host, connect them with several function calls and hide library internals from library user. Also expose possibility to hook up custom allocators, design internal machine states for host / client, so they could communicate with each other according to protocol, create interfaces for all needed functions and group them by purpose (json templates, requests / events / responses, DAP message creation, json serialisation/deserialisation etc..) and minimize header dependencies.

Host cannot accept any requests before client sends initialization request first followed by requests for setting various types of breakpoints. After that host can accept all other requests from client and send back results or an error. This ends configuration and host is ready for accepting requests. So, this is how it works in short.

When basic protocol started to work I have created platform independent driver structure, so there is one interface for serial and TCP/UDP communication, but implementation is different depending on platform/cpu architecture. Added TCP IP Linux drivers, Windows and Atari serial drivers and Amiga null drivers. Last developments was separating internal host/client states into structures and removing them from global scope.

And that’s it for now, to the next write up…