The long-awaited "better OSDK" is now available, and it comes with so many changes that instead of polluting the change history, I decided to dedicate an article for it.

Please fasten your seatbelt because it may be a long journey!

OSDK 2.0

First I need to address the version number:

Why 2.0 instead of 1.24?

There are a few main reasons:

• There are many, many changes, so it's worth a proper update
  
• The Visual Studio Code extension needs these changes, so a proper version bump makes it easier to track the problems
  
• This version is not 100% backward compatible, so by having two separate branches we can still update the 1.x without conflicts
  

How to setup your project

The previous OSDK basically had -O1 and -O2 options for the compiler, with -O1 being horrible code and -O2 being better code... and really there was no reason to use -O1 since you could not debug the result anyway.

The new OSDK has a bunch of knobs you can turn to provide different levels of performance:

There are three of them, and they are independent.

OSDKCOMP is the code generator level, OSDKMACRO=-O turns on the 6502 peephole optimizer that runs over the expanded output, and OSDKDEBUG=-g1 asks for the information a debugger needs.

Sizes below are real final.out bytes for two of the benchmark samples:

OSDKCOMP	OSDKMACRO	OSDKDEBUG	aes256	sieve	C debugging	Use this when
-O3	-O	unset	8 648	1 135	no	Shipping. Smallest and fastest
-Os	-O	unset	8 648	1 135	no	Shipping, when size matters more than speed
-O2	-O	-g1	9 177	1 364	yes	Normal development. Step through C, see typed variables
-O1	-O	-g1	11 608	1 530	yes, plus locals	Inspecting variables the higher levels hide
-O1	unset	unset	20 948	1 607	no	You suspect the toolchain, not your code

A few things the table shows:

• The peephole optimizer matters as much as the compiler level. Leaving OSDKMACRO=-O off costs 24% on aes256 at -O3 (8 648 becomes 11 320), and 45% at -O1. "I build at -O3" is ambiguous unless you also say whether the peephole is on.
  
• -g1 produced byte identical output on both samples. It only costs anything in a function holding a local that the optimizer would have removed, since keeping the variable is what makes it inspectable. That is why it is not the default.
  
• -Os produced the same output as -O3 on these two samples. It only diverges where a transform trades size for speed, such as unrolling a shift.
  
• -O1 with no peephole is a diagnostic setting. It is there to find out whether a bug is yours or ours, not something to ship.
  

Rather than dropping the whole project to a lower level just to inspect one function, use the pragma:

#pragma optimize(push, 1)
void the_function_I_am_debugging(void)
{
    ...            /* locals stay on the stack frame, visible to the debugger */
}
#pragma optimize(pop)   /* everything after this is back to the command line level */

That way the one function you are looking at stays readable while the rest of the program stays fast. On an Oric that matters: dropping everything to -O1 can stop a program fitting in memory.

What was wrong with OSDK 1.0?

The OSDK 1.0 inherited a long list of brittle changes added over more than two decades of work. As a result it was full of weird things, and some trivial operations required some convoluted patterns.

So here is a short list of the known issues:

• The C compiler generated very inefficient code, had non standard library includes, no support for 32 bit (long) variables, no support for C source code level debugging
  
• The linker had no granularity level to import library functions, so importing one function would drag the entire module even if you did not need the rest
  
• The assembler had quite a few issues like not always reporting errors in the code, no support for auto-chaining sections, limited support for advanced symbols
  
• The floating point support was incomplete and somewhat buggy in some cases
  
• The libraries were missing quite a few standard functions
  
• The debugger (in the emulator) was somewhat difficult to use and had a few annoying problems like not recognizing the local keyboard of the user and not accepting inputs on the num keypad
  

So quite a few of these issues have been addressed, there are obviously still some problems but I think we are now in quite a better position than we were in the previous version.

Backward compatibility

We¹ tried to make it as much as possible compatible with previous projects, and if you are just doing a simple assembler or C program, it will probably work as before.

But if you are a power user with your own build system, fancy macros, calling library functions manually, etc... it's quite possible that your program will not work as before, so here is a list of known traps:

• New command line parameters on all the tools. If your build system calls compiler.exe, xa.exe or link65.exe directly instead of going through make.bat, check the new options before assuming the previous ones still mean the same thing.
  
• Library function parameters optimization. A number of library functions now receive their arguments in registers instead of on the stack (see __fastcall below). If you call them from assembler, you must pass the arguments the new way.
  
• Linker ordering. The on-demand library modules are no longer placed where they used to be. If your project captures an "end of my code" label and relies on it to place something else, that label may no longer be where you think it is.
  

The linker one deserves a warning, because it is silent: the program builds perfectly and then misbehaves at run time. If you have a multi-part program that loads modules on top of each other, double check where your "end of code" marker actually lands now².

The new automatic section chaining described below is the clean way to avoid the whole problem.

If you want to safely test the system, you could simply unzip the new OSDK somewhere else (say C:\OSDK2) and modify your osdk_config.bat by adding a SET OSDK=C:\OSDK2 at the top so your local project will use the new OSDK.

If that works, you should see [OSDK 2.0] printed in the build log.

Performance numbers

Our friend ISS had maintained a performance benchmark comparing a few C compilers for the 6502, and the OSDK compiler results were not particularly impressive.

It did not perform too badly in terms of compatibility compared to some compilers like kickc and sdcc, but regarding executable sizes and execution speed it got slaughtered by VBCC and cc65 was also consistently better.

So here is an updated table that shows the difference between OSDK 1.23 and OSDK 2.0 on the same benchmark:

• Size is the raw final.out in bytes (without any header or instrumentation code)
  
• Speed is the number of 6502 cycles reported by Oricutron
  
• Optimisation levels are -O2 for OSDK 1.23³ and -O3 +peephole optimizer for OSDK 2.0.
  

Please note that some tweaks were required to get the test to build with OSDK 1.23:

• Some of the tests have a hyphen ("-") in their names, which made version 1.23 fail (fixed in 2.0)
  
• 1.23 was missing some include files and function names which had to be added
  

Benchmark	Size (bytes)			Speed (cycles)			Notes
	1.23	2.0		1.23	2.0
type-sizes	n/a	787	–	n/a	10 858	–	1
0xcafe	n/a	537	–	n/a	1 810	–	1
dummy	452	25	−94%	52	42	−19%
hello-world	615	88	−86%	404	394	−2%
bytecpy	462	31	−93%	66	50	−24%
memcopy	600	126	−79%	1 549 488	783 317	−49%
sieve	10 083	1 135	−89%	18 780 756	16 871 069	−10%	3
aes256	14 548	8 648	−41%	102 885 908	28 524 921	−72%
mandelbrot	1 189	1 588	+34%	195 600 122	26 840 545	−86%	2
frogmove	3 788	2 916	−23%	31 799 089	8 867 317	−72%
pi	2 993	1 998	−33%	45 947 678	169 786 764	+270%	2
shuffle	2 601	2 091	−20%	2 896 139	4 044 094	+40%	2
bubble-sort	2 399	1 578	−34%	17 838 418	12 119 030	−32%
selection-sort	2 461	1 624	−34%	11 587 350	6 667 583	−42%
insertion-sort	2 365	1 559	−34%	7 070 736	5 182 851	−27%
merge-sort	4 608	2 215	−52%	4 157 050	3 098 152	−25%
quick-sort	2 855	1 891	−34%	3 309 021	2 718 015	−18%
counting-sort	4 075	1 896	−53%	1 927 620	1 642 217	−15%
radix-sort	4 997	2 531	−49%	8 891 332	8 235 935	−7%
shell-sort	2 510	1 636	−35%	3 728 906	2 842 785	−24%
heap-sort	3 082	2 020	−34%	5 728 855	4 794 312	−16%
eight-queens	2 857	1 970	−31%	88 410 815	74 126 245	−16%
qrcode	38 264	22 862	−40%	146 634 751	129 636 205	−12%
TOTAL (comparable only)	107 804	60 428	−44%

Notes:

1. These two tests use features that 1.23 does not support: _Static_assert and 0b binary literal.
   
2. These three tests use the long type which on 1.23 was actually 16 bit, so getting correct results now requires real 32 bit arithmetic, which explains the performance hit.
   
3. The size difference was caused by an 8KB static buffer allocated in the .text section instead of .bss

The size reduction comes from multiple elements:

• The default "header.s" and "tail.s" contained things that should not have been there, which is why "dummy" was huge
  
• There was no fine-grained inclusion of library functions, so just using "getchar" would add the entire conio family, including putchar, puts, etc...
  
• The .bss section was not actually used for any of the internal allocations
  
• There was no 8 bit support, the code generator only supported 16 bit operations
  
• Every function call pushed parameters on the stack
  
• Every function had an "enter" and "leave" section even if they had no parameters and no locals
  
• No code optimizer to remove redundant loads and stores, etc...
  

Compiler

This is where most of the work went. The headline number: a "hello world" program that took 615 bytes on OSDK 1.23 now builds to 88 bytes, which happens to be smaller than what vbcc (99), llvm-mos (114) and cc65 (197) produce for the same source.

__fastcall - passing parameters in registers

The original compiler only supported parameters pushed on the stack, which on 6502 is very expensive.

A function can now be declared __fastcall, and its arguments arrive in registers instead:

__fastcall void _putc(char c);           /* char arrives in A */
extern __fastcall int isalpha(char c);   /* same idea, whole ctype family */

The compiler picks the register width from the prototype, so you get A for a byte and A:X for a pointer or an int. Most of the small, hot library functions (the ctype family, _putc, _puts, strlen, atoi, exit) have been converted, which is a large part of where the size win comes from.

32 bit long support

The original OSDK 1.x compiler did not have 32 bit support, long was actually 16 bits, so any program relying on 32-bit intermediates would do incorrect computations.

An example of that is the pi computation printing 0.0018094... instead of 3.14159..., because the 10000*2000 intermediate wrapped around.

Native 8 bit arithmetic

The previous compiler promoted almost everything to 16 bits, so a simple char addition turned into a two-byte operation with the high byte thrown away. The compiler now keeps char work in 8 bits for + - & | ^ ~, comparisons and shifts, uses inc/dec for in-place increments, and drops the redundant cmp #0 when comparing a byte against zero.

Choosing between size and speed

There is now an -Os flag for "optimize for size", next to the usual -O1/-O2/-O3. And because the right answer is rarely the same for a whole project, you can change it per function:

#pragma optimize(push, 1)
void MyDebuggableFunction(void)    /* built at -O1, locals stay on the frame */
{
    ...
}
#pragma optimize(pop)              /* back to whatever the command line said */

This is useful for debugging: keep the function you are stepping through at -O1 so its locals remain addressable, and let the rest of the project build at -O2 or -O3.

Dead local elimination

Local variables that are only ever written to and never read are now removed, and if that empties the frame, the whole stack frame setup ("enter" and "leave") goes with it.

Debug information

Under -g1 the compiler emits .csource directives mapping generated code back to C source lines, and .ctype annotations describing your variables and types (including pointers, arrays and structs).

That is what allows the VS Code extension to show you typed variables and to step through C source rather than disassembly.

Preprocessor and macro expansion

Between the compiler and the assembler sit two steps, and both were reworked.

A new C preprocessor

The bundled preprocessor is now mcpp 2.7.2 instead of the previous cpp.exe.

It is C99 conforming, it reports problems as file:line so your editor can jump straight to them, and it treats a missing include file as a hard error instead of carrying on and letting the compiler produce a cascade of unrelated errors later.

Its output was verified byte for byte against the previous cpp.exe across the whole test suite plus about 175 cc65 test pairs before it became the default.

The previous preprocessor is still shipped, so if you hit a difference you can go back to it:

SET OSDKCPP=%OSDK%\BIN\cpp.exe

MacroSplitter now expands the macros

The compiler does not emit 6502 instructions directly: it emits calls to the macros in MACROS.H, which then have to be expanded.

That expansion used to be a second cpp.exe pass; MacroSplitter now does it itself, with parenthesis-balanced argument parsing and nested expansion up to 16 levels. It also writes a comment above each expanded block showing the original macro call, which makes the generated assembly far easier to read when you are chasing a codegen bug.

If you need the previous behaviour:

SET OSDKMACROEXPAND=0

The 6502 peephole optimizer

MacroSplitter also carries a peephole optimizer, enabled with:

SET OSDKMACRO=-O

It detects and cleans up the things a macro-based code generator cannot avoid emitting:

• a store immediately followed by a reload of the same location
  
• a load whose value is already in another register (turned into txa and friends)
  
• redundant repeated stores
  
• a reload of an immediate value that is already loaded
  
• a jsr followed by rts that can be collapsed into a single jmp

It also resolves #<(N) and #>(N) constant expressions that would otherwise reach the assembler unevaluated.

On char-heavy code this pass alone is worth up to 24% of the code size.

Library

There have been quite a few changes made to the libraries:

• Many functions now have @function and @endfunction annotations to help the linker strip out unused code safely
  
• The dynamic allocator was repaired: malloc, free and realloc now actually work
  
• A pile of string functions were fixed or added: strchr, strstr, strcspn, strspn, strrchr, strtok, memccpy
• sprintf no longer forgets its NUL terminator
  
• The floating point support was fixed, including a runtime int-to-float conversion that called the wrong ROM entry
  
• Common portable C headers (stdint.h and friends) now work out of the box, so third-party C code has a much better chance of compiling unmodified
  
• The C runtime startup was slimmed down by nearly 400 bytes

Linker

Only link what you use

The linker previously worked at file granularity: referencing one function pulled in the whole module it lived in.

Library modules can now be annotated so the linker can strip the routines you never call:

; @function _putchar
_putchar
    ...
    rts
; @endfunction

There is also a @keep annotation for the routines that must survive even when nothing appears to reference them (interrupt handlers and the like).

On hello-world this roughly halved the program on its own, before __fastcall brought it down to the final 88 bytes.

Overriding library functions

The -d option can now be repeated, and the directories are searched in command line order, first match wins:

link65 -d my-funcs/ -d osdk-lib/ main.s

That means you can provide your own memcpy (or whatever) without modifying the OSDK library.

Library resolution is also deferred until every command line file has been parsed, so a symbol referenced early but defined by one of your own later files no longer drags in the library version and collides with it.

Assembler

Automatic section chaining

This is the change most likely to simplify your existing sources.

Previously, if you wanted your .data or .bss section to sit immediately after your code, you had to capture the end of the .text section in a label and then manually force the BSS to start there:

    .text
_EndText                    ; capture where the code ends

    .bss
*= _EndText                 ; ...and force the BSS to start there
my_buffer .dsb 256

Now .data automatically follows .text, and .bss automatically follows .data, so the same thing is simply:

    .bss
my_buffer .dsb 256

The assembler also publishes the section boundaries as labels you can use in your code: __text_start, __text_end, __text_size, and the same for __data_, __bss_ and __zero_.

Chaining only moves labels that sit at the natural section position.

The moment you pin an address yourself with a *= directive, that label and everything after it in the section keeps the address you gave it, so explicit screen, overlay and hardware placements are left alone.

Better error reporting

The previous assembler stayed silent in cases where it should have complained loudly:

• An unbalanced #if / #ifdef / #endif is now a fatal error pointing at the exact directive. A missing #endif used to silently swallow everything after it
  
• A section whose content would run past the end of the 64K address space is now an error instead of quietly wrapping around
  
• Assertion failures also report better. The message now sits on the same line as the file, the line number and the address, so your editor can parse it and take you straight there:
  

  loader.s(412):c000: Assertion failed: Vector address is incorrect, loader will crash
  

VS Code Extension

The extension is not part of the OSDK, but it needs specifically the OSDK 2.0 to be installed, because it relies on the new debug information the compiler and assembler emit.

With the two together you get C source level debugging: real breakpoints in your .c files, a call stack that walks from your C code down through the assembly and across interrupt boundaries, and typed variable inspection.

Debugging Encounter in VS Code

Since the extension has quite a few features, it will have its own dedicated article.

Wrap up

And that's it for this version 2.0.

I'm quite certain that bugs will be found and will have to be fixed.

I'm also quite certain there are more optimizations we can add down the line, but LCC was never meant to be a full optimizing compiler so it is missing some internal support for things like detecting loop invariants, or optimizing for loops, so that would require some significant changes to the code generator.

Anyway, I think this was a worthy upgrade, and if it does not work for your project, just keep using version 1.x but please provide some repro-case projects and explanations about the problems you encountered so they can be fixed.

Also of interest is anything that shows extremely bad results compared to other C compilers, that can often be caused by some use pattern that was not detected in previous test suites and is sometimes easy to fix.

Have fun!

---