Saturday, March 29, 2014

Why Delphi generics are annoying

First of all: I like generics, I love them. You can do all kinds of neat stuff with them... until they bite you. Then they crash your compiler which might make your IDE hang or actually generate wrong source code.

So today I will talk about about the smaller or bigger annoyances of Delphi generics.

They turn your binary into a fat bastard

This might not be a problem at first but you might end up in a situation where your application reaches 36MB just because you are heavily using generic types. And actually duplicated code that is exactly the same for every bit. This by the way is one of the reasons your empty default applications are growing constantly for the recent releases of Delphi: because now generic collections are used all over the place in the runtime. And when I use TList<TButton>, boom, another 10K added to your binary size (approx in XE5). Too bad if you are using more advanced and feature rich collection types like Spring4D. Then every use adds 65K (actually we got down to that number from close to 100K a few months ago).

How could this be solved? Either by the linker fixing this by figuring out identical methods and removing duplicates (the C++ linker has an option for that called COMDAT folding) or the compiler itself could be smart and generate code for equal types only once. Of course he should pay attention to any use of TypeInfo(T) or T.Create for example because that code is actually different. C# does something like this.

Even worse when you think you just use an abstract class (like TEnumerable<T>) in an interface and it suddenly compiles in TList<T> although you never ever created one! Why? Because in the implementation of TEnumerable<T>.ToArray a TList<T> gets created. And because that method is public and virtual (I guess) it won't be smartlinked out.

If you ever tried delegating a generic interface with the implements keyword you either suffered from the bug I mentioned earlier or you realized that this again adds to the binary size because the compiler creates stub methods that makes this possible. I tried to achieve this using interface delegation but that did not result in reducing the binary size to a number I like (actually it was 15K just for interface stubs! Yes the IList<T> interface has quite some methods)

  TObjectList<T: class> = class(TList<TObject>, IList<T>)
    // ...

Interface methods must not have parameterized methods

Also known as compiler error E2535. So you cannot write something neat like this:

  IEnumerable<T> = interface
    function Select<TResult>(
      const selector: TFunc<T, TResult>): IEnumerable<TResult>;

C# has them and hell even Java has them! As far as I know this is because of some implementation detail and the same reason why you cannot have parameterized virtual methods.

No helpers for generic types

How much easier would that make our lives because we actually could extract some code from the generic class to a helper which we might not be used for every instantiation we are using and thus help us with the binary size problem. If we also had helpers for interfaces we could realize something like the method above in a helper because it does not need to be implemented in some class by design (is just needs the iterator). Someone might scream that helpers are not meant to be used by design but I disagree. I actually would like to have them improved so we could use more than one at a time. Even more so now that we have all these helpers for intrinsic types sitting in the SysUtils unit that make it impossible to add own behavior without hiding (or copying, argh) the built-in one. It could also avoid making ugly design decisions to make the underlying array of a TList<T> public to everyone without caring about the fact that it might contain garbage.

The compiler just goes bonkers

If you ever really heavily used generics you might experienced all kinds of funky internal errors where it suddenly stops and the marker points to the line after the last one in a unit or the compile times just goes beyond minutes you know what I am talking about. Even more if you tried to find workarounds to the previously mentioned problems. Like you made a generic record with lots of methods and watch the compiler spinning in circles performing "shlemiel" lookups for the stuff it is compiling. Or it completely fails doing anything and instead running in circles allocating memory until everything is lost.

After I wrote this I noticed another problem. Imagine this class:

  TFoo<T: class> = class
  strict private
    procedure Bar(const item: TObject); overload;
    procedure Bar(const item: T); overload;

What method do you expect to be called when having a TFoo<TButton> and calling Bar passing in a TButton? The second of course. First it matches the argument and second it's the only public method. Do you think anything should be different when you have a TFoo<TObject>? No? Well the compiler does not agree and happily calls the private one.

No support for conditional compiling

This could solve the problem we discussed earlier where I have a generic but want to use specific implementations depending on the type parameter. Currently something like this is not possible:

function TCollections.CreateList: IList<T>;
{$IF TypeInfo(T).Kind = tkInterface}
  IList<IInterface>(Result) := TList<IInterface>.Create;
{$ELSEIF TypeInfo(T).Kind = tkClass}
  IList<TObject>(Result) := TObjectList<TObject>.Create;
  Result := TList<T>.Create;

Finding solutions

The good news for those concerned about binary size (and with Spring4D that supports the mobile platforms being released soon you should or your customers might hate you for apps that are dozens of MB big): Spring4D will support something similar to how C# handles generics. That means with the SUPPORT_GENERIC_FOLDING switch on (it is disabled by default and not supported on Delphi 2010) it will just create TList<TObject> and TList<IInterface> when you are using the TCollections.CreateObjectList<T> and the new TCollections.CreateInterfaceList<T> methods. That means only 2.5k overhead for every list you are creating that holds objects or interfaces (implementing something for pointers is left as an exercise to the reader).

Update 23.04.2015

With the Spring4D 1.2 version the previous described way is not necessary and supported anymore as using TCollections.CreateObjectList<T> or TCollections.CreateInterfaceList<T> automatically creates a very thin (less than 1K!) wrapper class to keep the type info for T but uses a TObjectList<TObject> or TList<IInterface> as its base since all the generated code would be 100% identical. If you are using XE7 and up this works even better because of the new intrinsic functions. With subsequent releases we will introduce this mechanism for more types but lists are usually the majority of the used collection types and thus caused most of the binary bloat.

Sunday, March 23, 2014

Sorry Uncle Bob - but NO!

I just came across this article and was very shocked by the code and so I decided to travel back in time to read the original article by Robert C. Martin and I have to say:

I sincerely have to disagree with you, Uncle Bob!

Let me explain why

While he is trying to separate DI container code from his application logic - which is a good thing but not necessary if you design your architecture without a certain DI container in mind (as Mark Seemann writes in his comment to Martins post) - he does one ugly mistake. He introduces a static factory to his BillingService class.

This thing is nothing else than a service locator call in disguise!

While he shows how nicely you can test the BillingService class he does not show how hard it will be to test the class that contains the logic he commented with "Deep in the bowels of my system." earlier in his post. Because this is where things start to get really messy. In this code he reaches into the BillingService.factory and uses it to create a new BillingService instance which in his real application will call into his DI container and retrieve a fully functional BillingService instance. But how are you supposed to test this? The constructor of DeepInTheBowelsOfMySystem obviously does not tell you anything about BillingService or a factory. This a case when the API lies to you - and in this case really badly. Since you not only call into some dependency that you might pass into your SUT. No you call into some static member and we know that global states are bad, right? This means for a test to succeed we need to setup the BillingService.factory which we will encounter after running into a null pointer exception (or access violation).

This code is as bad as calling into a service locator at this place to get a BillingService instance.

How to solve this problem?

Either pass in a BillingService into your DeepInTheBowelsOfMySystem class or pass in a BillingService factory. Both are things that can be done with "Poor Man's DI" and with most Dependency Injection Containers. Like Spring4D supports factories out of the box. If you register something like this:

myContainer.RegisterType<IBillingService, TBillingService>;

Then the container will automatically resolve something like this:

constructor Create(const billingServiceFactory: TFunc<IBillingService>);

When you see this constructor you immediately know what dependency it expects and it does not take you couple of runs or reading through the code to find out what you need to setup for a test. The SUT should tell you what it needs and nothing more.

As you can see it is very easy to step into traps that lead to hard to test code and it's not always trivial to solve this - so even knowledgeable people like Robert C. Martin might step into these.

Wednesday, March 5, 2014

Spring4D roadmap

Ok, actually less of a roadmap but more of some news regarding the project - but I thought with the recently published Delphi roadmap for this year this might be a good title. ;)

What has been done?

Spring4D has been moved to BitBucket some while ago as most of you know - for those that do not this is a reminder as the project on Google Code will not be continued. If anyone knows a nice way of redirecting from there to the new page let me know. While we did this we also changed from svn to git which made it much easier to work than with svn - except a few times when git was a bit confusing if you never worked with it before. I will share a few things on that in another post.

In the past few months we made some nice changes and cleaned up the source. The collection types have been improved and I am certain we found a good balance between complexity and usability. I hope they will be used by other projects to have one accepted collection libary (apart from the RTL ones) in order to be compatible between these projects (think of an ORM library that uses the same collection types as the MVVM or binding framework for example).

Also Honza RameŇ° and Jeroen Pluimers have been busy on making Spring4D ready for OSX, iOS and Android. These changes are currently still in an extra branch but if you are interested in using the library on these platforms check them out.

The are also some nice additions to the dependency injection container which I will talk about in detail in another post.

We are now down to fixing bugs that we (or you) find - please feel free to report them in the issue tracker. During that time we will also improve source documentation and write guides on how to use Spring4D. Our plan is to finally (the project turns 5 this year, can you imagine) officially release a version 1.0 by the end of march.

What's next?

After the release some work on DSharp will be done removing the redundancies (like the collection types and the multicast events) and it will be then using Spring4D.Base.

Some things that we are planning to add to Spring4D this year are:

- adding more collection types
- improving the RTTI extensions
- adding some utility types (like Weak<T>)
- adding expression types (similar to those from DSharp)
- improving the dependency injection container

While we want to add features Spring4D will still focus on some core features and leave things like GUI or database things for other projects. If you have suggestions or want to contribute in any way feel free to contact us. You are also welcome to discuss or ask things in the Spring4D google group. We are also eager to hear from you if you are using Spring4D in your projects.