Wednesday, October 29, 2014

The real power of generics

I just watched the "Effectively Using Generics in Object Pascal" Code Rage 9 session by Ray Konopka, where he showed a great introduction to generics which is mostly about putting things into lists, dictionaries or other data structures. So basically container types what generics are great for.

When someone asked how you can find out the type of T inside your generic type Ray responded that if you need to do that you may be doing something wrong and why that might be necessary because generic types should be type agnostic.

It might be correct if you only think of collection types when you think about generics. And this is what many people do, hence the discussion I have seen in several places why other pascal languages would need generics because they have powerful array support or other excuses.

But this is limited thinking very much. Generics are way more than a tool to build collection types.
You can do things like the multicast events in Spring4D where of course the generic type needs to find out the exact type info of the T in Event<T> to execute the correct code depending on how you are dealing with a event type or an anonymous method type.
Or think about using generics when using the Spring4D dependency injection container. How could we use TContainer.RegisterType<T> or TContainer.Resolve<T> without finding out the exact type. In fact in that API you mostly don't even pass a variable of that type but just use the generic methods as replacement for something like TContainer.RegisterType(TypeInfo(...)).

Want more examples? Sure!
Think of something like Mock<T> where T is then the type you want to create a mock for. You of course need to find out the type info to pass that into the code that does all the rtti magic beneath (like TVirtualInterface). Think of fluent APIs that return T and provide some typesafe way to use that API (which is what the container registration API and mocks do). Hey, even TComparer<T> from System.Generics.Defaults internally looks up what type the T is to call the correct comparison routine depending on the TTypeKind of the type.

I hope that showed some of the examples where generics give you great power beyond that overused "list of something" example. I am sure you can think of more possibilities!

Oh, by the way the answer to the question asked is: you can use TypeInfo(T) or some of the new intrinsic functions to get information of your generic type parameter.

Thursday, October 9, 2014

Generics and variance

When working with generic collections there is often the question: "Why can't I just assign my TList<TPerson> to a variable of TList<TEntity>? TPerson inherits from TEntity!"

There is something called covariance and contravariance - but the bad news are: Delphi does not support variance. But we are trying to understand what this is all about anyway, shall we?

Let's take a look at our TList<TPerson> again and find out why it is invariant as the Wikipedia article says. Invariant means it's not compatible with either TList<TEntity> nor TList<TCustomer> (TCustomer inherits from TPerson). Let's assume we would hardcast it to TList<TEntity> and continue working with it. It seems correct since every TPerson is a TEntity so a TList<TEntity> is a TList<TPerson> as well, no? No! And that is because data goes out and in. Just iterating over the list and taking the items and printing their name would work. But someone could also try to add a TOrder (it inherits from TEntity) into that list. This will work because at that point we have a TList<TEntity> but at a later point this might horribly crash since some other code still uses the list as TList<TPerson> that suddenly contains a TOrder which clearly does not belong there.

Now let's take a look at a TEnumerable<TPerson> that you might hardcast to a TEnumerable<TEntity>. This will work because you can only retrieve things from TEnumerable<T> but not add any. That is also why C# choses the out keyword for marking a generic type as covariant. And that is also why IReadOnlyList<T> is covariant. You can only retrieve items from it but not add any - even if you could remove items it would be covariant but not if you could add some.

Example: Imagine a box of apples. You can treat it as a box of fruits as long as you just take out some or count them. But you are not allowed to add oranges to it.

Now that we understood what covariance is we might understand easier what contravariance is. As its name implies its the opposite of covariance. And indeed it works the other way around. The Wikipedia articles says the subtyping is reversed. Think of an argument of TProc<TCustomer> that is passed somewhere. So for every TCustomer in a list it does something with that instance and what exactly is done is specified in the anonymous method that is getting passed. The code in that method gets a TCustomer and can access anything a TCustomer has. So now if we had contravariance this would apply here and we could actually pass a TProc<TEntity> because we only want access to the properties of a TEntity which is the base type of TCustomer. Now you see that the C# keyword in makes sense for a contravariant generic type.

If we had a labeler at hand it would be covariant because we can label not only apples but all kinds of fruits.

So how to deal with that kind of dilemma in Delphi?
Well there are several approaches to this. One is carefully(!) using hard-casts making sure that nobody does evil things (like adding oranges into the apple box). When using Spring4D collections you can make use of the ElementType property that every interfaced collection type has to make sure you are not adding wrong things to your list. And you can use the IObjectList interface when dealing with lists that contain class types. This is especially useful when connecting lists to GUI elements because they just have to know about IObjectList. It's almost the same as IList<TObject> but with a different guid and QueryInterface only succeeds for a list that contains objects. But most importantly take the correct type for what you are doing. If you just want to iterate things without manipulating then take IEnumerable<T> or IReadOnlyList<T> instead of IList<T> (see encapsulation).

But there is even more about variance and one is very interesting that not even C# does support - but Java and C++ do: return type covariance.

Imagine a TVehicleFactory with a virtual method CreateNew method that returns a TVehicle. Now you inherit a TCarFactory from it which overrides the CreateNew method. It still has TVehicle as result but it really makes more sense to let it return TCar. If that would work we had return type covariance.

I hope this article shed a bit more light on generics and variance and why certain things don't work the way one might expect.

Friday, October 3, 2014

New language feature in XE7

Nope, not the dynamic array enhancement that everyone talked about already.

In fact I am going to show you something that I haven't seen anyone talking about yet.
It wasn't even mentioned on the official XE7 shows and the reason probably is because it is not been finished completely as in used in generics in the RTL that would benefit from that.

While it is not the desired compiler/linker support for reducing generics code bloat it certainly is something that can help reducing it when used properly.

There are new compiler intrinsic routines called

function IsManagedType(T: TypeIdentifier): Boolean;
function HasWeakRef(T: TypeIdentifier): Boolean;
function GetTypeKind(T: TypeIdentifier): TTypeKind;

These functions when used are evaluated at compiletime and can be used in if and case statements.
Since the expressions of these statements are constant the compiler can eliminate the unused code paths.

Oh, and by the way TTypeKind has been moved to System.pas so no need to include TypInfo for that anymore, yay!

Sounds complicated? Let's see some code:

  TFoo<T> = class
    procedure Bar;

procedure TFoo<T>.Bar;
  case GetTypeKind(T) of
    tkInteger: Writeln('int');
    Writeln('not int');


Ok you say, before XE7 we had to write this:

class procedure TFoo<T>.Bar;
  case PTypeInfo(TypeInfo(T)).Kind of
    tkInteger: Writeln('int');
    Writeln('not int');

So what's the difference? This basically works similar to a compiler directive that we don't have.

When you compile the code using GetTypeKind in XE7 you can see that there is only a blue dot in the line with the Writeln('int') but not in the other one. That indicates only this line can be executed. It would be different when using the other version because that expression is not evaluated to a constant expression at compile time. Hence the compiler cannot remove any part of the case statement.

What does that mean? Well you can write generic types as thin wrappers around code that is written explicitly for the different types without blowing up your binary in the end.

I guess in XE8 we will see that been used in Generics.Collections to reduce the size of the resulting binary. However it cannot remove all duplicated binary code since regardless how thin the generic class is it gets compiled for every instantiation (any different type put in for the T) even if the code is exactly the same. So we will see what other ideas they come up with to solve that problem.

I am also investigating this further to find out if that might be any useful in Spring4D.

Oh, btw compiler guys - while you are adding compiler magic, can you please add the NameOf function aswell? Especially when dealing with databinding and things that are currently referenced as magic strings in the source code this could prevent breaking code by typos or refactorings. Thanks!

Saturday, September 13, 2014

Spring4D 1.1 release and roadmap

Today I am happy to officially announce the release of version 1.1. And also it is the 5th anniversary of Spring4D (at least according to the commit history).

Thanks Paul who started the project and all the others that made Spring4D what it is today:
A modern RTL extension and the most powerful dependency injection container written in Delphi that can compete with those in other languages easily.

For the 1.1 release we have fixed many bugs, refactored and cleaned up the internals of the container and implemented many neat new things. And of course added support for Delphi XE7.

After 5 years and finally getting past version 1.0 we felt it was time to get some official logo for Spring4D to be recognizable and here it is - thanks to Chris Wagner for his help!

New feature - automatic factories

There is one particular feature I am very proud of - and thanks to Philipp Schäfer who had brought me to this idea: automatic factories.

When you are dealing with dependency injection you often have the case that you cannot create everything upfront. Mostly because some information is missing that the user enters during the use of the application. That is when factories and lazy initialization come into play. You obviously created factory classes in the past and registered them into the container so they could be injected into other classes that used them.

You don't have to do that anymore because the container now can create these factories for you!

Let's look at some example:

  IMyService = interface
    procedure DoSomething(const msg: string);

  IDialogService = interface
    procedure ShowDialog;

  TMyService = class(TInterfacedObject, IMyService)
    fAnotherServiceFactory: IFactory<string, IDialogService>;
    constructor Create(const anotherServiceFactory: IFactory<string, IDialogService>);
    procedure DoSomething(const msg: string);

  TDialogService = class(TInterfacedObject, IDialogService)
    fMsg: string;
    constructor Create(const msg: string);
    procedure ShowDialog;

constructor TMyService.Create(
  const anotherServiceFactory: IFactory<string, IDialogService>);
  fAnotherServiceFactory := anotherServiceFactory;

procedure TMyService.DoSomething(const msg: string);

constructor TDialogService.Create(const msg: string);
  fMsg := msg;

procedure TDialogService.ShowDialog;

Some pretty straightforward code with only one thing that might look unfamilar: IFactory is a new type that can be resolved by the container automatically and comes from the unit Spring.Container.Common and is similar to TFunc but with $M+ which is required for the container to automatically resolve this type using TVirtualInterface.

So how does the registration code look like?

  container.RegisterType<IFactory<string, IDialogService>>.AsFactory;

  myService := container.Resolve<IMyService>;
  myService.DoSomething('Spring4D rocks!');

The new method AsFactory tells the container to treat this interface or delegate type (if it has method typeinfo on) as factory and it automatically creates the implementation. Any arguments passed are then passed to the container as arguments for the resolution process. That means in this example the string getting passed to the DoSomething method and into the factory is then passed to the container to be injected to the constructor of TDialogService.

What's coming next?

There are several things we want to do in the next couple of months.
  • As you might have seen we have something new in development that is known to some as marshmallow: an ORM that Linas Naginionis contributed to Spring4D. He and Cesar Romero are new on the team and are currently working on the migration and integration into the library.
  • Honza has been working on a better logging solution that will be used internally by the container and can be used in your applications and will have several backends (for CodeSite, SmartInspect or for simple text or console logs). It also can be easily extended to your needs.
  • I have been working on an easy to use interception (AOP) library that can be integrated into the container or used as standalone to do some nice things - those of you attending the EKON 18 in November in cologne will probably see more of it shown by Nick and me.
Along with these new features we also want to write more documentation and how-to articles introducing you to all the features in Spring4D and show advanced dependency injection techniques.

We are eager for your feedback!

As always your feedback is welcome. Are you using Spring4D in your software? Please let us know! You have ideas how to improve the library or want to contribute? Don't hesitate to contact us.

Friday, May 23, 2014

Software developers and the generation gap

Today I have a rather unusual topic because it's not that technical as usual. But something that has bothered me for a while.

I want to talk about different kinds of developers - but as the topic might imply it is less about the actual age but more about the mindset of software developers.

On the one side there are people that are open minded to new ideas and experiencing new technologies. I don't mean those hipsters that follow every new trend just to find that antiquated the next week. I am talking about those that are not stuck in the 80s or 90s when it comes to how software development is done in 2014. Those that care about their software because they don't want to pull their hair out the next time they have to implement a new feature or find and fix a bug. They also care about their coworkers because usually they are part of development team and like to share opinions and get feedback.

On the other hand I sometimes see people that have the attitude of a caveman when it comes to learning new things and breaking out of their habits. They are still coding like in TP 5.5 or Delphi 3 (I randomly picked these versions and my deepest sympathy if you are stuck with an old version for some reasons). If someone mentions things like clean code, principles or even design patterns (unless its the singleton pattern, that one is awesome! /sarcasm off) they run away or start an argument on how this is over-engineering stuff. But when it comes to unnecessary (i.e. premature) performance optimizations to squeeze out the last CPU cycle by using unreadable code they are unstoppable.

OK, I hear some of you screaming already - because I am exaggerating here. Of course you need to care about performance and optimizations. But in the right places. If your server uses gigabytes of memory and all cores just because you did some sloppy programming but optimize a routine that uses like 0.0001% of your CPU then probably did something wrong.
Also some people tend to solve every problem by using patterns and in the end their fizzbuzz looks like this. Principles and patterns should be used to write code that is testable and maintainable and not just because.

Also when it comes to new language features especially when they remind you of C# or Java some people seem to have an allergy to these things. If anything does not look pascalish it's probably something evil from a "low level" language like C++ (did someone say '98? No its '14!).

Oh and because it always worked well some of course still put all their code into the event handlers of forms and frames that not only contain hundreds of controls that are made visible when necessary. Data is stored in fixed length format files of course because that's super fast and databases are just overhead. They also don't need unit tests because their code works (which only they understand of course)!

That was a lot of ranting but the question that follows is: What can you learn from each other?

Be open minded. Question yourself if you are doing things the way you are doing them just because you always did. Try out suggestions you read or hear. Don't treat everything as a nail just because you only have a hammer. Use the tool that is the best for the task. Develop software that you still like to maintain and improve two months from now. Or make the one you hate working on less painful.
Learn from the seniors. They had their reasons to code in particular ways in the past because times were different (that sounds like a long time but a few years can be like eternity in IT). Don't just use algorithms and data structures but also try understand them. It will help choosing the right one for the task.

As always I am interested in your opinions - especially since this time I picked a very controversial topic.

Saturday, March 29, 2014

Why Delphi generics are annoying

First of all: I like generics, I love them. You can do all kinds of neat stuff with them... until they bite you. Then they crash your compiler which might make your IDE hang or actually generate wrong source code.

So today I will talk about about the smaller or bigger annoyances of Delphi generics.

They turn your binary into a fat bastard

This might not be a problem at first but you might end up in a situation where your application reaches 36MB just because you are heavily using generic types. And actually duplicated code that is exactly the same for every bit. This by the way is one of the reasons your empty default applications are growing constantly for the recent releases of Delphi: because now generic collections are used all over the place in the runtime. And when I use TList<TButton>, boom, another 10K added to your binary size (approx in XE5). Too bad if you are using more advanced and feature rich collection types like Spring4D. Then every use adds 65K (actually we got down to that number from close to 100K a few months ago).

How could this be solved? Either by the linker fixing this by figuring out identical methods and removing duplicates (the C++ linker has an option for that called COMDAT folding) or the compiler itself could be smart and generate code for equal types only once. Of course he should pay attention to any use of TypeInfo(T) or T.Create for example because that code is actually different. C# does something like this.

Even worse when you think you just use an abstract class (like TEnumerable<T>) in an interface and it suddenly compiles in TList<T> although you never ever created one! Why? Because in the implementation of TEnumerable<T>.ToArray a TList<T> gets created. And because that method is public and virtual (I guess) it won't be smartlinked out.

If you ever tried delegating a generic interface with the implements keyword you either suffered from the bug I mentioned earlier or you realized that this again adds to the binary size because the compiler creates stub methods that makes this possible. I tried to achieve this using interface delegation but that did not result in reducing the binary size to a number I like (actually it was 15K just for interface stubs! Yes the IList<T> interface has quite some methods)

  TObjectList<T: class> = class(TList<TObject>, IList<T>)
    // ...

Interface methods must not have parameterized methods

Also known as compiler error E2535. So you cannot write something neat like this:

  IEnumerable<T> = interface
    function Select<TResult>(
      const selector: TFunc<T, TResult>): IEnumerable<TResult>;

C# has them and hell even Java has them! As far as I know this is because of some implementation detail and the same reason why you cannot have parameterized virtual methods.

No helpers for generic types

How much easier would that make our lives because we actually could extract some code from the generic class to a helper which we might not be used for every instantiation we are using and thus help us with the binary size problem. If we also had helpers for interfaces we could realize something like the method above in a helper because it does not need to be implemented in some class by design (is just needs the iterator). Someone might scream that helpers are not meant to be used by design but I disagree. I actually would like to have them improved so we could use more than one at a time. Even more so now that we have all these helpers for intrinsic types sitting in the SysUtils unit that make it impossible to add own behavior without hiding (or copying, argh) the built-in one. It could also avoid making ugly design decisions to make the underlying array of a TList<T> public to everyone without caring about the fact that it might contain garbage.

The compiler just goes bonkers

If you ever really heavily used generics you might experienced all kinds of funky internal errors where it suddenly stops and the marker points to the line after the last one in a unit or the compile times just goes beyond minutes you know what I am talking about. Even more if you tried to find workarounds to the previously mentioned problems. Like you made a generic record with lots of methods and watch the compiler spinning in circles performing "shlemiel" lookups for the stuff it is compiling. Or it completely fails doing anything and instead running in circles allocating memory until everything is lost.

After I wrote this I noticed another problem. Imagine this class:

  TFoo<T: class> = class
  strict private
    procedure Bar(const item: TObject); overload;
    procedure Bar(const item: T); overload;

What method do you expect to be called when having a TFoo<TButton> and calling Bar passing in a TButton? The second of course. First it matches the argument and second it's the only public method. Do you think anything should be different when you have a TFoo<TObject>? No? Well the compiler does not agree and happily calls the private one.

No support for conditional compiling

This could solve the problem we discussed earlier where I have a generic but want to use specific implementations depending on the type parameter. Currently something like this is not possible:

function TCollections.CreateList: IList<T>;
{$IF TypeInfo(T).Kind = tkInterface}
  IList<IInterface>(Result) := TList<IInterface>.Create;
{$ELSEIF TypeInfo(T).Kind = tkClass}
  IList<TObject>(Result) := TObjectList<TObject>.Create;
  Result := TList<T>.Create;

Finding solutions

The good news for those concerned about binary size (and with Spring4D that supports the mobile platforms being released soon you should or your customers might hate you for apps that are dozens of MB big): Spring4D will support something similar to how C# handles generics. That means with the SUPPORT_GENERIC_FOLDING switch on (it is disabled by default and not supported on Delphi 2010) it will just create TList<TObject> and TList<IInterface> when you are using the TCollections.CreateObjectList<T> and the new TCollections.CreateInterfaceList<T> methods. That means only 2.5k overhead for every list you are creating that holds objects or interfaces (implementing something for pointers is left as an exercise to the reader).

Update 23.04.2015

With the Spring4D 1.2 version the previous described way is not necessary and supported anymore as using TCollections.CreateObjectList<T> or TCollections.CreateInterfaceList<T> automatically creates a very thin (less than 1K!) wrapper class to keep the type info for T but uses a TObjectList<TObject> or TList<IInterface> as its base since all the generated code would be 100% identical. If you are using XE7 and up this works even better because of the new intrinsic functions. With subsequent releases we will introduce this mechanism for more types but lists are usually the majority of the used collection types and thus caused most of the binary bloat.

Sunday, March 23, 2014

Sorry Uncle Bob - but NO!

I just came across this article and was very shocked by the code and so I decided to travel back in time to read the original article by Robert C. Martin and I have to say:

I sincerely have to disagree with you, Uncle Bob!

Let me explain why

While he is trying to separate DI container code from his application logic - which is a good thing but not necessary if you design your architecture without a certain DI container in mind (as Mark Seemann writes in his comment to Martins post) - he does one ugly mistake. He introduces a static factory to his BillingService class.

This thing is nothing else than a service locator call in disguise!

While he shows how nicely you can test the BillingService class he does not show how hard it will be to test the class that contains the logic he commented with "Deep in the bowels of my system." earlier in his post. Because this is where things start to get really messy. In this code he reaches into the BillingService.factory and uses it to create a new BillingService instance which in his real application will call into his DI container and retrieve a fully functional BillingService instance. But how are you supposed to test this? The constructor of DeepInTheBowelsOfMySystem obviously does not tell you anything about BillingService or a factory. This a case when the API lies to you - and in this case really badly. Since you not only call into some dependency that you might pass into your SUT. No you call into some static member and we know that global states are bad, right? This means for a test to succeed we need to setup the BillingService.factory which we will encounter after running into a null pointer exception (or access violation).

This code is as bad as calling into a service locator at this place to get a BillingService instance.

How to solve this problem?

Either pass in a BillingService into your DeepInTheBowelsOfMySystem class or pass in a BillingService factory. Both are things that can be done with "Poor Man's DI" and with most Dependency Injection Containers. Like Spring4D supports factories out of the box. If you register something like this:

myContainer.RegisterType<IBillingService, TBillingService>;

Then the container will automatically resolve something like this:

constructor Create(const billingServiceFactory: TFunc<IBillingService>);

When you see this constructor you immediately know what dependency it expects and it does not take you couple of runs or reading through the code to find out what you need to setup for a test. The SUT should tell you what it needs and nothing more.

As you can see it is very easy to step into traps that lead to hard to test code and it's not always trivial to solve this - so even knowledgeable people like Robert C. Martin might step into these.

Wednesday, March 5, 2014

Spring4D roadmap

Ok, actually less of a roadmap but more of some news regarding the project - but I thought with the recently published Delphi roadmap for this year this might be a good title. ;)

What has been done?

Spring4D has been moved to BitBucket some while ago as most of you know - for those that do not this is a reminder as the project on Google Code will not be continued. If anyone knows a nice way of redirecting from there to the new page let me know. While we did this we also changed from svn to git which made it much easier to work than with svn - except a few times when git was a bit confusing if you never worked with it before. I will share a few things on that in another post.

In the past few months we made some nice changes and cleaned up the source. The collection types have been improved and I am certain we found a good balance between complexity and usability. I hope they will be used by other projects to have one accepted collection libary (apart from the RTL ones) in order to be compatible between these projects (think of an ORM library that uses the same collection types as the MVVM or binding framework for example).

Also Honza Rameš and Jeroen Pluimers have been busy on making Spring4D ready for OSX, iOS and Android. These changes are currently still in an extra branch but if you are interested in using the library on these platforms check them out.

The are also some nice additions to the dependency injection container which I will talk about in detail in another post.

We are now down to fixing bugs that we (or you) find - please feel free to report them in the issue tracker. During that time we will also improve source documentation and write guides on how to use Spring4D. Our plan is to finally (the project turns 5 this year, can you imagine) officially release a version 1.0 by the end of march.

What's next?

After the release some work on DSharp will be done removing the redundancies (like the collection types and the multicast events) and it will be then using Spring4D.Base.

Some things that we are planning to add to Spring4D this year are:

- adding more collection types
- improving the RTTI extensions
- adding some utility types (like Weak<T>)
- adding expression types (similar to those from DSharp)
- improving the dependency injection container

While we want to add features Spring4D will still focus on some core features and leave things like GUI or database things for other projects. If you have suggestions or want to contribute in any way feel free to contact us. You are also welcome to discuss or ask things in the Spring4D google group. We are also eager to hear from you if you are using Spring4D in your projects.

Monday, February 24, 2014

Packages and initialization

Packages are an approved concept and they have been working fine for over a decade now, right? Well yeah, they should be. However my coworkers and me have been bitten by them not only once. Especially since our software consists of lots of DLLs that are linked against the RTL/VCL, 3rd party and some of our own packages.

Everything alright so far you say? Actually not because of some simple fact: initialization parts of units inside of packages might not be executed when the package gets loaded but when a binary gets loaded that uses it. Not a problem? Not really unless your initialization and finalization parts are doing the correct thing.

Let's take a look at some unit that might be part of a package:

unit Foo;


procedure Bar;



  list: TObjectList;

procedure Bar;
  if not Assigned(list) then
    list := TObjectList.Create;
  // do something with list




What do you think happens when you load a DLL that uses this unit given that no other module has used this unit yet? Yep, the initialization part gets executed. Since we are using lazy initialization here there is nothing to execute. Now we call Bar and the list variable gets initialized since as we know global variables are initialized with zero (or in our case nil). Then we unload the DLL and our finalization part gets called and cleans up the instance. What do you think happens when we load the DLL again and call Bar? Most likely an AV. Wait what? Yes, list has not been set to nil because remember this unit is part of a package and thus list has not been set to nil again!

There are several places in Delphi and 3rd party source I have seen that are coded like this. Our solution? Make sure every unit that causes trouble is referenced in the main application EXE to force initialization at startup and finalization at shutdown of the application.

P.S. Did anyone shout "Never use FreeAndNil" back there?!