Dependency Injection best practices - lazy initialization

“You gotta know when to be lazy. Done correctly, it's an art form that benefits everyone.” Nicholas Sparks, The Choice

I admit this was taken out of context a bit - but we will see how this applies to dependency injection.

Identifying the problem

When dealing with dependency injection - and when I am saying dependency injection I am referring to the software design pattern - you are facing one problem pretty quickly. The first instance you create (or resolve using the container at the very root of your application) needs everything injected that you will ever need, either directly or something you inject takes that. This will result in tons of wire up code - and this is one reason why using a DI container might be a good idea for an application that you design with this pattern in mind - however a DI container might save you from writing tons of boilerplate code but it still does not solve the problem of instantiating the whole object graph at the start of the application on its own.

There are several ways on how not to do dependency injection - please read the linked article first as it will explain some important things that I felt would be redundant to repeat because they are very well explained (and the few C# code snippets are easy to understand).

...

Things to avoid

You are back? Good.

Let's talk about the service locator and the GlobalContainer - both things you might have seen in several Spring4D examples. If you are writing an application with DI in mind - don't use the service locator.

The service locator might be a nice way of introducing the DI container to an already existing application because you can plug it in at some place deep within the existing code and then kick off the container from there. But don't (ab)use it just to replace constructor calls with something else or to get around actually doing dependency injection. Using the service locator is not DI!

Another important thing to know about the use of the DI container - it does not serve to produce value objects - objects that only hold some values (think of TCustomer or TOrder) but for service objects - objects that actually do some work (like TCustomerValidator or TOrderProcessor). With that in mind you understand the possible problem of "I need to pass some data that is only known later in the application to the container" actually is not a problem. If you need to create customer or order objects then you either register a factory to the container that takes the necessary arguments and constructs an instance or you create that object in your code. No, this is not evil and there is no tight coupling you need to remove - if you follow some guidelines that Misko Hevery explains in one of his articles.

Now what about that GlobalContainer singleton we see in all the Spring4D examples? Actually if you want to make it 100% correct you should create your own TContainer instance - you remember you only need that at the start of your application to register all your types and then resolve the first instance and from there on it will never be seen again throughout your whole application.

If you ever have heard someone telling you that you should put your classes in your implementation part of the unit and then register them in the initialization part of that unit - never ever do that please!
First you are making these classes untestable (because not accessible from outside) without the DI container - always write your code in a way that it is testable without a container. Second you are tightly coupled to the GlobalContainer instance - what if you created your own one - you would be screwed.

Solving the problem

But now let's get back to our initial problem. Having things that might be needed later or even never throughout one run of the application. That is when we need lazy initialization.

Let's see how the example from the other article would look in Delphi:

container.RegisterType<IExampleService, TExampleService>('default').AsDefault;
container.RegisterType<IExampleService, TAnotherExampleService>('another');

container.RegisterInstance<TFunc<string, IExampleService>>(
  function(name: string): IExampleService
  begin
    Result := container.Resolve<IExampleService>(name);
  end);

container.Build;

Assert(container.Resolve<IExampleService> is TExampleService);
Assert(container.Resolve<TFunc<string, IExampleService>>
  .Invoke('another') is TAnotherExampleService);

So if we had another class that would take this factory as an argument on its constructor the container would inject this anonymous method there - keep in mind that we used RegisterInstance which returns the same anonymous method every time. In this example it is completely valid because the anonymous method has no state but pay attention when you use variable capturing.

You can imagine that this will be much code to write if you have many service types and you want to resolve many of them lazily. But just like other DI containers the Spring4D container has built-in support for that. Take a look at this code:

type
  THomeController = class
  private
    fService: IExampleService;
    fServiceFactory: TFunc<IExampleService>;
    function GetService: IExampleService;
  public
    constructor Create(const serviceFactory: TFunc<IExampleService>);
    property Service: IExampleService read GetService;
  end;

constructor THomeController.Create(const serviceFactory: TFunc<IExampleService>);
begin
  inherited Create;
  fServiceFactory := serviceFactory;
end;

function THomeController.GetService: IExampleService;
begin
  if not Assigned(fService) then
    fService := fServiceFactory();
  Result := fService;
end;

container.RegisterType<IExampleService, TExampleService>('default').AsDefault;
container.RegisterType<IExampleService, TAnotherExampleService>('another');
container.RegisterType<THomeController>;
container.Build;

Assert(container.Resolve<THomeController>.Service is TExampleService);

As you can see we did not register the factory method anywhere. The container took care of that and passed it to the constructor.

We now have the lazy initialization logic inside the getter because the container did just give us a factory method. Every time we would call it the container would run its logic and create a new service (unless we registered it as singleton of course). But Spring4D contains the lazy type - so we can reduce this code a bit and make use of that:

type
  THomeController = class
  private
    fService: Lazy<IExampleService>;
    function GetService: IExampleService;
  public
    constructor Create(const service: Lazy<IExampleService>);
    property Service: IExampleService read GetService;
  end;

constructor THomeController.Create(const service: Lazy<IExampleService>);
begin
  inherited Create;
  fservice := service;
end;

function THomeController.GetService: IExampleService;
begin
  Result := fService;
end;

The rest of the code is pretty much the same. The Spring4D container has built-in support to TFunc<T>, Lazy<T> and ILazy<T> where T needs to be a registered service type that can be resolved without a name - either by only having one implementor or having specified AsDefault on any.

So this was the first article on dependency injection best practices - more will follow. As always your feedback is very much appreciated.

By the way - in case you missed this - Spring4D is now hosted on Bitbucket.

Future plans for Spring4D and DSharp

Some of you may have read the public letter about Spring4D project two weeks ago. So this is the promised post about the roadmap (well more of some plans that are not set in stone yet but in our minds) for the future.

First of all I am really happy when I hear that people are using Spring4D in their projects and that it is not just an experimental thing to bring some Java or .Net flavor into the Delphi world. And as always software only becomes better when it gets used. In the past there were many features added because you as the users needed them.

However communication and sharing experiences between you as users and us as the development team is still not as good as I wished. In the past there were many channels used to share experiences or ask for help. Getting them focused more in one place will help to share and build knowledge - so for everyone that is not following us please join us on groups.google.com/forum/#!forum/spring4d.

The project is now hosted on Bitbucket in a Git repository which will make the workflow easier than it was with Subversion. We will follow the Gitflow Workflow which might look confusing first but is really easy. For you as user it means, clone from the master if you want the stable and thoroughly tested source. For stuff currently in development either clone the develop branch or one of the possible feature branches.

Some changes have been made to the collection types recently which are important things I will talk about in a few. These are mostly non breaking changes but there were some small API changes which are easy to apply after you got the latest version.

The DI container got some really nice features in the past months which will be subject of an extra blog post that will follow soon.

Ok, but what about the future?

One thing that is often criticized about the project is the poor documentation - apart from some blog posts here and there - this is something we want to change.We will improve the source documentation (using DocumentInsight actually makes this fun!) and the general documentation about the different features of the framework as well as best practices for dependency injection.

Another goal will be to improve unit tests as some parts are not as well tested as they should be. This will also help adding new features in the future and/or refactor existing ones.

Some other projects will work more closely with Spring4D - one being about ORM - more about that when we have more details.

We want to make Spring4D a de facto standard for Delphi developers.

There are also some interesting things about Spring4D in Nicks upcoming book - so you might to take a look.

Another project that closely works with Spring4D is DSharp. Currently there are some redundant parts in both frameworks and if you are using both it sometimes is hard to decide which one to use. Also this might cause incompatibilities. We are working on reducing and finally removing these.

DSharp had a very simple but powerful MVVM implementation for some while. This was very basic and more of a proof of concept than something you can use for bigger applications but I am happy to tell you that we are bringing you a MVVM solution for Delphi that is based on Caliburn Micro. Some people might have seen some previews by Jeroen Pluimers on the ITDevCon or the BE Delphi this year. Currently we are aiming for a beta in next spring and more details on that project will follow.

Are you interested in helping in one of the projects, have questions or just want to share your experience?Please let us know!

Implementing custom iterators with delayed execution

The spring4d users amongst you will be familar with methods like IEnumerable<T>.Where which just returns the elements of an enumerable (mostly a list) that match a certain condition passed as delegate.

Today we will take a look how that is achieved and what requirements should be met when implementing such a method. The method itself looks like this:

function Where(const predicate: TPredicate<T>): IEnumerable<T>;

You can see that it just returns an enumerable so in fact you could chain multiple Where calls and get an enumerable that only contains the elements that match every condition.

The execution of the filtering will be done delayed so as long as we don't use that enumerable in a for-in loop nothing will actually happen. In fact you can still modify the list you called Where on and it will consider these elements when starting the iteration. That means when we call Where another enumerable object will created that just saves the informations to perform the filtering: the source and the delegate.

Also the execution will be streamed. That means the elements in the original source will only iterated once and only when they are needed. So if you for example cancel the loop early it will not have iterated all elements in the source enumerable.

In the spring4d branch there is a new class called TIterator<T> that implements both IEnumerable<T> and IEnumerator<T>. Usually you have seperate classes for enumerable and enumerator but in our case the enumerable is just saving the state for the operation and passes these informations to the enumerator class because that one is doing the real work in the MoveNext method. So we can get rid of the redundant information and put them into the same class.

So we look at this:

type
  TIteratorBase<T> = class(TEnumerableBase<T>, IEnumerator)
  protected
    function GetCurrentNonGeneric: TValue; virtual; abstract;
    function IEnumerator.GetCurrent = GetCurrentNonGeneric;
  public
    function MoveNext: Boolean; virtual;
    procedure Reset; virtual;
  end;

  TIterator<T> = class(TIteratorBase<T>, IEnumerator<T>)
  private
    fThreadId: Cardinal;
  protected
    fState: Integer;
    fCurrent: T;
  protected
    function GetCurrent: T;
    function GetCurrentNonGeneric: TValue; override;
  public
    constructor Create; override;
    function Clone: TIterator<t>; virtual; abstract;
    function GetEnumerator: IEnumerator<T>; override;
  end;

Let's look at the GetEnumerator method:

function TIterator<T>.GetEnumerator: IEnumerator<T>;
var
  iterator: TIterator<T>;
begin
  if (fThreadId = TThread.CurrentThread.ThreadID) and (fState = 0) then
  begin
    fState := 1;
    Result := Self;
  end
  else
  begin
    iterator := Clone;
    iterator.fState := 1;
    Result := iterator;
  end;
end;

This ensures that you get a new instance as enumerator when necessary. In the case of the same thread and its first iteration it just returns itself (which is the most common case).

To implement the Where operation we basically just need to implement 3 methods: Create, Clone and MoveNext - let's take a look at the class for that:

type
  TWhereEnumerableIterator<T> = class(TIterator<T>)
  private
    fSource: IEnumerable<T>;
    fPredicate: TPredicate<T>;
    fEnumerator: IEnumerator<T>;
  public
    constructor Create(const source: IEnumerable<T>;
      const predicate: TPredicate<T>);
    function Clone: TIterator<T>; override;
    function MoveNext: Boolean; override;
  end;

As said above the real work is done in the MoveNext method - constructor and Clone method are a no brainer.

function TWhereEnumerableIterator<T>.MoveNext: Boolean;
var
  current: T;
begin
  Result := False;

  if fState = 1 then
  begin
    fEnumerator := fSource.GetEnumerator;
    fState := 2;
  end;

  if fState = 2 then
  begin
    while fEnumerator.MoveNext do
    begin
      current := fEnumerator.Current;
      if fPredicate(current) then
      begin
        fCurrent := current;
        Exit(True);
      end;
    end;
    fState := -1;
    fEnumerator := nil;
  end;
end;

Now this is the real interesting part. As in the requirements the source is not iterated as long as we do not iterate the filtered enumerable. When MoveNext is called for the first time we get the enumerator from the source. And it is only proceeded as much as needed. Since Delphi does not support some yield syntax and compiler generated iterator blocks like C# this source looks a bit more complicated as it would with that syntax support but I guess it is easy enough to understand and implementing these iterators is pretty easy since you only need to implement the MoveNext method (apart from a pretty much straight forward constructor and Clone method).

However just to tease you - wouldn't it be nice if we could write something like this?

function TEnumerable<T>.Where(const delegate: TPredicate<T>): IEnumerable<T>
var
  item: T;
begin
  for item in Self do
    if predicate(item) then
      Yield(item);
end;

If you are interested more about that topic I suggest reading Jon Skeets Reimplementing LINQ to Objects series.

Using ARC with non iOS compiler - possible?

Did you ever have the situation with 2 lists sharing the same objects and possibly one or both had the OwnsObjects property set to true? Especially when using interfaced lists like in Spring4d or DSharp you want to make use of reference couting and automatic memory management. But what if some instances are shared between different lists or moved from one list to the other. You cannot use the Delete or Remove method and then add the object to the other list of vice versa if the first list has OwnsObjects true because it will destroy the instance when it gets removed. That is why there is the Extract method which removes the instance but does not destroy it. This can get complicated and possibly leading to memleaks or exceptions quickly.

Using interfaced objects for simple data storages might not be a very good idea. It requires you to write interfaces with the properties just for that purpose and if you use some kind of RTTI based binding you cannot do that as there is no RTTI for interface properties (which are just syntax sugar anyway).

So what would we give for some easy memory management in these cases. How about using the concept we know from interfaces and other types for objects?

Of course we could write some TRefCountObject and inherit our data classes from that base class and handle these in our Notify method inside the list when items get added or removed. But that would be to easy and not magic at all. ;) And more seriously it does not always work to change the base type due to several reasons.

So what do we need? Basically just a new field inside our object to keep track of the reference count. Keep in mind we cannot do a full ARC implementation because that would include assignments and parameter passing which would need compiler support. We just want it for when objects are put into lists.

The method that is responsible for allocating the memory of new instances is TObject.NewInstance. So we need to replace that:

procedure InitializeARC;
var
  Buffer: array[0..4] of Byte;
begin
  Buffer[0] := $E9
  // redirect TObject.NewInstance
  PInteger(@Buffer[1])^ := PByte(@NewInstance) - (PByte(@TObject.NewInstance) + 5);
  WriteMemory(@TObject.NewInstance, @Buffer, 5);
end;

What this code does is place a jump instruction at the very beginning of the TObject.NewInstance method that redirects it to our NewInstance routine which looks like this:

function NewInstance(Self: TClass): TObject;
begin
  // get additional memory for the RefCount field
  GetMem(Pointer(Result), Self.InstanceSize + SizeOf(Integer));
  Result := InitInstance(Self, Result);
end;

It does basically the same as the original except that it allocates 4 bytes more for our RefCount field and then calls our version of InitInstance (which is responsable for initializing the object):

function InitInstance(Self: TClass; Instance: Pointer): TObject;
const
  Buffer: Pointer = @BeforeDestruction;
begin
  Result := Self.InitInstance(Instance);

  // initialize the RefCount field
  GetRefCountFieldAddress(Instance)^ := 0;

  // replace TObject.BeforeDestruction
  if PPointer(NativeInt(Self) + vmtBeforeDestruction)^ = @TObject.BeforeDestruction then
    WriteMemory(PPointer(NativeInt(Self) + vmtBeforeDestruction), @Buffer, SizeOf(Pointer));
end;

Since TObject.InitInstance just zeroes the memory the RTL knows about (obtained by calling InstanceSize) we need to set our field which sits on the last 4 bytes in our instance:

function GetRefCountFieldAddress(Instance: TObject): PInteger; inline;
begin
  // the RefCount field was added last
  Result := PInteger(NativeInt(Instance) + Instance.InstanceSize);
end;

Along with the reference couting we want to make sure that the instance is not getting destroyed when it is still managed by the RefCount (because it sits in some list). That is why the BeforeDestruction method gets replaced. Why not detour like NewInstance? The implementation in TObject is empty so there are not 5 bytes of available that we can overwrite to jump to our implementation. But as it is virtual we can replace it in the classes VMT. Like its implementation in TInterfacedObject it will raise an error when the RefCount is not 0.

procedure BeforeDestruction(Self: TObject);
begin
  if GetRefCount(Self) <> 0 then
    System.Error(reInvalidPtr);
end;

Implementing the actual AddRef and Release routines is pretty easy aswell:

function __ObjAddRef(Instance: TObject): Integer;
begin
  Result := InterlockedIncrement(GetRefCountFieldAddress(Instance)^);
end;

function __ObjRelease(Instance: TObject): Integer;
begin
  Result := InterlockedDecrement(GetRefCountFieldAddress(Instance)^);
  if Result = 0 then
    Instance.Destroy;
end;

The most important thing: You need to add the unit which contains this as the very first unit in your project  (or after ShareMem) so the NewInstance method gets patched as soon as possible.

Time to test if it does what it should:

implementation

{$R *.dfm}

uses
  DSharp.Collections,
  DSharp.Core.ARC;

type
  TList<T: class> = class(DSharp.Collections.TList<T>)
  protected
    procedure Notify(const Value: T; const Action: TCollectionChangedAction); override;
  end;

procedure TList<T>.Notify(const Value: T;
  const Action: TCollectionChangedAction);
begin
  case Action of
    caAdd: __ObjAddRef(Value);
    caRemove: __ObjRelease(Value);
  end;
end;

procedure TForm1.Button1Click(Sender: TObject);
var
  list1, list2: IList<TObject>;
begin
  list1 := TList<TObject>.Create;
  list2 := TList<TObject>.Create;

  list1.Add(TObject.Create);
  list1.Add(TObject.Create);
  list2.AddRange(list1);
  list1.Delete(1);
end;

initialization
  ReportMemoryLeaksOnShutdown := True;

end.

When we click the button both objects get added to both lists and the last list containing an object will cause it to get destroyed when removed (which happens if the list gets destroyed aswell).

So far this is more of a proof of concept but I think this can make some code easier and less complicated especially when working a lot with lists and moving around objects without knowing what list at the end owns the objects.

You can find that code in the svn repository and as always your feedback is welcome.

Why no extension methods in Delphi?

I have been wondering this for a long time now. Why does Delphi not have this great language feature? Sure, we have helpers but they are not the same. First they are still officially considered a feature you should not use when designing new code. However it opens up some interesting possibilities - some might call it hacks... but that is a topic for another day. Second they don't work on interfaces or generic types. And interestingly though that is what I want to talk about today.

But first - if you don't know it already - I suggest you read what an extension method is - really I could not explain it any better. Oh, and to all readers that want to jump the "stop the dotnetification of Delphi - keep these new language features away" wagon - this post is not for you, sorry.

Ever had a class or a set of classes you wanted to add some functionality to? Sure, there are ways to do so like the decorator pattern. But did you see the problem there if you have a type you cannot inherit from because either you cannot modify the code or it's not a class but an interface? Well, then create a new interface and add that functionality there, someone might say. How, if you cannot extend the given type? Use the adapter or bridge pattern? You can see where this is going. You might end having to change existing code or introduce lots of code to apply your additional functionality.

The most prominent example of extension methods (and not surprisingly the reason they were introduced in C# 3.0) are the extension methods for IEnumerable<T>. If you want to use the foreach (or for..in loop in Delphi) all you have to implement is the GetEnumerator method (and actually the only method that IEnumerable<T> got). So if you ever need to implement that in some of your classes you implement just one method and got access to almost any query operation you can imagine - not saying they all make sense in every context, but you get the idea.

Extension methods are great. You don't clutter your class with things that don't belong there directly but apply to an aspect of your class (in our case being enumerable). They follow good principles like the dependency inversion principle. The way you are using them is more natural and makes more sense than having static methods (or routines) where you pass in the instance you want to call the method on as first parameter.

Even without the fancy LINQ Syntax without question it is much more readable to write

for c in customers.Where(HasBillsToPay).OrderBy<string>(GetCompanyName) do
  Writeln(c.CompanyName);

instead of

for c in EnumerableHelper.OrderBy<string>(
  EnumerableHelper.Where(customers, HasBillsToPay), GetCompanyName) do
  Writeln(c.CompanyName);

And that statement has only two chained calls - imagine how that grows in length if you got a more complex query with grouping or something else. Also in that case it is the order on how the query gets processed - easier to read and to write.

But - you remember - no helpers for interfaces and generics! Well, we can implement these methods in our base TEnumerable<T> class and/or put it on our IEnumerable<T> interface, no? Yes, we can. And everything would be fine if there wasn't one tiny detail - how generics are implemented in Delphi and how the compiler handles them: it generates a type for every specialization. Which means the same code compiled for every possible T in your application, for TCustomer, TOrder, TCategory and so on. Only with a small set of methods implemented (and possible classes for more complex operations like GroupBy for example) this means you get hundreds of KB added for each TList<T> you will ever use - even if you never touch these methods. That is because the linker cannot remove any method inside an interface even if never called.

So how to work around that problem (which is what I have been doing in the Spring4D refactoring branch lately)? Let's take a look again on how extension methods are defined. Nothing prevents us from creating that syntax in Delphi, so a Where method could look like this:

type
  Enumerable = record
    class function Where<TSource>(source: IEnumerable<T>;
      predicate: TPredicate<TSource>): IEnumerable<T>;
  end;

Simple, isn't it? But how to call it? We need a little trick here, let's see:

type
  Enumerable<T> = record
  private
    fThis: IEnumerable<T>;
  public
    function GetEnumerator: IEnumerator<T>;

    function Where(predicate: TPredicate<TSource>): Enumerable<T>

    class operator Implicit(const value: IEnumerable<T>): Enumerable<T>;
  end;

As you can see we use a record type that wraps the interface we want to extend and add the method there. We can implement the "extension methods" there or direct the call to our extension method type if we want to keep it seperatly.

We now have a nice way to do our query just like we wrote it above if customers where from Enumerable<T>. Or we can perform a cast (since we have an implicit operator that will get used). Also notice how the result of the Where method is of the record type. That way we can chain the calls easily. And because we implemented GetEnumerator we can use it in a for..in loop just like any IEnumerable<T>.

What's also nice about the record type is that the linker can now be smart and remove any method that we never call and save us dozens of megabytes in our binary (not kidding).

So our life could be so much easier if we had extension methods (or call them helper) for interfaces and generic types. But as long as we don't have that, we have to find some clever workarounds.

If you are a Spring4D user, check out the changes in the refactoring branch and let me know what you think.