Originally posted at 1/7/2011

This is another oft requested feature that we just implemented. The new feature is available for the full suite of Uber Profilers:

• NHibernate Profiler
• Entity Framework Profiler
• Linq to SQL Profiler
• LLBLGen Profiler
• Hibernate Profiler

You can see the new feature below:

I think it is cute, and was surprisingly easy to do.

Uber Prof have recently passed the stage where it is mostly implemented using itself, so I just had to wire a few things together, and then I spent most of the time just making sure that things aligned correctly on the UI.

Originally posted at 11/25/2010

In a recent post, I discussed the notion of competitive advantage and how you should play around them. In this post, I am going to focus on Uber Prof. Just to clarify, when I am talking about Uber Prof, I am talking about NHibernate Profiler, Entity Framework Profiler, Linq to SQL Profiler, Hibernate Profiler and LLBLGen Profiler. Uber Prof is just a handle for me to use to talk about each of those.

So, what is the major competitive advantage that I see in the Uber Prof line of products?

Put very simply, they focus very heavily on the developer’s point of view.

Other profilers will give you the SQL that is being executed, but Uber Prof will show you the SQL and:

• Format that SQL in a way that make it easy to read.
• Group the SQL statements into sessions. Which let the developer look at what is going on in the natural boundary.
• Associate each query with the exact line of code that executed it.
• Provide the developer with guidance about improving their code.

There are other stuff, of course, but those are the core features that make Uber Prof into what it is.

The following features apply to NHProf, EFProf, L2SProf.

In general, it is strong discouraged to data bind directly to an IQueryable. Mostly, that is because data binding may actually iterate over the IQueryable several times, resulting in multiple queries being generated from something that can be done purely in memory. Worse, it is actually pretty common for data binding to result in lazy loading, and lazy loading from data binding almost always result in SELECT N+1. The profiler can now detect and warn you about such mistakes preemptively. More than that, the profiler can also now detect queries that are being generated from the views in an ASP.Net MVC application, another bad practice that I don’t like.

You can find more information about each warnings here:

• Data Binding & Queries Shouldn’t Mix
• Don’t Query From The View

WPF detection:

WinForms detections:

Web applications:

The following features apply to NHProf, EFProf, HProf, L2SProf.

The first feature is something that was frequently requested, but we kept deferring. Not because it was hard, but because it was tedious and we had cooler features to implement: Sorting.

Yep. Plain old sorting for all the grids in the application.

Not an exciting feature, I’ll admit, but an important one.

The feature that gets me exciting is the Go To Session. Let us take the Expensive Queries report as a great example for this feature:

As you can see, we have a very expensive query. Let us ignore the reason it is expensive, and assume that we aren’t sure about that.

The problem with the reports feature in the profiler is that while it exposes a lot of information (expensive queries, most common queries, etc), it also lose the context of where this query is running. That is why you can, in any of the reports, right click on a statement and go directly to the session where it originated from:

We bring the context back to the intelligence that we provide.

What happen if we have a statement that appear in several sessions?

You can select each session that this statement appears in, getting back the context of the statement and finding out a lot more about it.

I am very happy about this feature, because I think that it closes a circle with regards to the reports. The reports allows you to pull out a lot of data across you entire application, and the Go To Session feature allows you to connect the interesting pieces of the data back to originating session, giving you where and why this statement was issued.

I recently got an estimate for a feature that I wanted to add to NH Prof. It was for two separate features, actually, but they were closely related.

That estimate was for 32 hours.

And it caused me a great deal of indigestion. The problem was, quite simply, that even granting that there is the usual padding of estimates (which I expect), that timing estimate was off, way off. I knew what would be required for this feature, and it shouldn’t be anywhere near complicated enough to require 4 days of full time work. In fact, I estimated that it would take me a maximum of 6 hours and a probable of 3 hours to get it done.

Now, to be fair, I know the codebase (well, actually that isn’t true, a lot of the code for NH Prof was written by Rob & Christopher, and after a few days of working with them, I stopped looking at the code, there wasn’t any need to do so). And I am well aware that most people consider me to be an above average developer.

I wouldn’t have batted an eye for an estimate of 8 – 14 hours, probably. Part of the reason that I have other people working on the code base is that even though I can do certain things faster, I can only do so many things, after all.

But a time estimate that was 5 – 10 times as large as what I estimated was too annoying. I decided that this feature I am going to do on my own. And I decided that I wanted to do this on the clock.

The result is here:

This is actually total time over three separate sittings, but the timing is close enough to what I though it would be.

This includes everything, implementing the feature, unit testing it, wiring it up in the UI, etc.

The only thing remaining is to add the UI works for the other profilers (Entity Framework, Linq to SQL, Hibernate and the upcoming LLBLGen Profiler) . Doing this now…

And we are done.

I have more features that I want to implement, but in general, if I pushed those changes out, they would be a new release that customers can use immediately.

Nitpicker corner: No, I am not being ripped off. And no, the people making the estimates aren't incompetent. To be perfectly honest, looking at the work that they did do and the time marked against it, they are good, and they deliver in a reasonable time frame. What I think is going on is that their estimates are highly conservative, because they don't want to get into a bind with "oh, we run into a problem with XYZ and overrun the time for the feature by 150%".

That also lead to a different problem, when you pay by the hour, you really want to have estimates that are more or less reasonably tied to your payments. But estimating with hours has too much granularity to be really effective (a single bug can easily consume hours, totally throwing off estimation, and it doesn't even have to be a complex one.)

An interesting thing happened recently, when I started to build the profiler, a lot of the features were what I call Core Features. Those were the things that without which, we wouldn’t have a product. Things like detecting SQL, merging it into sessions, providing reports, etc. What I find myself doing recently with the profiler is not so much building Core Features, but building UX features. In other words, now that we have this in place, let us see how we can make better use of this.

Case in point, the new features that were just released in build 713. They aren’t big, but they are there to improve how people are commonly using the products.

Renaming a session:

This is primarily useful if you are in a long profiling session and you want to mark a specific session with some notation:

Small feature, and individually not very useful. But you might have noticed that the sessions are marked with stars around them. They weren’t there is previous builds, so what are they?

They are a way to tell the profiler that you really like those sessions :-)

More to the point, such sessions will not be removed when you clear the current state. That lets you keep around the previous state of the application as a base line while you work to improve it. Beside, it makes it much easier to locate them visually.

And finally, as a quicker way to do that, you can just ask the profiler to clear all but the selected features.

Not big features, but nice ones, I think.

It has been a while since we had a new major feature for the profiler, but here it is:

The expensive queries report will look at all your queries and surface the most expensive ones across all the sessions. This can give you a good indication on where you need to optimize things.

Naturally, this feature is available across all the profiler profiles (NHibernate Profiler, Entity Framework Profiler, Linq to SQL Profiler and Hibernate Profiler).

After trying it out on NH Prof, profiler subscriptions are now opened for all the profilers.

• Linq To SQL Profiler
• Entity Framework Profiler
• Hibernate Profiler

A profiler subscription allows you to pay a small monthly free (~16$) and get the full profiler capabilities along with the assurance of no upgrade cost when the next major version comes out.

In addition to the monthly subscription, I got requests for a yearly subscription. I am not sure that I quite follow the logic, but I am not going to make it harder for people to give me money, so that is available as well for all profilers.

This is Josh’s feature, since we wrote most of the code for it together. Basically, it recognize a very common performance problem, queries that uses too many joins, such as this one:

Which would result in the following warning:

Queries with too many joins might be a performance problem. Each join requires the database to perform additional work, and the complexity and cost of the query grows rapidly with each additional join. While relational database are optimized for handling joins, it is often more efficient to perform several separate queries instead of a single query with several joins in it.

For OLTP systems, you should consider simplifying your queries or simplifying the data model. While I do not recommend avoiding joins completely, I strong discourage queries with large numbers of joins. Another issue to pay attention to is possible Cartesian products in queries contains joins, it is very easy to create such a thing and not notice it during development.

This isn’t a new feature, because you can’t use it right now, but it is a really nice feature that we are working on, and I couldn’t resist showing it off hot “off the press”, so to speak.

Given the following query:

SELECT this_.id             AS id7_1_,
       this_.title          AS title7_1_,
       this_.subtitle       AS subtitle7_1_,
       this_.allowscomments AS allowsco4_7_1_,
       this_.createdat      AS createdat7_1_,
       posts2_.blogid       AS blogid3_,
       posts2_.id           AS id3_,
       posts2_.id           AS id0_0_,
       posts2_.title        AS title0_0_,
       posts2_.TEXT         AS text0_0_,
       posts2_.postedat     AS postedat0_0_,
       posts2_.blogid       AS blogid0_0_,
       posts2_.userid       AS userid0_0_
FROM   blogs this_
       LEFT OUTER JOIN posts posts2_
         ON this_.id = posts2_.blogid
WHERE  this_.id = 1 /* @p0 */

SELECT this_.id            AS id0_1_,
       this_.title         AS title0_1_,
       this_.TEXT          AS text0_1_,
       this_.postedat      AS postedat0_1_,
       this_.blogid        AS blogid0_1_,
       this_.userid        AS userid0_1_,
       comments2_.postid   AS postid3_,
       comments2_.id       AS id3_,
       comments2_.id       AS id2_0_,
       comments2_.name     AS name2_0_,
       comments2_.email    AS email2_0_,
       comments2_.homepage AS homepage2_0_,
       comments2_.ip       AS ip2_0_,
       comments2_.TEXT     AS text2_0_,
       comments2_.postid   AS postid2_0_
FROM   posts this_
       LEFT OUTER JOIN comments comments2_
         ON this_.id = comments2_.postid
WHERE  this_.blogid = 1 /* @p1 */

The profiler can show you the query plan using this UI:

And here is how the same query looks like using the query plan feature in Management Studio:

So, why implement it?

• This isn’t limited to SQL Server, the profiler can display query plans for: SQL Server, Oracle, PostgreSQL and MySQL
• This let you keep yourself in the flow, just hit a button to see the query plan, instead of copying the SQL, opening SSMS, displaying the query plan, etc.

Don’t discount the last one, making it easy is one of the core values of the profiler.

The idea is that if you make it easy enough, the barriers for using it goes away. If you can instantly see the query plan for a query, you are far more likely to look at it than if it takes 30 seconds to get that. At that point, you would only do it when you already have a performance problem.