Who You Calling a Jesse?

Accelerator Metrics and Developing Entrepreneurial Talent

I spent a little time at StartupWeekendHamilton3 in April as a mentor and was talking to one young founder that proclaimed that there was one great accelerator in Canada. Who he said it was surprised me a little and got me thinking, what makes an accelerator “the best” and why should an eager founder care? The baseline in my mind is Y-Combinator. No one can argue it is the best seed stage accelerator based on its results. What is difficult for everyone to agree upon is what does it do to achieve those results or even harder, what defines success?

In my opinion the key things it does:

• Social Capital via Paul Graham – how he teaches founders and the hacker culture he has built provides entrepreneurs with access to the very best social capital that exists for anyone starting a technology based company.
• Peer mentorship – the structure of the 12 weeks enables peers to hold each other accountable. This competition amongst comrades is powerful as it turns around the human nature of playing to our own strengths and pushes founders to “keep up with the Jones’s.”
• Hungry founders – funding is minimal. After a bit of a bump it has since been decreased and I would bet if you look at the successes out of YC the biggest ones started off with the least amount of financial resources.

There is some striking similarity to what YC does and the thinking/observations behind the Goldmine Effect by Rasmus Ankerson (watch it, it is interesting). The basic point is that if you can find the talent that has the potential vs the talent that already been refined you will get a better result. Money and facilities do not make a difference, identifying underdeveloped talent does. I think there are three core factors that go into determining the quality of a given program.

• Where is the program located? Are there companies in the immediate area just a stage or two ahead that can help you grow?
• Who is backing the program and what did they invest to make it happen? Do they get involved in the companies they invest in or do they “spray and pray” with their investment?
• What type of companies have been successful in the accelerator in the past? Who gets funding afterwards? Are the B2B or B2C, SaaS or something else, etc.

What is less important:

• Demo Day: The rock show nature of Demo Days is not a good environment for investors but you need to take advantage of the intros and the social capital on offer to build those connects yourself.
• Money: Funding amounts from the accelerator should not influence your decision to go there. Good companies will get funding, build a good company and spend as little as possible doing it.
• Mentor walls: In Canada there is a relatively small pool of people with both time and capital but there are a lot of people that can help you move the needle in different ways.

Right away some might say that the above “less important” items are what builds momentum and if you look at the YC companies momentum being 3x that of TechStars then how can I say that is less important? These things have the greatest effect after the startup object is already in motion, in my opinion. The less important items are used all too often as *the* way to get the startup object moving.

A simple score card to find out who’s best for you

If a score card was set up to measure a program it should look something like this:

• The program is located near companies that I am interested in working with
  • 1 – none that I know of
  • 3 – some interesting founders
  • 5 – who we would exit to and/our would like on our advisory board are within walking distance
• Investors in successful companies that have been in the program are
  • 1 – Not involved in investments
  • 3 – one of 12 investors in the companies that graduate
  • 5 – take a board seat and/or a significant position in the financing round following completion of the program.
• Companies that have been successful in the program in the past are
  • 1 – nothing like us, we are B2B SaaS and all the successful companies are gaming companies
  • 3 – some are similar to us, there is no particular pattern to the type of company
  • 5 – just like us, we are a hardware company and everyone that has done well post-program are hardware companies
• Funding we receive from the accelerator program is enough to
  • 1 – we can go 6-12 months no problem, its great to not have to raise or find revenue right away
  • 3 – it is ok but in 6 months if we don’t have revenue or financing we are done.
  • 5 – we can pay rent while in the program but we have to move and stay lean to survive.

This is by no means research quality metrics but it does start to assign some way to weight rankings… for you. If I was going to score YC I would give them a 5, 3, 4, and 5 which would total at 17/20.

What else should be on this scorecard?

How does research spending in higher education translate to startups?

The issue of startup funding falling short in Canada is talked about in startup circles just as much as the weather in this country. This topic is something I have shared my opinion on before but that post was aimed at early stage companies. I am not sure if there really is a problem with funding or just with the companies in Canada that are at that stage. A more serious worry about this conversation is the rational that academic research (ergo the institutions that conduct them) are less important than VC investment in economic development:

“We’ve bought into the idea that academic research is the engine of economic development and that’s a fallacy,” says Dr. Patricia Lorenz, chair of NAO.

Canada spends roughly $11.3 Billion on Higher Ed based research. The top school on research spending is the University of Toronto with $915 Million of that. The next highest school is at $575 Million (UBC) then at #6 it drops to $325 Million.  This isn’t far off from US schools but there are a lot more schools with research spending over $100 Million. Alumni from ‘top schools’ in the US have received $12.5 Billion in funding across 559 deals since 2007. These are people that have been exposed directly or indirectly to the environment that is created around the research spending of those schools. We don’t have similar data in Canada (that I know of).

What were the Federal (government) research dollars spent at the “top schools” in the US?

• Stanford University ($840 million)
• Harvard University ($686 million)
• University of California, Berkeley ($694 million)
• New York University (I couldn’t find a number)
• University of Pennsylvania ($770 million)
• Massachusetts Institute of Technology ($677 million)

If you assume NYU is a bit above the average of the above in spending, that amounts to an annual of roughly $4.5 Billion in research spending at just 6 schools. Schools develop the talent that builds the companies that require the funding. Those are big number unless you contrast that with Canadian company R&D spending which is pegged at $10.9 Billion last year (I don’t know what amount of that goes to sponsored research in universities). RIM and Bombardier, top of that list, account for $1.54 and $1.34 Billion each.

In total, three times the research spending of the University of Toronto is going to closed research to aide mobile devices, snowmobiles, planes, and trains. I am not saying that is a bad thing at all. What I wanted to point out is that, relatively speaking, Universities really don’t spend that much on research factoring in the diversity of the research and the number of people that benefit from it directly or indirectly.

The persistant question that people tend to oversimplify, how much of that research spent at Universities translates into economic development? The answer to that depends on your metrics. Typically I think people point towards the commercialization results of a university. That is a only a part of the picture. The numbers above from 6 schools in the US that spend $22.5 Billion over five years ($4.5/yr) turned out students that raised $12.5 Billion in financing. Are you going to question any of those top US schools commercialization or their role in being an “engine of economic development?”

btw, ATI was founded in a dorm room at UofT in 1984 and exited for how many billions after changing the world of computers? Ebay? WattPad? Do we need to list them all? We probably do.

Why I like a Customer Advisory Board

Seeking out customer feedback and using it to build a great product is not a new concept. Great designers have been doing it for a long time as have great companies. The Lean Startup manual (or startup bible to many) talks about involving the customer while developing that Minimum Viable Product: “The minimum viable product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.”

Where that generally leads people is straight to building a simple application that might not be sexy in its design but it is functional or a landing page about a new product that might not exist yet. Using Google analytics and collecting email addresses along with some ‘conversion’ point becomes what you focus on. However, if you forget to talk to actually talk to customers as well you could be wasting a lot of time. Especially when you are moving passed your MVP or have a product that people are paying for.

Live and die by automated metrics

Sales numbers and in application metrics don’t give you the whole story.  Your sales numbers tell you that your marketing is working and your sales people are doing their jobs. Especially if sales are going up and to the right. What those numbers don’t tell you clearly enough is if people are finding the product they actually want or if it just simply close enough they want to give it a try. In application usage metrics might help identify if people love it or what part they love but it might put you in a situation that you have to react instead of being pro-active on product. By the time you know you might just have good marketing and sales people and the product is off the mark based on people not really using the product (it gets obvious when your churn rate climbs) you loose momentum.

Momentum is so very important for a startup or any business for that matter. If people aren’t coming back for more, it is a big problem for most companies. Talking to customers regularly helps you understand the metrics better but taking customer feedback and translating it to product effectively eludes many because it is an art, not a science.

I like the ‘old fashioned’ process of drawing out a prototype, getting time with who you think your customer is, and spending time listening to them along with answering their questions. Automation can come later. If I want to collect the most information with the least effort I want paper prototypes and discussions with customers. I can emphasize enough how important that first highly talkative but very loosely connected (socially) customer is (the Alpha customer).

Once you have built something you can track all the metrics you want but I would argue you will not get good insight on your customer with automated measurements alone. Having sat in enough usability studies over the years I don’t think how people use something represents their interest in using that something, most of the time. By sitting down and talking to customers you build a much better persona in your mind of who your customers are and what those automated numbers actually mean.

Going back to the customer regularly

If you want customer feedback to be enshrined into the organizational culture you need to put a little bit of a formal process around it. The established way to do this is by setting up a ‘Customer Advisory Board‘ made up of your most feedback giving customers. You can all it something else if you would like but don’t leave out the basics.

• Treat the customers in this group with the same respect you treat your Board of Directors. They are volunteering their time and providing invaluable feedback.
• Be as open and transparent with this group as possible. Create an environment where they aren’t afraid to offend you about your product and me totally open with them about future product direction. If you feel they must sign a Non-Disclosure agreement to be comfortable with being open, ask them to do so.

How it works is really simple. You have a scheduled meeting at a regular interval, use your favorite screen sharing conference application (Google hangouts work really well btw), and start listening to your customers. Take lots of notes, write a summary of the meeting, and share it with the board members. You can use that to keep the conversation going via email between meetings. From there you can decide what level of participation and engagement works for you. It is a process to start using a Customer Advisory Board well so ease into it and enjoy it.

The advantage to setting up something a bit more formal is that you are making a commitment to use it. The timing on doing this for a startup could be on day 1 as you build out your MVP and if you are a company with a product that is selling now but don’t have a customer feedback process in place, you should think about doing it.

Meaningful metrics for incubators and accelerators

Incubators and accelerators are businesses just like the businesses they intend to help develop as they travel through the startup lifecycle. As with any business, there are indicators that they can measure to give them a better idea of how they are performing besides the big public relations buzz around a company being funded.

You need to measure these numbers so that when a success happens you can hopefully gain some insights on how to help the other companies better. The problem is that even though the model of an incubator or accelerator is generally known, how to take 10 companies and have 10 successful growth companies come out the other side of the program is not.

The issue of what metrics to use is an important but complicated problem to solve.

Set the baseline at the application process (pre-program)

There are far more applicants than slots offered in an incubator or accelerator program. However, it is at this point that a program is gathering it’s best intelligence. You need a baseline measurement at the start of the program that you can measure every team against. What you should be tracking:

• Who applied to the program that you didn’taccept (this is your control sample)
  • Track their progress on Angellist, Crunchbase, and/or go back to their web site in 3, 6, 12 months.
  • Keep a ratio of who is still in business and what their status is.
• Maintain, in a CRM system, information on the applicant founders and their team members.

Measure the incubator/accelerator clients (in-program)

At this point there are X number of startups with Y number of founders and maybe Z employees. What you want to measure are things that demonstrate they have improved (or not) and which are things you would expect to see improve as a result of the services provided by any incubator or accelerator:

• Current customers and revenue per customer (for most that will be 0 at the start) that will work across revenue models: CAC, ARPU, churn rate.
• Sales funnel – do they have leads? How many? Are they qualified leads? What are they worth?
• Average user growth in the last month.
• What mentors or advisors did they meet through the program? What role did they take with the company?

Run these numbers at the start and at the end of the program. If you are a pure research focused incubator, ignore this section. You have a much longer time to see success – but few are truly research focused.

Monitor the graduates: Alumni (post-program)

This is a very important thing an incubator/accelerator can do — build and maintain its alumni connections. These folks not only help at every stage of running future programs but their success lifts the profile of the program, just like how alumni of prestigious business schools make the business schools prestigious.

There should be reporting milestones at a set interval (probably financial quarter based) where you gain the following insights on the company:

• Customer growth percentage: CAC, ARPU, and churn rate all expressed as percentage growth.
• Sales funnel growth expressed as a percentage.
• Average user growth in the last month.
• What mentors or advisors are currently active with the company?

Ideally you should have a position that is equivalent to a close advisor or board observer with the company once it graduates from the program.

Defining success

If an incubator or accelerator program is successful, the graphs should be heading up and to the right at a much faster pace than they would have been had startups not entered the program.

The only baseline data I know of is from the Startup Genome. In their report they explain the stages and the average length of time it takes a company to go through them. For an incubator or accelerator to demonstrate that they work, I would expect a successful company to move through the stages faster than the average. I would also expect them to fail faster than the average.

Tracking metrics puts a lot more overhead on an accelerator. It is likely more than they budgeted for to start. However, if you want to know if the program is successful it is worth the investment of an admin salary to track and crunch data. This is just a baseline, track more and figure out what the indicators of success are for you.