Last month, we sent out an early screener of our new feature length documentary to nearly 1500 people. We also asked this audience for objective feedback that could help us improve the film, and boy did they deliver.
Now before we get into all the data and how we used it to improve our film, make sure you check out last week's article. It's all about the exact methods we used to conduct this focus group (and how you can use them yourself), and it really sets the stage for this article. Plus, you get to see how we designed our surveys, including the exact questions we asked. It's cool stuff.
Also, this article is going to cover the basics of what our focus group data revealed, but if you want to check out the full 15-page report that breaks it all down in detail, you can download it down at the bottom.
With that out of the way, here's a quick sizzle reel (we're still working on the official trailer) for The New Hustle to give you a taste of the project:
Let's Talk About Sample Sizes
At the top of this article, I mentioned that we sent out early access to The New Hustle to nearly 1500 people. The number is 1267 to be exact.
Can you guess how many people took the time to watch the full hour-long film and answer all the survey questions?
Now, you might be thinking that's an insanely low number. After all, it's less than 7% of the folks who received the (multiple) emails we sent out with access to the screener. But that's totally ok. We still got incredibly useful data from that relatively small group of people.
Here's the main takeaway from this. When you're doing your own focus groups, it really is a numbers game. Especially as the length of your film goes up, the likelihood that people will watch it all the way through and answer all of your survey questions goes down. We're all busy people, and even if we want to help out and contribute, most of us just don't have a spare 70 minutes for something like this.
With that said, if you decide to conduct a focus group, don't worry about getting huge numbers of people to respond. Even 20 thorough responses is more than enough to glean some good, meaningful data that you can use to make your project better.
So with that out of the way, let's dive into what we actually learned from those 87 people who responded, and how we're improving our film.
Were People Confused By the Film?
This is probably the single most important thing we were looking for in the data. After all, a film that confuses people is a film that gets turned off.
Luckily, in our Comment Bubble statistics (this is the tool that lets people react to the video in real time), there weren't many individual moments in the film that consistently confused people. That was a good sign.
However, in the survey afterwards, when we asked people directly if there was anything that confused them about the film, we got some super useful answers. Here are a few:
"Some of the intercutting from one story to the next resulting in momentary confusion."
"Need something to signal/cue me that we're moving from one company to another. Took some time, more than once, to realize, 'oh, we're back to Luke...now what is it his company does...?'"
"It wasn't clear who the guy in the grey shirt was. But he was very informative and gave interesting insight into the tech startup world."
"The segue in the beginning between Vinomofo, and Safety Culture. Something about the transition left me feeling like it could use some finessing."
"It was hard to keep track of the characters without name keys. Even if they were up once, I might have needed a refresher part way through, it was a lot of characters."
What People Thought We Could Improve
The other area where we found consistencies in the data is when we asked people how they thought we could improve.
Here's a small sampling of how people responded
"Reduce the duration a little bit."
"I'm not really sure. There's an arc to the story, but not really any tension. Took a while to get invested in each of the three characters."
"SHOW us more of Luke's and Mel's products. In the sizzle reel, we got to see their products in use, in context. We could see what why they were so excited, and that made me, as a viewer excited and engaged. In this film, there's not enough of those concrete examples."
"In the first half, I had difficulty following the threads of the characters. I do not remember any of their names. In the middle, just when one thread got exciting, it switches to the other characters."
"Make it shorter, it drags a bit, at moments."
"Whilst there were great moments where storylines reached a moment of tension/conflict (eg. Andre & his wife with no money or Tom and his team stressing over the migration), I struggled to feel engaged because I'd be whipped onto another character."
Based on all of that, I'm sure you're starting to get a sense of the issues that came up consistently throughout the data. So here's what we did.
The Main Takeaways from the data
When it was all said and done, there were three key areas where that the data consistently indicated our film needed a bit of work: the transitions between scenes, the ambiguity of certain characters, and a lack of good examples of the different products showcased by each entrepreneur.
So here's what we're doing to fix those issues.
1. SMOOTH AND ADD CLARITY IN THE OPENING.
Why? To increase connection to the characters and reduce confusion early on.
A great deal of confusion came from not knowing who the characters were, and from being confused between storylines. The addition of title cards for each character, and potentially a reminder mid-way through the film, will be a welcomed addition.
2. EASE THE TRANSITIONs BETWEEN SCENES.
Why? To reduce the jarring feeling viewers are reporting.
Adding more b-roll to ease the transition between scenes will also have viewers prepare and recognize for where the story is headed. This, paired with the titles above, should ensure that the film can be followed by those unaware of who these founders are.
3. INCLUDE EXAMPLES OF USER STORIES FOR EACH COMPANY.
Why? To increase the connection with the mission behind the companies.
One of our main objectives for the film was recruitment. While our character favorability ratings are high, the connection to the companies could be increased with the addition of quick user stories, which would make what they create more tangible and real.
Want more Insights into the data? Download the free 15 page report
So there you have it. That's our plan for taking The New Hustle to the next level.
Once we iron out those story issues and get some proper sound mixing and color, this film is going to be ready for the world to see. And we're fairly confident that people will enjoy it (and understand it) because we took the time to conduct this focus group and listen to what our audience had to say.
And hopefully, these two articles gave you some ideas for how you can do the same with your next film.
Now go check out the focus group report, because it's pretty damn cool!