How I Tested That

David J Bland (0:2.283)
Welcome to the podcast, Scott.

Scott McLeod (0:4.216)
Thanks for having me, David.

David J Bland (0:5.947)
I'm so excited you're here. It's been so long. If nothing else, I'm going to use this podcast just to catch up with you ⁓ because ⁓ we had the pleasure of working together back at Neo and I had a blast on the projects we worked together. Maybe we could talk a little bit about those and our way we approach things. But you've been up to so much amazing stuff since then that I would love for our listeners to learn a bit about you and your background and what you're doing now and share any stories we can along the way about how you tested things in practice because you're one of the
most rigorous discipline people I know about testing that I've ever worked with. And so I imagine this could be a two hour long episode, but we will do our best to keep it short today.

David J Bland (0:48.267)
So ⁓ do you wanna chat about maybe a little bit about yourself before we jump into any testing stories?

Scott McLeod (0:55.054)
Yeah, I love, guess there's like the context in which I met you, David, at Neo was I came as a designer and like my background formally trained as graphic design to web design, the mobile design, and somewhere in there became a product manager as well. So my background kind of came from a design thinking perspective. I didn't necessarily go to school. Just kind of self-taught a lot of that stuff. So developing WordPress sites and mobile apps and bringing some of that like all well-rounded generalists to Neo was part of.
I think the process in which we met each other. ⁓
Since the Neo days, when we were acquired by Pivotal Labs, and since then, I went to continue some of that innovation consulting. So I helped Toyota for a while. I then worked at Facebook and Meta and helped them with early ⁓ user growth with ⁓ another ex-Neo employee. So was a nice kind of tangential jump from corporate innovation to doing it inside these organizations. ⁓ And then the last almost 10 years now, I've been at Resident, which we created a family of direct-to-consumer and omnichannel. ⁓
furniture and home furnishing brands, specifically mattress brands. ⁓ now I've kind of worn many, many hats, but in the context of meeting was definitely under product design and generalist kind of entrepreneurship skill sets.

David J Bland (2:13.473)
Yeah, when we met, this whole kind of lean startup thing was pretty new. And we were just trying to get people excited about experimentation or do any kind of experimentation. And you fast forward today and so much has happened since then. But back then, I remember working on some projects with Adobe and all these things that we were trying to test out, all these different things. it was just, it felt like there was some excitement there and it was a little bit of kind of wild west of
hey, we're just gonna be creative and try to find ways to test these before sinking way too much time and money into them. And I felt like we had this great little process where we jumped from from a project to project, you know, with different clients trying to figure out what are you trying to learn? What's the riskiest assumption? How do we go test that? ⁓ It was, ⁓ I think it was formative for both of us going through that process because it really, we were learning all together and sharing out what we were learning together.

Scott McLeod (3:9.920)
It definitely felt like the Wild Wild West at that time. It was great. I think we learned that like, we can launch these ads and test things. And for sure the learnings we had within Neo of test and learn, what's the risky assumption? How do we de-risk anything we want to experiment with? And especially how do you do that? Like on the least amount of capital necessary. And at that time, I think we were looking, people are raising crazy money for these startups that would go dead. And that question of like, how do we test and learn with the least amount of money or energy or effort was like a fun lens to put things through. ⁓
I think we were lucky at that time that ads were accessible. I remember launching those Twitter ads with you, Facebook ads, like we do things to test our idea before we made them true. ⁓ And I've only continued to use that thinking for the last 10 years of everything we do is like, how do we, what's the goal? How do we measure it? How do we test it? How do we like de-risk our assumptions here? ⁓ And how do we invest as little as possible with the most upside? So that's.
very interesting informative time. think we're pretty lucky to get to play around there. think you're right. The Wild Wild West in the sense of like just discovering that all together. Now some of it I take for granted that it's just like how I think and how the people around me start to think. It's not still standard, but it's becoming more and more of a talking point of testing.

David J Bland (4:24.693)
Yeah, I don't know if everyone understands that some of the like really polished apps they use today started off as off-brand experiments with you and I just kind of hacking stuff together with a couple of their people. And ⁓ I remember one of my favorite stories, I have to share this with our listeners was we were trying this sort of ⁓ social media type image generator ⁓ and we almost had no back end. And so we just had a Trello board. So people would... ⁓
basically put in what they wanted and it would drop a Trello card in and then you would manually make it and send it back to them. And that was such, like it didn't scale. And I was surprised you still talked to me today after that experience, but it was, it was like, we were so scrappy trying to figure out would anybody use the thing before we could, of course, build all these features and all these things, but we have no idea if people are actually going to use this. And so you would manually put them.

Scott McLeod (4:54.242)
Yep.

David J Bland (5:14.069)
You know, ⁓ here, here's your image. And then people would give you feedback and we use that to fine tune it. But that became one of the bigger products people use today.

Scott McLeod (5:22.530)
Yeah, that's a great and I use that reference today all the time for a Wizard of Oz experiment like how do we? Provide the solution or the end value to the consumer and they don't need to necessarily know how it gets done And that was the best like yeah They everyone put their requests for the image they wanted and then the little spinny thing would see great You'll get it soon and sometimes that was an hour and sometimes that was two days basically like how fast could I design their silly little image? ⁓
Yeah, what a fun set of it. I just log in every morning and like, what do I have to design today? Are there the most random little images? But even that was like, now we know based on the submission requests, what are the little features we need? What are the text boxes even? I think the first time we launched that experiment, it was like open-ended paragraph. What kind of social media image do you need? Like, oh wow, we're getting lots of different things. How do we start to structure that in a way? Even that iterative approach is like.
⁓ It's super interesting as a way of like providing the end value experimenting and we didn't have to build any complicated back end It was literally just a form into Trello, right? It was great

David J Bland (6:24.063)
Yeah, we use that on other ones as well. I remember one specifically, we thought we had a B2B play for small business owners. And then when we put it out in the wild, it was just people submitting ⁓ pictures of their pets. ⁓ Do remember that one? ⁓ Like, can you make my dog sparkle? ⁓ And it was ⁓ such

Scott McLeod (6:37.090)
Yeah.
No.

David J Bland (6:42.303)
It was so interesting thinking we had this. So it's looked amazing. We had a strategy. We had it all defined out, you know, ⁓ and then we go to test it in the wild. And of course it resonates with pet owners instead of B2B. It's just stuff like that. I feel you can't simulate that. You do have to go out and just test.

Scott McLeod (7:0.776)
Imagine if we raised two million dollars for that thing and then three weeks in you're like all we're getting is dog photos No one wants this for their like yeah, they're b2b ads anymore. It's Yeah, that was definitely a funny one. I kind of forgot about all those dog animations we had to do ⁓

David J Bland (7:17.385)
Yeah, that was, ⁓ I think that was the precursor to almost like live photos ⁓ that Apple rolled out. We were testing all these different things of what could be as like cinemagraphs and stuff at really early days. And it was such a fun time. So we could reminisce about those days forever. But I do want to move on to residents because your ⁓ your experience there has been amazing. I mean, just from the outside being an observer and seeing how that's grown and how you've embedded this culture there. Can you maybe test?
Tell us a little bit about how you joined resident and sort of your mindset behind how you want to take all that stuff we learned together and apply it there.

Scott McLeod (7:56.578)
Yeah, I think it's a ⁓ really interesting approach of like some of the experience I had at NEO, particularly this testing and methodology approach. The common thing there was data. So collecting data to understand our assumptions. ⁓ And I have this like Peter Drucker's quote stuck in my head. If you can't measure it, you can't improve it. And I kind of pervert that to say like, if you can't measure it, you can't manage it. And knowing that everything needed to be managed in our business from a really early day, you know, I was one of the first five original team members. ⁓
A big piece I imagined is let's make sure every dollar we spend is well invested back in the business. So we tracked everything we can ⁓ and you compare us to our cap like mattresses relatively capital intensive, but our competitors, the Casper's, the purples, the other, they raised hundreds of millions of dollars and we did it. So we we had to be diligent in the early days of every dollar we spent. We had a confidence in ROI. ⁓
What that really looked like is measuring every single thing in the business, quantifying them, and then clearly looking at, hey, ⁓ how do we improve certain metrics? So that whole framework was always test and learn is great. Return rates high. How do we improve this? Bounce rates high here, add to carts high here, every single thing we'd kind of look at and then launch experiments to try and improve these core metrics. So that first year was a lot of like price offer product. How do you position it? What paid channels are working, et cetera. So a whole bunch of testing as a method.
⁓ and because we had this data-driven approach to everything we did and really like the hypothesis test and learn shifted for me to just be data-driven like everyone thought about using data to navigate the business and you don't just look at data as like map you have to constantly have a hypothesis of how to adjust or change those things ⁓ and that was our first year is being really data-driven measuring everything and particularly like multi-touch attribution so understanding the value of every channel was a big piece of our
test and learn methodology, I guess, is how do we improve the marketing mix? ⁓ How do we improve the offers and products on site? And how do we improve the digital experience? I like joke to people sometimes that, you know.

Scott McLeod (10:0.234)
At the time in our early days, ⁓ now we're four in 10 mattresses sold online. like we're most of the beds sold online. In the early days we'd be compared to a Casper or whoever, a Tempur-Pedic. We go, not quite like them. If you haven't been shopping in the market for a mattress, you may not have heard about us. And we were very diligent about only people who are really warm in the funnel we would be converting. We weren't running billboard ads or TV ads. And even today, know, largest direct to consumer mattress brand, we're still really methodical about what we test and learn in terms of deploying dollars.
dollars in time and energy. So early on we saw it as a portfolio of bets and we had data and had to try and improve the business. ⁓ And Eric, our CEO, will always kind of reference back to like, it's a coin operated game. We're just having fun. If you're measuring everything. Now our goal as a team is to improve metrics. ⁓ Everyone's coming up with ideas and hypotheses and then you just go and test them. And we created this really friendly, healthy ⁓ test and learn. ⁓
Culture from the first five to the next 10 to the 20 to ⁓ now three, 400 employees, still this idea of like, what are the metrics you're responsible for? How do we improve them? What are your hypotheses to improve them? Let's go test and learn. And like, this is good in theory. And I think ⁓ me in educating and creating that culture within the organization was, it's like a friendly place to make failure. It's like, it's okay to fail because you're gonna fail most of your experiences and most of your ideas. ⁓
So think from the early days, I kind of cascaded some of this learning in. Fast forward 10 years after being acquired now, now I'm exporting a lot of my Neo capabilities, which is how do you help a bigger organization pick up some of this learning and approach to work, right? Measure things, being data driven, having hypotheses. ⁓ So it's been a very full circle moment because now I'm starting to leverage all of our corporate innovation consulting because we were acquired by a bigger corporation. ⁓
So like it was definitely key to our success, measure everything, have hypotheses. And then the two buckets I think is paid marketing, optimizing our, or I guess three buckets, paid marketing, optimizing the website, like CRO as a whole has been a really big piece to our success. And then optimizing the supply chain, logistics, et cetera. So like having these three constant always test and learn machines going definitely is why we were successful. like.

Scott McLeod (12:14.114)
continuing to have leadership deploy that, like, how do we measure it, how do we manage it, how do we improve it, has been ingrained in the company from the early days.

David J Bland (12:23.583)
I love that it's ⁓ part of sort of almost like the principles of the company, how you approach your work. ⁓ And I mean, you told the story of going from five, which like ⁓ from five people, I think it might be easier to have that culture where, you know, we're small, we're scrappy, we're going to go and test things and all that. And ⁓ even when we talk about teams, you know, we try to say, I'll keep the teams kind of small so we can test and go quickly. Going up to 300, even maybe before the acquisition.
⁓ How do you or what worked for you ⁓ as far as keeping that culture in place ⁓ as you scale? Because I've met so many people who really struggle with the scaling part of the company and they realize, we had the test and learn culture, but then we got too big and we could do it and we got away from it. ⁓ Are there any things that really helped you keep it with you as you scaled or anything you can share around that with our listeners?

Scott McLeod (13:21.912)
That was definitely ⁓ a challenge in scaling the organization from revenue and headcount. So I think ⁓ one of the reasons I've held this title as chief of staff, particularly in after the first couple of years of owning growth and the website and some of the early data teams and product teams, et cetera, I kind of owned quite a few of the functions as the generalist. And I transitioned to being a chief of staff. And part of why I transitioned to that role in title is I'm now allowed to be in every room in every meeting in most things. Right. And my goal mostly was how do we continue to.
⁓ Have the organization work and think this way so I was kind of like a cultural agent and like, you know Almost the Trojan horse to do so ⁓ so helping be across key projects and making sure this methodology was there was one just having a true shepherd and that was me and it was instilled by Eric our CEO and a lot of the other co-founders and team but like I kind of continue to beat the drum on data-driven management of the business ⁓ so having a key person me I couldn't help shepherd that but putting myself aside of just like the role of making testing
important. We use this analogy and like
The early days when you're those five to 20 or 20 people were like a bunch of pirates on a pirate ship trying to do whatever we can. And my fear is as you get bigger, you become the Navy where there's like hierarchy and rank and file and everyone has a little job. And this idea of that's not my job was like a toxic thing for us. Like you wanted generalists. So like I learned and we learned the hard way, like hiring stage appropriate people was a key piece of this scaling. You want as many generalists as possible. think I don't even know if it came from the Neo days, like pie shaped and
home-shaped people, right? Like these generalists who have a breadth of experience but go deep in a couple. Having lots of the generalists early allowed us to scale and only now do we have specialists. Unfortunately, there's periods of our business we hired too many specialists early and they weren't able to have this kind of generalist test and learn methodology. So like hiring stage appropriate is really important.

Scott McLeod (15:18.412)
And then making sure you had this like approach. Frankly, there's been many, many iterations. The first need for like these separate pods, we call them, we had little pods to break them out to yours. Like it's easier to create these, well, we have two pizza teams, two burger teams, having a two pizza team, like under 15 people who clearly are accountable through things. We had a big growth rate before COVID where we wanted to continue to launch more brands. So mattresses has been our big namesake, Nectar Sleep, DreamCloud. We continued to launch new mattress brands. Each one of those was
a giant test and learn methodology, right? Like there's a segment in the market. How do we launch this brand to go address that market? And we kind of went segment by segment, right? So ⁓ we had a smart beds. We've had kind of all the different segments attacked nectar, dream cloud, a war, a Sienna. So each one of these takes a different piece of the market. They were all micro experiments. We tested and launched and learned. ⁓ Those are the relatively successful ones. There's a long list of failed brands. We had a bean bag brand, a sofa brand. We had a rugs website. We had lots of these things.
that similar to our approaches is how do we test it with as little as possible, most of which was we can't distract the whole organization from these bets. Like this really portfolio approach to brands. So we'd staff up these three to five person generalist teams and for a while we had ⁓ one team dedicated to launching these things. So that kind of made it easier where those, ⁓ you know. ⁓
the different horizons of innovation. You had the core people focus on Nectar and DreamCloud, which were making our money, but we still wanted to continue to be able to create a bigger portfolio of brands. So we had this learning and testing engine. I say that because it's probably a relevant reason as why we kept that culture across the whole organization. People saw we were testing and learning and it was okay to spend a few million dollars over a year to launch a brand and go, didn't get the indicators we need, move on to the next one.
I'll say and I've said to lots of like as you build this portfolio of bets and approaches of testing, you always run this bet of like the big businesses you have as an organization will continue to get the focus and attention. So you always wrestle with, well, did we give this enough energy? Did we give this enough investment? And I think this is like.

Scott McLeod (17:24.898)
the piece where we kind of refined back and back and back. Like how do you maintain this testing culture? ⁓ It was easier in some ways when you're launching more and more brands, because people could very tangibly see, we're going to launch something. We know what success looks like. If it doesn't hit those metrics, we're going to shutter it and move on to the next one. Now, if you're like less launching big brands, more micro by the teams, it's been something we've kind of been, you know, myself particularly focused on is like, how do we keep that testing and learning methodology there? I think the real answer is as we started, stopped launching so
many brands having a QBR process, a yearly planning process where there's metrics for every team and if every team clearly has metrics the goal then was like how do I know what I'm supposed to work on? How do I influence the business? Everybody from a customer success agent to a developer to someone on the marketing team they should know what metrics are important to them. They should have a dashboard to help them manage their business.
And the impetus for a lot of this was during the COVID era, we were already remote, totally distributed, asynchronous, ⁓ all the things before COVID. And when COVID hit, it just accelerated that value we had as an organization, which is how do we collaborate remotely? If we have data and metrics, it's easy to manage it. And then it's easy to get people thinking through a hypothesis. So if your yearly targets are set out and you have quarterly targets, some of that's incentivized for you to hit. Now everyone's thinking, how do I improve my numbers? Right? So I think early
I was like hack and slash and now it's like incentives, good planning and great metrics, right? Because then you can get everyone aligned and it goes back to creating this culture of a game. How do I influence the metrics in my team?
Just that question alone is a hypothesis and you get the brainstorming. So when we do our yearly planning, ton of stuff to influence metrics, you break that down quarterly by team, everyone's coming up with ideas and hypotheses to affect the metrics that affect the business. And I think that's continued to be the way as a more mature organization, we manage the testing culture is everyone knows what metrics matter to them. They have dashboards to manage it.

Scott McLeod (19:25.536)
Now how do you improve that? And that's a constant thing we're asking on a quarterly and a yearly basis. How do you go improve your numbers? ⁓ Every answer is almost a hypothesis, right? Like nothing's for sure to improve those things. So maybe a little long winded, but it's been a constant kind of ebb and flow of how do we maintain a data driven culture? ⁓ Easier in the early days in some ways, just different now as a more mature organization.

David J Bland (19:52.066)
Yeah, that's, mean, it's a hard question to answer. So I don't think that was long-winded at all. I think a lot of people will learn from your answer there because I have met so many startups in San Francisco that get to a certain head count and it all starts to wobble a bit. You know, it's fine when they're under, let's say 50, but they get to 150, 200 and that culture that they held so dear early on is being changed and they're not equipped with how to
keep that test and learn culture in place. And I'm wondering with you, ⁓ something you said there that really jumped out to me ⁓ is even back in the Neo days, we were saying, we're gonna get our data from our biggest assumptions. We're gonna do that by running experiments. We're gonna have a check-in. ⁓ We're gonna review everything against our strategy. We're gonna pivot either into like a different market or maybe a different channel or.
We're going to pivot our value prop and stick with the same market. But there was something we were going to correct. And then we would say, OK, we're going to persevere or commit to continue to do this, or we're going to kill this or cut it. And it's not something we're going to proceed. I've noticed, as I've kind of learned over the last, I don't know, 10 years or so, that companies have a really hard time killing ideas. It's almost like, you hinted at this earlier, where
Did we give it enough attention? Maybe if we go a little longer, this is going to turn around and we're going to see what we need to see to invest more. How are you handling, from whatever you can share, killing ideas inside the company? Because you mentioned some things that I didn't even realize resident had tried. There's something like the bean bag, for example. And then you kind of said, matter of fact, well, we all know what the threshold is now, but I can't imagine you always knew what that threshold was. So how did you begin to sort of
as a leadership team realize, okay, here's the line we're gonna draw in the sand of how we begin to cut our losses with some of these. And I'm sure you learned so much from those that you applied to other product lines. ⁓ How do you begin that process or how do you stick with it? Because I think it's just a big challenge I find in the teams and execs I coach where they don't really wanna kill anything. They kinda wanna it to linger on a bit and hope that it's gonna turn around.

Scott McLeod (22:15.404)
is a good question. think that like the outside is things become people's babies and like they become attached to the idea and the concept and the vision. ⁓ We often go back to that like fall in love with the problem, not the solution. And if that's your real goal from the beginning as a group and team, ⁓ everyone's an understanding that you're gonna ⁓ have to sunset something. I think being strict and stern with success metrics early and the investment early, ⁓ the easiest one for us that slips is.
Not committing to the investment you need in time and energy and money. So I think like making sure you feel comfortable with what's the real accurate investment you take to test this idea. ⁓ For software that was a little easier sometimes for physical products. It's a little more expensive because we need to order containers and get the products ordered and shipped and there's like this longer runway. And how do you do that in a way but there's still this what's the minimum we need to feel comfortable moving forward. Really that's can we place one order of product and then see if we can sell through it in a profitable or break even way. Our metrics earlier like
How unprofitable are our first thousand sales? Really, it's like that metric, right? And make that gut check. It's like, how big's the TAM? What's our opportunity here?
What do the competitors look like in the landscape? So there's a little bit of like qualitative overlay to the quantitative, which is like, what's the business opportunity? How much should we invest against it? To the killing of ideas. This is where you need to, in the smallest of ways, think demonstrate failure is good. What's worse is to drag on zombie ideas because the opportunity cost for every day you work on that dumb thing is another day you're not working on what could be your home run. And I think instilling that culture in the team is healthy. ⁓
Even today as a mature organization we sunset things that for other people would be whole companies for us billions You know billion plus in revenue ⁓ the threshold now has changed and it continues to change of what success 20 million dollar business isn't really worth our time anymore. That's different than five years ago probably right so I think ⁓ there's a threshold of

Scott McLeod (24:14.164)
What will it be, ⁓ how much will it take for it to be important to the business? I think that's a great. And then what's our chance we can actually get there? I'm always back to that ice scoring of impact and our confidence and effort is like really just quickly doing that on these things. Before you even ⁓ start up internally to the organization or maybe a new feature, really what does success look like and being strong on what ⁓ actually is success? For most people, it needs to move the needle in your revenue at the end of the day, your profitability.
Okay, cool, well what's the plan? 12 months or 24 months to actualize some of that? Then I think you can be really rigorous. At six months in, we need to start to see this much savings or this much revenue and have some call, like some option. For us, we use that yearly and quarterly planning process. We're setting your yearly plan, you're building your revenue targets. We clearly know how much new initiatives are gonna take of that.
You get to June and it's not eaten away at your goal, it's pretty easy to start to make that ripcord decision of this is distracting the organization and it's usually the most expensive thing is the executives ⁓ or frankly, I think that...
high output individuals who are working on these ⁓ high risk, high reward projects, their time can be better deployed to other areas of the organization. So I think a lot of that is like starting rigorously at the beginning of what success look like here. Like honestly, how's this gonna move the business forward? The answer is yeah, it could become 10 % of future revenue or whatever it is. Then you clearly start to know what your investment could be in terms of time and money. And for us, that's it. It's like.
Fortunately or unfortunately as our core business units have grown it becomes harder and harder for things to meet a threshold because like the whole business is you know

Scott McLeod (25:53.046)
It's just so much larger for what we consider success. And I think that's been more of the challenge as your core business units rise. Because early for us is there was always the opinion of being a portfolio of brands had a lot of value. So that's why we have many mattress brands is to take different pieces of the market segment. That's still a giant hypothesis that we're working out. Right. Maybe that is or isn't true when two of our brands ⁓ are behemoths compared to some of the more niche brands. It's still like one of these ones of how do you allocate your time and energy to these smaller things? Sometimes you go back to, well,
This is someone else's whole startup or whole company and we're over here evaluating its success or not, right? So I think ⁓ a ⁓ lot of setting the success criteria up at the beginning, making sure you feel comfortable and having a real time horizon. And for us, it all rolls back to having good yearly and quarterly planning functions where you can have these moments to gut check as a group. I think it helps by having these as different teams. for...
I don't know, four years of the last 10 years when we were launching all these brands, we had a labs team. That was it. They're basically tasked with the innovation projects. They're the ones close to it. We have to make a case to the board every single quarter again, why funding this thing makes sense. So you go to make that QBR deck and you go.
just gut check yourself. Like, I'm not here, my job won't be threatened if this idea fails. And that was a big piece of instilling in the team is what's more important is we fail fast. We fail like quickly and move on to the next idea. So I think it's like, you know.
Having a dedicated group really helps. I think that's not always the answer, but making sure you have clear line of sight to what success could look like and then what kind of time horizon makes sense. I think quarterly, ⁓ setting a plan for the year and checking in every quarter of how are we pacing. And for us, it was easy. It's revenue. If you don't see it addressing our revenue chunk, it's not worth our time. You're constantly comparing to how do you spend your time to generate more revenue.

Scott McLeod (27:47.608)
I think there's another thing you had mentioned there of like, how do you scale into this stuff as an organization? And I think that 50 to 150 employee, whatever that like growth trajectory is, all of a sudden now.
You need leaders and management and you need organ and organization structure. I think that's where we had learned before of not hiring stage appropriate hurts. I think ⁓ hanging onto generalists as long as you can is good. Sometimes the problem there is your early generalists don't become your mid to late stage leaders. ⁓ So this balance is a really ebb and flow. I think of how do you continue to hire generalists? And then how do you, how and when do you start to hire specialists and who are going to be your management and leadership function? That is definitely, I think the risk.
keyest piece there. We went through our own rifts and reorgs and we've taken many processes and steps to get to a place where you have this culture continue to maintain. And I think my learning is try and hire stage appropriate put off expensive specialists as long as possible because generally ⁓ generalists who are startup type people wear many hats and can be adaptable and flexible. I think ⁓
to the main point of testing, they're gonna be less held to a single solution and be open to exploring problems. And I think like, how do you build that culture? It's the right people and the right processes. So people stage appropriate process I think is yearly and quarterly planning where you're really rigorous on what success look like for our experiments.

David J Bland (29:12.243)
I like how you define success criteria up front. ⁓ It sounds trivial, but I still see companies not do that. And then something that I think was really insightful in your answer was about generalists versus specialists. Maybe if you're hyper specialized, then your kind of your perspective is kind of locked in to exactly how this should be tested. Whereas if you're more a generalist, you're more open to different possibilities of testing. I think that's a really interesting insight.
to think about, I had never thought of it that way. One of the ways I tend to think about it is if you only know how to do like landing pages and surveys, you're gonna try to test all your assumptions using landing pages and surveys and there's not always appropriate to use. I also think what you mentioned there, cause you're in mattresses, know? And so a lot of the stuff we worked on was mostly software, you when you and I worked together. And I think a lot of these techniques were sort of pioneered in software and...
Most of my clients now are hardware. ⁓ So ⁓ one of my clients, know, we were, it was paint and we were trying to figure out a, you know, a way to test a new offering. And we literally just spun up a Shopify store and manually shipped ⁓ containers to people. And it wasn't scalable. There's no way we could do this for very long, ⁓ but you know, we were trying to test it to see, there any traction whatsoever here?
And we knew we were gonna spin down that store and offer it ⁓ in the retail stores if it worked. But that stuff, it taps into the creativity of your team. ⁓ Even if it's not software, you can find creative ways to test things at small scale and find out, should we invest in this or more? So I love that you've been sort of playing in this space too. I know you have digital as well ⁓ blended in here, but you look at mattresses and you think, this is a pretty traditional.
industry that hasn't, I don't know, changed too much. I mean, you've seen some innovations, it's the way you make mattresses is the way you make mattresses. ⁓ And so how you kind of test your ideas in that constraint, I find, you know, if you have a room full of generalists and you let them use their creativity, I mean, you're going to find a path forward, ⁓ which I just wonder if ⁓ people don't have the space to do that in some of these companies.

Scott McLeod (31:29.758)
I think it's a fair thing. It's like if you can give the solution.
The problem or the solution. The problem is like, you know, in some ways, how do we grow this unit of revenue? And with the generalist just thinking through what are the most lightweight ways to try that? think you mentioned a good like I talk in broad strokes of testing, which a lot of it's easier on digital because it's easier to measure. It's more instantaneous. ⁓ Millions of website visitors and tens of millions of impressions a week or whatever. It's easy to test ads and websites and this stuff. like we definitely have a whole rigor and science around digital testing that I would say is insane. The particular ad creative testing is like a whole
whole another level of science and not to digress into it too far. It's just like everything's been measured. ⁓ have crazy data analytics around what's in our ads and tagging and figuring out trends and like the most mature science for testing for us is definitely in media mix and ad creative. So those two are like, I would say bar none the largest teams we have for testing ⁓ because we have, we spend so much on media and we need to make sure it's effective. So the most mature testing organizations at our company, have constant flywheels going in these buckets ⁓ are the
digital side. The physical side, you mentioned, well, like the legacy world, the large competitors who are public, you know, they run focus groups 24 months ahead of their launch. So that's their test cycle. Let's bring 50 customers in and have them touch all our beds. That's one approach, right? We don't, A, we never approached, we want a 24 month launch cycle. So it took me a while to get used to like, oh, well, if we want to have an idea, it still takes a few months for it to get here. It's a physical product. So like, how would we help test earlier up funnel?
without having to wait months for something to get made in America or come off of a boat. ⁓ Do a lot of surveys and insights. ⁓

Scott McLeod (33:11.296)
Most of our covers are asking, you know, there's a lot of stuff I think you can still test physical samples and physical products ahead of having to commit to minimum order quantities of things. And I think that's where it's really fun as a physical company. Surveys, small batch samples, working with influencer panels or like your favorite customers, like creating this way in which you can test this stuff before committing to physical inventory. Because unlike software, there's just so much waste if that's a failed bet. ⁓
containers full of stuff, you have to offload it, you pay storage fees, there's like a whole lot there. So we continue, and I'd say being part of the Ashley family, continue to find ways that we can run a data-driven approach to launching products. How do we increase our hit rate and decrease waste is like the high level approach there. ⁓
⁓ There's a lot of ways you can go about doing that without committing to whole containers, right? But even then committing to a full container, if we can run something in six months and test the products, you we'll launch something for a week and ship it to a thousand customers, take it off the site and go on to the next thing, right? And this is like another testing vehicle you can have as an organization. So a lot of it's how do you take this digital approach and bring it to the physical world?
I think there's lots of other cool ideas I've had that we'll continue to get closer to of like, how do you A-B test products and features and this stuff in the physical world? Well, you need to make them and actually ship them and get them to customers' homes. Things like, you if you shipped multiple variations of a product...
and just looked at return rate or cancel rates or customer satisfaction. We're like absolute nerds about NPS. We measure it like 10 times across the user flow. This helps us with some of this testing methodology around customer experience, right? So I'd say like the customer experience becomes this in between the digital and physical world where if you measure everything in all the customer touch points, you can start to have a better read earlier of how do your products perform.

Scott McLeod (35:4.844)
So like there's the legacy way it's focus groups. They still happen. The in between is like how do you get that customer feedback at scale sooner? ⁓
Cause a bed's all about feel. You'd hate to ship a new bed that feels like crap. Now you got two million of those beds coming and customers don't like them. So how do you really get ahead testing the physical experience? ⁓ I think mattresses are unique in that way that like, it's about feel. It's very subjective too. What's firm to you, David, and firm to me are really different or a soft bed, et cetera. So now you have the subjective thing around feel. So it's actually been a really interesting test case in the physical world of how do you like de-risk your assumptions of what's a good bed? ⁓
⁓ What's a good bed look like and what does it feel like becomes a huge piece of like our product development workflow I would say. ⁓

David J Bland (35:52.019)
I like how you mentioned that it's digitizing. ⁓ You mentioned like digitizing the hardware experience or how do you bring that in? ⁓ Also, I mean, if you want to harken back and geek over lean, right? A lot of lean was about eliminating inventory waste in factories, you know, and ⁓ we sort of, lot of us learn lean through kind of the lean startup lens and now we're backing our way ⁓ into more of traditional stuff. But I think at the principal level, it's all still very
relevant, whether you're working in hardware or software. So you've been at Resident for quite some time and sort of built this sort of experimentation engine there at scale. What gets you excited about next year? What's upcoming as far as experimentation or big assumptions you're looking for? How do you use maybe like AI to test things? What's getting you excited right now?

Scott McLeod (36:45.646)
Yeah, it's a good question. So like after our acquisition, ⁓ I've taken a larger role at the Ashley organization. So I get excited working with the largest manufacturer of furniture on the planet in America and worldwide, helping them with this approach to work. Right. So ⁓ I love that when I go to the Ashley office, they're like, you know, Todd and team are exceptional leaders. They appreciate.
Dirty fingernail culture is something where you roll your sleeves up and get into the work. So I appreciate that. And then you apply our testing methodology. Like most of the offices have the Toyota way, the lean leadership book, right? Like it's something that's as manufacturers, they've approached building product this way for a long time. And now you're bringing this to how do you sell the product and market it? I think becomes this, it's exciting opportunity for us as the largest mattress brand online ⁓ to help the largest furniture manufacturer, you know, export some of our capabilities. And we say like,
It's kind of the innovation arm to Ashley in some ways where we're still a smaller startup, know, a billion something in revenue They're much bigger. Their organization is larger. There's more of a global footprint, etc So I get really excited finding ways to export those capabilities Particularly as you're talking point of AI is things are changing so quick So as I look at the future organization structure for creative and product marketing and the tangential teams They probably look really different in 12 to 18 months than they did 12 to 18 months of the past. So I'm excited at what's
what future team look like, what are the roles and responsibilities, and I think we're right in the middle of everything changing. ⁓ So that's really exciting to me to work with at such large scale. Working at Facebook, the largest social network in some way, and now working at Largest Furniture Manufacturer, it's just fun to work at scale and export some of my thinking and capabilities, and ⁓ it's an awesome opportunity to figure out how do we work smarter, not always harder, ⁓ with the world's largest furniture manufacturer. ⁓
It's a really fun new day for me, think. I've continued to absorb more responsibilities at the actual organization, so finding ways that we can become the future of furniture.

David J Bland (38:49.761)
That's amazing. Yeah, it's like again, full circle in a bit, you know, going back into big companies. So I just want to thank you so much for hanging out with me. We started with kind of reminiscing our days in Neo. We could have easily made an entire podcast just on our Neo days. But I really wanted to catch up with you and understand this because I've been watching your resident kind of from the outside and just hearing your story, how you think about things, how you embed this experimentation culture at scale. was just amazing to hear your thought process. I'm sure a bunch of people
people have listened to this and learned a lot from you, ⁓ if they want to reach out to you and they're like, hey, ⁓ I would love to learn more about how Scott thinks about things, what would be the best way for them to get in contact with you?

Scott McLeod (39:32.718)
Maybe through LinkedIn, I guess. I think it's me, Scott McLeod. That's a great question. Yeah, and I'm a pretty active angel investor and advisor to startups. So it's like the other way that as I'm helping at scale, I still like to have my hands working with entrepreneurs and early stage startups, both for my own equity upside, but also because it's fun to work with people who are small, scrappy teams, and it could be high impact for me. So I'm happy to chat with anyone. They can find me on LinkedIn. It's a great place to kind of start.

David J Bland (40:1.857)
So we'll include his link in the description so you all could find that as well. I really enjoyed just hanging out with you, Scott. It was just fun catching up and hearing how you think about things and how you apply them. And I just really appreciate you taking time out of your busy schedule to hang out with us today.

Scott McLeod (40:16.098)
Yeah, it was awesome. Thank you so much, David. It was a lot of fun.