Office Hours Ep #2: What Are Best Practices When Dealing With Duplicates In SFDC?

Home     /     Podcasts     /     Office Hours Ep #2: What Are Best Practices When Dealing With Duplicates In SFDC?

  • Today's topic: my org has a lot of problems with duplicates. What are some best practices to tackle this issue? [00:00]
  • What does it mean to be a duplicate contact, lead or person? [02:12]
  • How would you recommend structuring accounts for companies that own multiple companies? [04:00]
  • Do you use anything else besides websites as a unique identifier? [06:00]
  • If you have a person that has changed companies, should you create a brand new contact or move the contact into a new account? [10:56]
  • The idea of being a fresh contact, what do you mean by that? [13:44]
  • How would you clean everything up after stopping the influx of duplicates? [16:43]
  • Does your team go through the UI or do you use tools for mass merging? [18:17]
  • Do you have tools that you use to merge the information? [21:04]
  • Does your script auto converge and merge or does it do quarterly batch duplicate catches? [21:40]
  • What types of rules or processes would you put in place to prevent the creation of duplicates? [23:08]

What are some best practices when dealing with duplicates in SFDC

Kyle: Hi Greg. Thanks for being on. The topic today that we got was: my org has a lot of problems with duplicates. What are some best practices to tackle this issue? This came in anonymously so we're not going to shame anybody for this. How have you seen duplicates come up? What are the best practices you recommend for some of your customers?

Greg: So I think to start with is how do duplicates get created? I would say there are generally two main paths that cause it. The first is when you link your product or website or product database with Salesforce if you have leads automatically created. Number two is when you give reps the ability to, through like a discover org lead enrichment tool, the ability to mass important in Salesorce. Also on the accounting side is when you link oftentimes you're not really checking if that account already exists so a new account gets created. Does that kind of summarise your understanding too why duplicates get created?

Kyle: You mention the first one is like leads are coming up through the website. Other option reps are just mass importing leads from various tools. It's definitely what I've seen as well. My issue is that most folks that sign up to a website, they're gonna run through a marketo or hubspot or something like that, so should be de-duped against existing records. But are you saying like leads will come in as a new inbound lead and then eventually someone else will import the same lead from ZoomInfo or DiscoverOrg or something like that and that's how a dupe is created? Or do you find on the creation of inbounds that you see those duplicates being created?

Greg: I think Marketo? and those tools are generally pretty effective preventing dupes on the person side because they're just direct matching off the 3-D mail. But when you convert those leads, you end up with a ton of duplicate accounts.

Kyle: How do you define a duplicate because this is like a critical thing that I think a lot of folks don't do is they don't step back and say: a duplicate record means x. So we can separate it from companies and people, we can maybe talk about both. What does it mean to be a duplicate contact, lead, or person? What do you think of that?

What does it mean to be a duplicate contact, lead, or person?

Kyle: I think on the person's side it's a bit easier because it's literally the same person. I know some companies will sell to consultants. You'll have a contact for the consultant but also have an email for the company they work with. It's pretty straight forward it's the same human being you're trying to reach out to on the contact side.

On the accounts side, I think of duplicate as the same potential customer. If you're selling to Intel and there are multiple divisions within Intel all who have different buying processes, all which could be customers, I don't necessarily call that duplicates. I would want to parent accounts, that hierarchy which would present that but I don't necessarily view that as a duplicate but we can have a smaller company or there's an instance where there are multiple accounts but they cannot all be the customers, they all funnel into the same contract, that I would consider a duplicate.

Kyle: Yes that's interesting. I'm thinking about on the accounts side. One big problem. So my background is primarily B2B Enterprise Sales, so I don't have experience in transaction sales personally, mostly around big ones. And I can remember Disney being a massive challenge because it owns the Walt Disney Studio's, who owns ABC, who owns ESPN, who owns ESPN etc. In some instances that could be an account that you can consult to individually like for example selling brooms for an office. Or if you are selling one product like furniture to Disney you want to sell to everyone at once. So like if a company came to you with that problem, saying: hey, we've got Disney. (It's one of the most complex companies you'll find). How would you recommend structuring it? If it's based on if I can sell to a company it should be one account or if it's got the same address it's considered an account?

How would you recommend structuring accounts for companies that own multiple companies?

Greg: The way I think about it is, can I sell to this team multiple times? And if you did do that, what would the management structure on your side look? Meaning, if you start selling to Disney, would have a European rep handle your Disney Europe and a US rep handle the US? So then I would split that out. Would you have Disney as a subsidiary on the east coast but Disney's headquarters on the west coast? Would you have different reps handling that? So I think the definition of a duplicate is driven by your definition for your rules of engagement.

Kyle: Yes, the way I think about it is: if this account could buy independent of any other account it should be its own account. ESPN can probably buy 90% of the things they buy independent of ABC. But if you've got something that could only be sold to ABC, and ESPN couldn't buy it, you would want a separate account. Would you agree with that?

Greg: Correct. It's just if you don't separate it up, it just gets unnecessarily complicated. I was talking to a company who's tabloid was acquired by Salesforce. So they were like should the tabloid sales rep lose that account and we just give it to the sales corporate person? At that point, no because tabloid is going to operate as its own company, they're not going to merge everything tomorrow. Same thing with a lot of companies Google fires. Should Nest rep lose that account or should go to Google rep? That's not a good use of your time and I don't think that's your best sale. Because then you end up with one rep who's only account is Google.

Kyle: What I typically use to identify duplicates for accounts are websites because that is the most unique thing you can have, right? There's only on but you can get 10 Intel companies within the US or the world. So websites are typically easy to use, right? Is there anything else you use as a unique identifier that you use consistently?

Do you use anything else besides websites as a unique identifier?

Greg: I think websites is what I often use. Some companies are in industries where there might be identification numbers or especially anyone who sells to government-regulated industries will have numbers for all those things. Websites are your best bet. You could use like in the case of Intel US and Intel UK you'll have different websites for that so you'll use that for the unique thing. So I generally prefer to make the website the unique identifier, so you can't have two accounts with the same website. What are your thoughts on that?

Kyle: I'm a big believer as well and you shouldn't have any forward slashes either. So it should be and not or EU or UK. That's my gut because really their group domain should be the account you sell to. I'm trying to think of a good example of, like if you go to Walt Disney Studio's it may redirect you to let's say. Then you create this weird situation where they're kind of one company but they're also separate. I think what I recommend companies to do is understand how the company you're selling to is structured and realize that there may be situations where you have to break your rule but that should be a conscious decision rather than it happening all the time or happening never and you not realize it. Some companies may have a data team or an operations person who's ensuring that accounts are in Salesforce on purpose not accidentally duplicated.

It's hard to handle some of those more complex enterprises and it's really hard for transactional SMB companies to handle, what about companies that don't have websites? Like what if you sell to people that come clean your home, that person might just have a Gmail email address that you send to, so they might not have a website. So that creates its own host of challenges of companies and people that are independent. I don't think there's necessarily a slam dunk solution because not every company has its own unique identifiers but I know that companies are going to be spending more time on this. And where I typically see them make the mistake is, let's say we have identified a problem with duplicates. We know how we want it to look like and this is how we classify companies as being separate, they immediately start to clean things up. They buy a tool that will de-dupe. They'll hire a team overseas to start cleaning records up or do data enrichment but they haven't actually sorted out the root problem for the source of duplicates and the analogy that I use is like imagine SRM is a lake that is polluted with a river dumping pollution into it. Before you start to filter the water, you should probably stop the pollution coming from the stream before you filter the water. Once you stop the inflow of duplicates, then you start to clean it up. Figure out the problem. Stop it. And then start to clean those things up even though it feels like those things are going to take longer.

Greg: I totally agree. It's a bummer. Lots of time there is this pressure to show progress. If you want to stop the river from polluting your system, you need engineerings' help cause a lot of that is coming from your website. And that's going to take a long time and people don't want to feel like we've been talking about this for a month now and we are not making progress. It's definitely an area of stress. But I would echo you that you need to solve the core issue before you go and fix.

Kyle: Yes, a lot of companies know they have a problem with duplicates. They know they're hurting their sales team. And they feel it should be solved right away. Like, why is it taking so long? We should buy a tool. It will merge all the duplicates and it will be done. There is one Warren Buffet quote: You can't create a baby in one month by getting 9 women pregnant, which I really like. There are certain things that just take a certain amount of time. Like engineers love to say it will be done when it's done. And the reality is sometimes things just literally takes time and it can be a drag on your team but for the most part, some companies have been dealing with it for a long time and they haven't had a solution for it they just got frustrated with it.

For people, it's kind of a different thing because you have situations where, let's say, you've got a contact for Greg at KeepTruckin. Now he works at Clarus. How do you square that? Should a company be creating a brand new contact? Or should they be moving the contact into the new account from your perspective?

If you have a person that has changed companies, should you create a brand new contact or move the contact into a new account?

Greg: Good question. Having a contact status which is: "no longer at company", so mark that for the old me and then create a new contact for my new area. Because I don't like the idea. From a marketing perspective, it's a terrible idea and also from a sales perspective. I don't like the idea of if you move the contact over to a new account it's going to look like all that activity happened on the account.

Kyle: I can't agree with you more. Because a contact in Salesforce represents a person. That person is a single individual, not a person in a role in a company. So your unique identifier is not your email address, it should be your LinkedIn profile (assuming the person has a LinkedIn profile).

Greg: Okay. But you still have this problem where you're bringing over all this activity, yeah?

Kyle: True, but you've had that conversation with Greg in the past so if Greg moves to Clarus and he comes as an in-bound do I want to know that I've never talked to Greg at Clarus or that I spoke to him three times at Admiral and then twice at KeepTruckin? I would think you would want that intel for your reps to be able to have that historical information cause you could be tied to close a deal but they wouldn't have a concept of that if you create a brand new one.

Greg: Interesting.

Kyle: Challenge it. Cause that's been my thesis for a long time. Some people have pushed back saying, well, if I move Greg from the KeepTruckin account to the Clarus one then all of his activity will flow with it and then I lose all the information under the KeepTruckin account. Which is a super valid concern cause it could create some friction for companies.

Greg: So when you move the contact, the ID on the tasks won't change but it's like super messy because, on the new account, that activity won't show up on that account. It will only show up on the old account but it will show up on the old/new person.

Kyle: Yes. It kind of makes sense. If I go looking for a contact for Greg, I would want to know that you were at Admiral and you were a customer and you were at KeepTruckin and you were not and now you're at Clarus Designs and you're not. That feels like a history that a sales rep would want to know.

Greg: I'm not in sales so I can't totally answer that but I think I would prefer to have a fresh start. Like if I move companies, I would want you to treat me as if I'm a new person who has not heard of this company.

Kyle: It feels like a possible solution you use is related contacts in Salesforce? Greg can be under the Clarus Designs account, so your contact would move. But you can have him or her as an account contact relation as the name of the object tied to KeepTruckin into Admiral and any other company that you've worked within the past. So you can see that he was here but is no longer. That's maybe another solution but that doesn't solve the problem of like is this person fresh or not? That idea of being fresh. What does that mean to you? There's a meaning behind that. What is it that you solving for?

The idea of being a fresh contact, what do you mean by that?

Greg: I think what I'm solving for is if you're kind of running an operational machine with your SDR Sales Team, Marketing is looking at how long has it been since last touch. Marketing is looking at that so they don't re-receive that and I think for me if they're switching companies they could be a totally different buyer now. The new company might have a very different news case than the old company for your product. In which case, I wouldn't want to say don't send this person this collateral because they would receive something similar or they might have received something different.

Kyle: Right because you were a VP of Operations at one company and now you're a CO of another. Those are different personas. You might want to use different messaging and different collateral when you're messaging to a person with a new role, right?

Greg: Correct. Yeah.

Kyle: That said, wouldn't you want if contact Greg has been in a specific campaign. Like, say we're busy with some gimmicky campaign, where we're gonna send someone a bottle of Muller or something like that. If Greg got that at KeepTruckin I would want to know that so that I don't do that exact same thing at Clarus because he has already received that. And if you create a new contact you'll lose that understanding of their historical campaigns, right?

Greg: Yeah. I think that's fair. I might be leaning towards the idea of just moving the contact. It is so annoying the history gets taken off the previous account, like on the contact side but sounds, either way, there's going to be some annoyance here. But I can get on board with your argument. I think that's probably a stronger argument.

Kyle: I'm trying to think of other situations where it maybe wouldn't apply. Maybe highly, highly transactional companies where you have a person who just signed up. They paid with a credit card and swiped and they go straight into your CRM as a brand new lead, converted to a contact. That feels like a situation where you don't move the contact you just convert them. That's a situation that I think if it's highly transactional but I think my gut is that, and again my bias is always about Enterprise Sales cause that's what my experience has been. Where you have a pretty finite world and not a ton of people you're selling to so it's worth moving a person around. But if you're selling to a drifter you don't have to go through all that.

Greg: Correct. I think in general the more transactional your sale is, the less it really matters. You can delete the contact for all I care. Because reps do not have time to look through the history.

Kyle: Okay so we've come to a rough DMZ on leads and contacts and kind of an agreement on accounts. Let's say you have a customer that calls you tomorrow and says we know we have a bunch of leads we've stopped the influx of duplicates, now we've got to clean them up. I have a way that I tackle this. How would you tackle it or how have you done it in the past? Of merging and combining everything?

How would you clean everything up after stopping the influx of duplicates?

Greg: We've written some code to just auto-merge leads based on website or email or phone number. Things like that. Generally what we'll do is we'll take the first pass and merge the low hanging fruit. Then for everything that's left, we'll try to use a rules-based model to determine like which ones are garbage so we can just ignore those. And then the ones that remain we'll have our team go through.

Kyle: So actually have people go through and manually make merges of whatever remaining records are there.

Greg: Correct we have a data quality team that does that for companies.

Kyle: Yes that's kind of nice. But that's hard because customers have to explain to you like I want to prioritize lead sources and then after that, it's who's been contacted and then it's what kind of campaign are they in. A lot of times they'll come up with really complex rules for this. Do you typically push back on that or do you say great we'll support all of those things and we should take those things into consideration or is it overkill?

Greg: I think in general it does not matter what rules you choose. We've done this so many times that we have a rule set that we'll present and say this is what we usually do. This works well. Are you okay with us doing this? Most people don't want to think about it so it's like, yeah Greg it sounds great. Sometimes we'll have to make some changes on custom changes that they really value but most of the time it's a pretty easy conversation.

Kyle: And do you typically have your team go through the UI of Salesforce? Or do you use any tools for merging in mass?

Does your team go through the UI or do you use tools for mass merging?

Greg: We'll use tools if it is a kind of a straight forward merge. We'll go through it if there's a case of more than two dupes or if the company is a bit nervous about losing stuff or if it's a complicated ruleset.

Kyle: Yeah, the way I typically tackle this is, one of the first things I'd kind of recommend to companies is try to delete a bunch of records. Cause often times your CRM has a lot of people you don't need to begin with. So you might as well delete some portion of those if they're no longer there, they've never had an activity and you don't really care about them. Just wipe them out cause they don't really matter. That often times will clean up a bunch of duplicates. Then from there, I'll typically do an export of all leads and contacts and I will do a pivot.

So basically there are 3 ways I de-dupe. I have lead to lead. Contact to contact. And then lead to contact. And that's kind of the order that I take it. So I export all leads and I'll pivot on the email address or back-up email address. And I try to create groups of email addresses., if there were three leads with that then that's group 1. Then group 2 will be And then essentially I do a batch update of all of the leads so that they're in groups. It's just a custom field. I'll call it duplicate group or something like that and then do a re-import so I can run a report in Salesforce of all leads that have a duplicate group.

Then I'll do the exact same thing for contacts but I'll make the email addresses identical. So the group ones will be but I'll do that for contacts as well. So when I do custom scripts like the one you are talking about where we are going to merge, we essentially look through every group, and we'll merge them based on whatever one we want to be the master. Typically our customers are quite lax where they'll say, look we want the one that was created first to be the master and then if there's more than one then look at the one that has had the most recent activity. And then we merge those into groups so that they show one more lead under group 1, one lead under group 2. Then I do the exact same process for contacts. Then we merge those together where we have a script that will convert the lead and then merge it again into the contact. I think you can do this because in a script when you're converting the lead you can convert into a matching contact and because we have already done that V-lookup match it does that for us. So that's my process.

So we covered how to think about accounts. How to think about leads and contacts. How to try mass do this. I personally haven't used any tools. Do you actually have tools that you like merging this stuff? They always feel like they're pretty heavy and I don't always trust them.

Do you have tools that you use to merge the information?

Greg: Clouddingo is the one that I've seen that have the best results. But I'm kind of in the boat where if you mess up a merge the consequences as a consultant is so heavy that I would rather trust the code that we have versus a third-party tool.

Kyle: So you say you have a script that will auto converge and merge or do you run that as a batch process on a quarterly basis? Like an ongoing catch for duplicates or how do you do that?

Does your script auto converge and merge or does it do quarterly batch duplicate catches?

Greg: It depends on if we have our data quality team we'll do it on a daily basis. Otherwise, we will send a monthly report to the sales ops team we work with letting them know which ones they should look at.

Kyle: Oh, so you don't just send them like here are ten batches of leads that we think are leads among duplicates or whatever and then you just let them handle it?

Greg: We just give them the list so they're aware of it and they can go through it if they want. And then either we or they will handle it but we used to kind of merge it but we had this happen once were Marketing were doing some tests where they were treating some current leads, they wanted to purposefully create duplicates so they could market to them differently. So dumb. But anyway so we merged all their duplicates and messed up their project and they got pretty angry so since then we are more careful about merging.

Kyle: So if Clarus was ten sales reps and you were heavy outbound. Cause I know most of your business comes from referrals and word of mouth cause everyone knows about you. What types of rules or processes would you put in place to prevent the creation of duplicates? Would you just use Salesforce as a duplicate prevention tools. Do you want your team to check every week or month. How would you do this for your company if you can do whatever you wanted instead of having any restrictions from a company?

What types of rules or processes would you put in place to prevent the creation of duplicates?

Greg: So on the lead contact side I would make email unique and on the account side I would make website unique. And then what I would do is have some way of identifying duplicates. I would run that periodically. Also, I would look at the duplicates that do get caught, I would look at who's creating them. It's usually 5% or 10% of your sales team that create your dupes. And try to address that through like training.

Kyle: I have a different perspective. If you have 50 sales reps it's too hard to keep them all trained and up to date and they don't really care about duplicates the way we do. My thought is you should create it so that they can never create a duplicate lead or contact. Like, if that email already exist, you can't create it. Barring any rules that allow for it.

Greg: I would definitely do that if you have unique fields for website and email. Are you saying if it passes those filters also have somebody check to make sure that it's unique or what are you thinking there?

Kyle: No my thought is, let's say I'm a rep and I found a lead and I'm going to add it to Salesforce and I type in the domain but I mistype it so instead of I type in (we already have an IBM account). We should catch that if the rep every changes the website to That should be flagged as like, 1) this is a duplicate you cannot do that. You can't have two accounts with the same domain barring any exceptions that we've got. Or maybe a systems admin or an operations person should be able to allow for that. And we need notification to prevent reps from inserting those or if they want to create a contact and that contact already exists in the system it should be blocked.

Greg: I'm with you. So I think we're on the same page there. Making those fields unique will prevent people from creating duplicates.

Kyle: Looks like we covered everything we wanted to cover for dupes.

Greg: Yeah. Send us an email if there are other topics regarding dupes you want us to cover.

Kyle: and

Greg: Thanks.

Kyle: Rock and roll. Talk soon, man.

Recent Kaizen Podcast Episodes