David Dawson walks through a recent incident and response.
David Dawson is a Senior Engineer at Community IT on the escalation team for our help desk. Recently he led the response to a cybersecurity incident at a nonprofit client. In this Community IT podcast, he answers Carolyn’s questions about the flow of the response, best practices, and gives tips on how your nonprofit can be prepared to respond to phishing or hacking attempts. Knowing who to call and how to respond to a cybersecurity incident at a nonprofit can be the difference that makes a quick and complete recovery.
Listen to Podcast
Like podcasts? Find our full archive here or anywhere you listen to podcasts: search Community IT Innovators Nonprofit Technology Topics on Apple, Spotify, Google, Stitcher, Pandora, and more. Or ask your smart speaker.
The takeaways:
- When staff know what to do and who to call it saves valuable time and leads to more confidence in your response. Cybersecurity Awareness Training – particularly anti-phishing training – is a crucial part of your nonprofit cybersecurity defense.
- Having a single point of contact handling the communication at the nonprofit was important both to provide helpful information back to the IT provider quickly and to communicate effectively with 100+ staff that the incident was being resolved and what they needed to do.
- Of course, if your single point of contact is on vacation it can complicate your response. Having an Incident Response Plan with multiple backups will help guide your response.
- If you haven’t reviewed your Incident Response Plan recently, you should! Better yet, gather the stakeholders and hold a tabletop exercise to run through some scenarios and see how your team would handle them. This kind of an exercise doesn’t cost anything to run except your stakeholders’ time, and can help identify single points of failure or areas where the plan is good but your staff need training on what is in it.
- Many nonprofits initially handle their IT management internally. As your nonprofit grows, consider when it becomes appropriate to call on a trusted partner like Community IT to help with cybersecurity, help desk, and strategic planning. Are your cybersecurity investments up to date? What does your cyberliability policy cover? Could you resolve and recover from a cybersecurity attack?
Presenters
David Dawson works as a Senior Network Engineer and serves on the Board of Directors of Community IT Innovators. He builds Windows Server and VMware ESXi networks. David’s past work experience was with restaurants and he joined the IT world in 1996 working in a Network Operations Center for Sprint. He holds the Microsoft Certified System Engineer, Cisco Certified Network Administrator and VMware Certified Professional certifications. David enjoys working with clients that are making a difference in the community and internationally.
When he joined Community IT in May 2001, he felt that his faith, social, and professional lives became more integrated. David studied English at Virginia Commonwealth University and continues to pursue his interest in cooking.
To learn more about David, and about working on a team at Community IT, please see our interview Community IT Voices: David Dawson
Carolyn Woodard is currently head of Marketing and Outreach at Community IT Innovators. She has served many roles at Community IT, from client to project manager to marketing. With over twenty years of experience in the nonprofit world, including as a nonprofit technology project manager and Director of IT at both large and small organizations, Carolyn knows the frustrations and delights of working with technology professionals, accidental techies, executives, and staff to deliver your organization’s mission and keep your IT infrastructure operating. She has a master’s degree in Nonprofit Management from Johns Hopkins University and received her undergraduate degree in English Literature from Williams College.
She was happy to have this podcast conversation with David Dawson on how to respond to a cybersecurity incident at a nonprofit.
Ready to get strategic about your IT?
Community IT has been serving nonprofits exclusively for twenty years. We offer Managed IT support services for nonprofits that want to outsource all or part of their IT support and hosted services. For a fixed monthly fee, we provide unlimited remote and on-site help desk support, proactive network management, and ongoing IT planning from a dedicated team of experts in nonprofit-focused IT. And our clients benefit from our IT Business Managers team who will work with you to plan your IT investments and technology roadmap if you don’t have an in-house IT Director.
We constantly research and evaluate new technology to ensure that you get cutting-edge solutions that are tailored to your organization, using standard industry tech tools that don’t lock you into a single vendor or consultant. And we don’t treat any aspect of nonprofit IT as if it is too complicated for you to understand. When you are worried about productivity, change management, and implementation of new technology, you shouldn’t also have to worry about understanding your provider. You want a partner who understands nonprofits and understands how to respond to a cybersecurity incident at a nonprofit.
We think your IT vendor should be able to explain everything without jargon or lingo. If you can’t understand your IT management strategy to your own satisfaction, keep asking your questions until you find an outsourced IT provider who will partner with you for well-managed IT.
More on our Cybersecurity Services here. More free resources on Cybersecurity for Nonprofits here.
If you’re ready to gain peace of mind about your IT support, let’s talk.
Transcript: How to Respond to a Cybersecurity Incident at a Nonprofit
Caroyn Woodard: Welcome everyone to the Community IT Technology Topics Podcast. I’m your host, Carolyn Woodard, and I’m here today with David Dawson, who’s going to tell us a little bit more about incident response. David, would you like to introduce yourself?
David Dawson: Hi, I’m David Dawson. I’ve been with Community IT for more than 20 years now. I work on the team taking escalated service requests.
Typical Cybersecurity Incident: Phishing Email, Compromised Account
Caroyn Woodard: Can you just walk me through the summary that you gave me before of how the incident unfolded? You said there was someone at the nonprofit who clicked on a phishing e-mail, and it went from there. What happened?
David Dawson: Yes, the first person who, somebody fell for a phishing e-mail, and we all get these things. And in this case, it was very convincing. She clicked on an e-mail which sent her to a sign-in screen that was legitimate, looked legitimate.
But as soon as she signed into it to read this document, her credentials were compromised. Then the attacker immediately logged into her e-mail and sent out hundreds of messages as well. And again, very convincing, looked like e-mails that would be sent from this person.
This person does e-mail a lot of people in the organization, and it’s just a very routine kind of an e-mail. And unfortunately, a lot of people fell for it also because it looked very convincing.
Caroyn Woodard: And then you said they also some of the e-mails went out to external.
David Dawson: Yes, yes, internal and external. We were able to look at how many were sent out. It was close to a thousand.
Caroyn Woodard: Wow.
Wow. Just in the first minute or two after.
David Dawson: Yes, very quickly. And this particular issue was a security incident for one of our clients that has a couple of hundred staff.
Immediate Response: Understanding and Containing the Incident
We got notified about this issue because we had a lot of their staff call us to report the problem.
So a lot of people were calling our help desk saying that they were getting these phishing e-mails from their colleague, and we were responding to them one by one on our help desk by helping people reset their passwords, look at their MFA methods, and some people had already compromised their account, and some people were just reporting problems. But our help desk was really able to absorb the flood of phone calls from them. And then my supervisor asked me to take the lead on it, quickly centralize the effort, rather than having each individual help desk person responding.
Obviously, that was really helpful to have our help desk because I couldn’t call all these people myself. But the flood of calls dampened down after the initial warning, initial alerts. So, I looked at it and we were able to respond pretty quickly.
I was working with the technical person, the person at this organization that was responsible for the technical work. And between me and her, we were able to draw a circle around the damage that had been done, remediate the problems by resetting passwords. And then, I think really crucially, have confidence that we had resolved the problem.
She wanted to know that the accounts, all the accounts that have been compromised have now been, how many accounts have been compromised and have we fully recovered? And we were able to have some confidence around that.
Caroyn Woodard: So, you started getting this flood of calls into our help desk, and because the help desk works together, they realized that they were all getting these calls from the same client, and it was the same issue.
David Dawson: Exactly.
Caroyn Woodard: And you said at that point, they decided to escalate it to you.
Communication is Key
David Dawson: Exactly. Yeah. And she was aware of the, by the time I talked to our point of contact at this organization, she was already aware of the problem, and she had sent out an email, a follow-up email, or the person whose account was initially compromised, they responded by, the password was changed, the credentials were changed, we locked out the person who was, who was attacking her account.
And, and then they found the message in the sent messages folder, and they ended up deleting it, which was actually in this case unfortunate, but that’s a different part of the story. But then they sent out a subsequent email immediately, saying, please don’t open the previous email that I sent, because it is fraudulent. And they sent it, I’m not sure who they sent it to, internally only or externally as well.
Caroyn Woodard: And then your tech point of contact also sent out an email to all staff.
David Dawson: Yes, saying that they were on it. And they also asked me, don’t respond to individual people anymore. It’s just causing a flood of emails and a lot of confusion.
Please communicate just with me. So, it was great for me as a technician, just to not have like 20 people that I need to check in with and confirm that they’re comfortable, or that they had all their questions answered. It was really helpful for me to have somebody internally who’s able to do all that communication and allow me to do just a security investigation.
Security Investigation: Technical Aspects
Caroyn Woodard: And so how does that security investigation unfold? You said that immediately you knew who the compromised account, the initially compromised account was. You were able to change their password, lock out the attacker.
But then what did you do after that?
David Dawson: Yeah. So that was kind of the fun part. We were able to just find the malicious actor.
We ended up using sign-in logs in Microsoft 365, using the sign-in logs and the mail, sent mail logs to see where all these messages were coming from. And we were able to narrow it down to just one IP address. Then we were able to identify the person by this IP address and look at all the activity from there.
Using that, we were able to see how many people’s accounts were also being signed into from that IP address. And that’s when we knew whose accounts were compromised. That’s how we were able to draw a circle around that.
Caroyn Woodard: From the point of view of the staff at this nonprofit, they got the bad email to begin with. Then they got a follow-up email. Don’t click on that.
Then they got an email from the point of contact saying, you know, we’re working with our MSP. We’re resolving the issue. You go about your business. You can use email, et cetera. From their point of view, this seems like it was handled relatively quickly. But I imagine from you and the point of contact’s point of view, there were more tasks that had to be done.
What did you do after that initial problem had been cleaned up?
Investigation: Assessing the Damage
David Dawson: Yeah, that was really complicated. We had to sort of get to a point of confidence. And it really was working with her. It was really helpful.
This organization has a lot of people who, like a lot of our organizations, they’re remote-first kind of an organization. So, we’re looking at the sign-in logs, and I don’t know if this person lives in California, or is that a not legitimate sign-in. I don’t know if this person is signing in from Florida. Is this legitimate or not? So that was important that she was able to determine that.
She was looking at it very quickly with me. Even before I asked a question, she said, okay, this looks good, this is right, this is right, that person is there. That was really helpful. Otherwise, I would have had to chase down every single one of them.
I did a little summary, I ended up just pulling a lot of reports, and I think this really illustrates how well, how important it was to work with somebody inside the organization for so many reasons. I was able to pull reports for her with all the maybe, for me, looked like suspicious sign-ins over the last day, or since this incident began. And she was able to say, yeah, we can dismiss the concern, but with most of these, the only one. We really were able to determine, yes, it’s just this one IP address from this one city, this one state.
This is where all the malicious activity is going from. This isn’t that common because if somebody is working, if somebody is attacking from one IP address, they can very easily switch to another IP address and pivot. So, it was very important to pull report after report, and she was able to look at these and say, yeah, this looks okay.
Caroyn Woodard: You had two essential questions. How much damage had been done and whether it was ended, whether the recovery was complete? How did you go about talking about those two aspects?
David Dawson: It’s all the logs in Microsoft 365. There’s a lot of different kinds of logs. There are sign-in logs that show where people are coming from.
There are activity logs, audit logs which show what is being done. What are SharePoint files being deleted or links being created to sensitive areas, permission is changing, emails being sent out. We are able to review all of that as well, to see that there is no more malicious activity going on.
And then we’re able to look at even something as specific as multi-factor authentication devices. If somebody’s password is compromised, very often these attackers will immediately add their own phone as a multi-factor authentication. So even if you kick them out, even if you change the passwords, they can find some way to reset the password later to get back in.
We pulled all these kinds of identification information, sign-in information, activity information, and analyzed it, but it’s really by IP address that was helpful. We were able to determine that there’s very little malicious activity except for this email blast. And for each individual person, we’re able to determine that we’ve removed all of the recently added MFA devices, we’ve reset passwords, and we knew that specific people who were impacted, and fortunately in our case, it turned out to be less than 10 people. Responding with person-by-person response was not completely difficult. Yeah.
The Value of Security Awareness Training
Caroyn Woodard: And I would think that that is a plug for security awareness training. You said there were hundreds of staff, but most of them knew not to click on it.
David Dawson: Yes. And I will say, yeah, I wanted to bring that up at some point too. I had really great conversations with our point of contact, and I said, yeah, let’s talk about all the things that we could have in place to prevent this next time.
And she said, okay, yeah, that’s a great idea, so we can do some review. And we already do a security awareness training. I think our staff is very aware of how not to fall for these kinds of things.
Caroyn Woodard: And that’s a great opportunity to give your staff kudos. Exactly. Good job.
Security training is working. It’s worth taking those little quizzes or responding to the watching the videos that we recommend. So that’s really, that’s good to hear.
Of course, we always talk about not doing security training in a way that shames anyone. I mean, anyone can fall for it. And like you said, it was very convincing. And with AI, it’s getting more and more convincing. I think also like that second part of the security awareness training of if everybody, it’s very possible that you will click on something that you shouldn’t click on. What do you do next?
Who do you tell and tell them right away?
David Dawson: It’s great to bring that up too, that the non-shaming part. We ended up having a great conversation about, all it takes is a convincing email, and you’re tired, and you’ve been making decisions all day long, and it’s very easy to click on something. And it comes in email, but it also comes all over the place too.
She was telling me about selling something on Facebook, and it was a very convoluted thing that she fell for, and she still thinks about it 10 years later.
Caroyn Woodard: Yeah, just the different scams that are out there on our different devices and tools that we use.
Post Incident: Next Steps and Documentation
So, then you were able to, you said you printed out a whole bunch of reports showing that of those 10 or so accounts that were compromised, logins, their password had been changed, and did you have to have them set up, reset up their MFA for all those accounts?
David Dawson: Yes, we often do for just sort of a matter of course. In this case, I don’t think in this particular case, we found any MFA devices had been added, but we were able to scrutinize that and just confirm that there were no other MFA devices.
Caroyn Woodard: Then she could have a lot of confidence that the incident was over.
David Dawson: Yes.
Caroyn Woodard: And you could too. As the MSP, cybersecurity provider, we want to know the bad guys are kicked out.
David Dawson: Absolutely. I think using our processes that we’ve been developing over years, that’s where we really want to get to. We’ve done the work, and we know that this is the result.
We’ve kicked out the malicious actors and we’ve locked the doors so they can’t get back in. And we know what they’ve done while they were inside. Yeah.
Nonprofit Cybersecurity Incident Lessons Learned
Caroyn Woodard: Those are all just super important to have. And then it seems like you added on another step of how do we use this incident as a learning teaching moment to maybe put some more safeguards in place for that organization to prevent something like this from happening again, knowing that cybersecurity risks are always evolving. It may not be this exact one next time, but we hope if this exact one comes in, that we’ll know, we’ll have a good response to it.
David Dawson: Yeah. That’s always the question about what else could we have done. And I think in this case, I’m very glad to say that they had everything in place to be able to respond very quickly.
What can you do better? There’s always something else. There’s always something else.
And so, we set up additional notifications. We would have been, we would have gotten a notification. But honestly, they were calling us just as fast as, we would have gotten, seen the notification anyway.
Caroyn Woodard: And that’s another plug for the cybersecurity awareness training.
David Dawson: Yes. And I’m really glad that they knew to call us right away. They had, they knew what to do. They had our number handy, and the calls came flooding in. So, I think actually that was a good thing. Yeah.
Caroyn Woodard: It’s one of those things where, well, it seems like a negative that someone fell for this, but actually, it’s a very positive thing that the training worked. And that we, as an MSP, were able also to, our techniques and tools and methods could work.
I guess I want to put in a final plug for having an incident response plan.
I don’t know at this client how, whether they were relying mostly on the security awareness training, or if they did have this security, you know, an incident response plan to fall back on.
But we have a lot of resources on our site to create an incident response plan. If you don’t have one, if you have one, but it’s just sitting somewhere in a file and you haven’t reviewed it in the past year, we would recommend you should review those annually.
Because as you’re saying, David, like, there are new threats that are coming in, new incidents that you’re responding to. So, you want to make sure that that Incident Response Plan is an evolving living document that your executive team has eyes on. And you just, you know, go over and think about it.
There’s another plug I can give for the webinar that we just did on the tabletop cybersecurity exercises, where you can actually take that Incident Response Plan out and run through some scenarios with the stakeholders and see, well, in this case, we’re supposed to call this person. What if they’re on vacation? Who else would we call?
So, that’s a good document to have for a lot of cybersecurity insurance, liability insurance. You’re going to need to have an Incident Response Plan policy as part of your policy. So, we do have a lot of resources on our site about how to get that going if you don’t have one.
But it sounds like this client, like, knowing what to do in the emergency is really just gets you that one step closer to fixing it faster than trying to search down, like, oh, who are we supposed to call and what are we supposed to do?
Collaboration, Communication, Continuous Improvement
David Dawson: I think maybe for me, just a couple more things too. It was really great working with somebody who was able to ask all the questions, and I was able to provide it from a technical perspective. So, I think that actually having two heads there was very important.
She asked a lot of questions that I would not have thought of asking. And when I provided those to her, she got a lot of confidence and then I got more confidence because she’s providing that communication back, that she’s talking to her staff and that they are confident as well. I think that’s good.
And then as far as the Incident Response Plan, there’s, or any kind of a security plan, I think there’s maybe paralysis because people want like sort of the best one or the perfect one. And then, you know, you can, there’s always good, better and best. And, you know, there’s always something else that you could be doing.
Things are changing all the time as well. Having something in place, having someone who’s able to communicate risks and have questions in plain English is really helpful, because, you know, we love to do that, the technical, translating that into technical plans as well. So, I think having that communication with us, you know, this is exactly what we like to talk about as well.
We love it when people are asking these questions, because we want to have that conversation.
Caroyn Woodard: Yeah, and I would say we talk a lot about you do not have to be a technical person with a strong technical background to manage nonprofit IT. Someone at your nonprofit needs to own IT, and that person, if they don’t have a technical background, you need to ask those questions, and ask them in plain language, and your MSP or IT provider or your IT director, whoever it is, should be able to translate and provide the reports and talk about them in a way that your questions get answered.
And if they aren’t answered or you don’t understand something, or you’re not sure that the incident is completely closed, just keep asking those questions until you get the answers that you need to have and to be able to understand, for sure.
David Dawson: Exactly.
Communicating Externally About an Incident
Caroyn Woodard: Well, I have one more question, which is, in this case, it was emails. And so, there was the potential for the people who externally received those emails also to have had their accounts compromised by opening that document and clicking on the link in the document. And it was very convincing, as you said.
But sometimes, an incident could be, you know, there was a data breach. Somebody got some of your files, or once they were able to log in to your systems, you know, they changed some of the passwords and gave themselves access to, like, your HR files or your financial files, those sorts of things, or data on your constituents or your donors. So, how important is it?
I know you’re the technical side of responding to an incident, but it seems like that also needs to be covered in your response plan. Like, if you have a data breach, what is your policy going to be for, you know, letting people know that their data was compromised or that they, you know, the email that they received was fraudulent? So, it seems like that also just needs to be covered in your policy.
David Dawson: That makes a lot of sense, yeah. You have to consider that unfortunate fact. And I imagine cyber insurance processes, they would maybe cover any damages there, or at the very least, they would be able to provide some sort of a roadmap for how to consider what to do in that case.
Caroyn Woodard: Yeah. And I think, you know, we’ve talked on other webinars and podcasts about, you should be pretty familiar with that policy, and your insurer is going to have a lot of resources, too, such as when to talk to the FBI, you know, when to, you know, follow up, when you have to report something. And also, they just have resources on how to resolve the incident, also, so you can check in with them about those next steps, also.
So, all good information. Thank you so much, David, for sharing this incident with us without naming names. And I think that it’s just so helpful to hear from peers about, you know, what happens when something like this happens.
And I love that you gave us the flow of like, well, we did this first, then we did this, then we were able to create these reports. So, you know, I’m glad, I’m sure that it was very helpful to your point of contact to have you on the other end of the phone, too, being reassuring and finding out what you created quickly and handling the incident from our end. So, thank you so much for sharing this with us today.
David Dawson: You’re very welcome.
Photo by Kaffeebart on Unsplash