tv Senate Intelligence Hearing on Solar Winds Hacking CSPAN February 28, 2021 1:55am-3:45am EST
extension of unemployment benefits, and a hike in the federal minimum wage. after the vote mostly partyline, the measure now heads to the senate. the house is back for legislative business on monday. and when the senate returns, they'll continue to work on cabinet nominees, with votes on mcgill cardona to be education secretary and a calmer secretary. as early as tuesday, the senate may take up the covid relief package. it would need a simple majority to pass under reconciliation rules, but the senate parliamentarians ruled the minimum wage provision is not allowed under senate rules. watch the senate live on c-span2 and the house on c-span. >> tech company executives testified about last year's cyber security breach that left several federal agencies and businesses vulnerable to hacking. this senate intelligence committee hearing focused on how the cyberattack occurred, why it took so long to detect, and what can be done to prevent a similar situation in the future. this portion is an hour and 45 minutes.
careman to thank you for your partnership and friendship. i'm confident we'll be able to keep working together in a bipartisan way in the 117th congress. i'd also very much like to welcome our witnesses today. the president and ceo of solar winds, brad smith president of microsoft, and i believe remotely george curts, president and ceo of crowd strike. i would like the record to note also we asked the representative from amazon web services to join us today, but unfortunately they declined. but we will be expecting to get a full update. we've had one update from our friends at amazon, but it would be most helpful if in the future they actually attended these hearings. today's hearing is on the widespread compromise of public and private computer networks in the united states by a foreign adversary. commonly called the solar winds
hack. whilemost infections appear to have been caused by a trojanized up take of solar winds orion software, further investigations are revealed additional victims who have not used solar wind tools. it's become clear there's much more to learn about these incidents, its causes, its scope, its scale and where we go from here. this is the second hearing this committee has held on this topic. the first was the closed hearing on the now infamous january 6th with government officials responding to the incident. it's going to take the combined power of both the public and private sector to understand and respond to what happened. preliminary indications subject that the scope and scale of this incident are beyond any that we've confronted as a nation and its implications are significant. even though what we've seen so far indicates this was carried
out an espionage campaign targeting more than a hundred government agencies. the reality is the hackers responsible for gained access to thousands of companies and the ability to carry out far more destructive operations if they want them to. and i want to repeat that. this intrusion had the possibility of being exponentially worse than what has come to pass so far. the foot holds these hackers gained into private networks including some of the world's largest i.t. vendors may provide opportunities for future intrusions for years to come. one of the reasons the solar winds hack has been especially concerning it was not detected by the multibillion dollar u.s. government cyber security enterprise or anyone else until the private security firm fire eye, and i want to again compliment my friend who appeared before this committee a
number of times, on their own without the requirement to report actually announced it detected a breach of its own network by a nation state intruder. a very big question looming in my mind is had fireeye not detected this compromise in december and chosen on their own to come forward would we still be in the dark today? as deputy national security advisor anne newberger who's been chosen by the president to heed the response to this solar winds hack said last week, the response to this incident from both the public and private sector is going to take a long time. all of the witnesses today are involved in some aspect of the private sector response to this incident. i want to hear from them on the progress so far, the challenges we'll need to overcome these hackers and how we can prevent supply chain attacks like this in the future. i'd also like to hear from them about their experiences working with the federal government,
namely the unified coordination group in mitigating this compromise. the solar winds hack was a sophisticated and multifaceted operation. a software supply chain operation that took advantage of trusted relationships with software providers in order to break into literally thousands of entities. combined with the use of this sophisticated authentication exploits it also leveraged vulnerabilities and major authentication protocol. basically granting the intruder the keys to the kingdom, allowing them to deftly move across both on premises and cloud-based services all while avoiding detection. while many aspects of this compromise are unique, the solar winds hack has also highlighted a number of lingering issues that we've ignored for too long. this presents us an opportunity for reflection and action. a lot of people are offering solutions including manbatory
reporting requirements, wider use of multifactor authentication, requiring a soft bear bill of goods and significantly approving threat information sharing between the government and private sector. i've got a number of questions, but there are three i'd like to pose in my opening. one, why shouldn't we have mandatory reporting systems? even if those reporting systems require some liability protection so we can better understand and better mitigate future attacks. as i've pointed out senator collins was way ahead of of us on this issue literally years and years ago when she and senator leiberman first put forward legislation that required reporting on critical infrastructure. there's an open question, though, who should receive this report even if you put that mandatory reporting in place. do we need something like the national transportation safety board or other public/private entity that can immediate
examine major breaches and see if we have a major problem as we seem to in this case? i think there's some truth if a foreign nation state sends their a-team against any ordinary company in the world, chances are they're going to get in. but that cannot be an excuse for doing nothing to build defenses and making it harder for them to be successful once inside an enterprise. i'm very interested in hearing from the witnesses what they think our policy response should be and what solutions they think will actually approve cyber security and incident report in the united states. beyond the immediate aspects of the solar wind hack are larger issues this committee feeds to consider. do we ned to finally come to some agreement on common norms in cyberspace? hopefully again on an
international basis that potentially enforceable and these says to our advarsaries if you violate these norms there will be consequence. we have military conflict that exists but there's been for some time a norm you don't knowingly bomb a hospital or bomb an ambulance that got a red cross shield on it. should we therefore consider efforts that subvert patching, which are all about fixing vulnerabilities, to be similarly off-limits? once again i want to thank our witnesses for joining us today both in person and remotely. i personally talked to nearly all of our witnesses, in some cases multiple times since this incident was first report. i appreciate their transparency and willingness to be part of this conversation. after our witnesses conclude their remarks we'll move to a round of five-minute questions based upon order of arrival. as a reminder to my colleagues this incident is not over, so too the investigations by the
fbi. so there might be some questions our witnesses cannot answer. however, i'm confident we'll get those answers at some point as we'll move forward. i now recognize the vice-chairman for his statement. >> thank you, mr. chairman. and thanks for convening this hearing. i'd like to welcome our witnesses who are here to help the committee's examination of what is the largest cyber supply chain operation ever detected, so we really do appreciate you being with us. as the chairman mentioned we had extended an invitation to amazon to participate. the opration we'll be discussing today used their infrastructure at least in part required it to be successful. apparently they were too busy to discuss that here with us today, and i hope they'll reconsider that in the future. this operation involved as already bip said the modification of the solar winds platform which is widely used software program.
it included a malicious back door downloaded by 18,000 customers. it hijacked the very security advice promulgated by security professionals to verify and apply patches as they are issue. so there are many concerning aspects to this first of its kind, at least at this scale, operation that raised significant question. my understanding if fire eye had not investigated an anomalous event last year it's possible this would be a continuing and unfettered operation to this day. as it appears they've been in the system for close to 5 to 6 months it was detected. the bottom line question what
are we still missing and how do we make sure it never happen again. second i think there's great interest in knowing what these actors did. based on what we know to include what the government has stated publicly ather seemed to have undertaken operations to the small subset of oprations they potentially had access. aside from the mechanical aspects what do we know about the actors chose the targets they did, what actions did they undertake in those networks, and what do we know that we do not know? i always love that question. what do we know what we do not know? in essence what are the open questions now and in the future about the tools and what do we still have open ended and who has the single auchl cahensive view of the totality of the
action under taken. and what is it going to take to rebuild our confidence. and one of hallmarks of this operation was the great care taken by this adversary to use be spoken infrastructure and trade craft for each victim. unlike other operations there is no template here that can be used for remediation. so what's it going to take to have confidence in both government and private sector networks again? fourth, what do we need to do to raise it bar for the cyber security of this nation? the cyber deterrence and achievable goal, how do we need to enhance information logging and sharing across the spectrum to protect against apts in the future? and finally though this is a question for the government rather than the witnesses here today, i think it's important for this committee to ask itself and inform the members of the senate what does the united states government need to do to respond to this operation? government officials initially stated this was an intelligence
gathering operation. just recently, however, the white house stated, quote, when there is a compromise of the this scope and scale both across government and across the u.s. technology sector to lead to follow on intrusions, it is more than a single incident of espionage it's fundamentally a concern for this to become disruptive, end quote. those are not the facts that are in front of us. everything we have seen thus far indicate at some level this was an intelligence operation and a rather successful one that was ultimately disrupted. while there are a myriad of way interest sovereign states to respond i caution against the use of certain terms at this time until the facts lead us to the use of terms such as attack and so forth. i always advocate for standing up to our adversaries. i think that's important. i will continue to advocate for that, but i want to know today what the actors intent seems to be and to the extent of damage
before we categorize it, it may very well reach that level. this committee and the rest and perhaps we should consider mandating certain types of reports as the relates to cyber attacks. we must approve the information sharing of this there is no doubt between the federal government and private sector and i look forward to being an active and constructive participate want in these debates on these new issues as i know everyone on this committee is. with that i want to welcome you. it is important that public understand the current, persistent information conflict the united states finds itself in against nation state adversaries like russia but also like china and iran and north korea. so thank you, mr. chairman. >> thank you, senator rubio. i think we're going to go ahead and we'll just tradeoff. i believe the order of the
speakers is going to be fire eye, solar winds, microsoft and crowd strike. so kevin, you want to start us off, that'd be great. >> thank you, mr. chairman, ranking member, rubio, and the rest of the members of the senate intelligence committee. it's a privilege to be here to speak with you and as the first witness i'm going to discuss what happened as a first-hand experience as a stage 2 victim of this intrusion. i have opinions who did it, what to do about it, but in the next four minutes i don't have time to get through all that so i look forward to your questions. responding to breaches is what we do for a living. we have a whole bunch of quincy type people that do forensics 2,000 hours a year, and people hire us to figure out what happened and what to do when they have a security preach. we responded to over 1,000 breaches in 2020. it was a tough year for chief
information security officers. and as i sit here right now testifying to you we're responding to over 150 security breaches. in short this is what we do for a living, and what we're going to tell you today we tell you with high confidence and fidelity on the intent of attackers and what they did. now i want to present the anatomy of this attack. whoever this threat actor is, and we all pretty much know who it is, this has been a multidecade campaign for them. they just so in 2020 -- the first part of the saga, stage one you had to compromise solar winds and the attackers modified the build process, which means it's a more portable attack than solar winds. when you modify the build process, you are doing the last
step before code becomes production for buyers and customers, which shows it's a sophisticated attacker and once they did that, we did not find it until 2020, the attacker did something interesting, they did a dry run in october of 2019. they put an innocuous build in to make sure that it made it in to the environment. there's no magic wand to say where's the next implant. we were set up you to do that investigation. it's what we do. we put 100 people on the investigation. almost all of them had 10,000 hours, of doing investigations and we unearthed every clue we
could find and we still didn't know how did the attacker break in h and after exhausting every lead, the only other evidence was the solarwinds server. we had to tear it apart. there was 18,000 files in the update. we had a million lines of code, if you have not looked at assembly, it's something that you need specialized expertise to review, understand, and piece apart and we found the needle in the hay stack. the implant. how did we get there? thousands of hours of humans investigating everything else and that's one of the reasons i share that, is you wonder why people missed it, this was not the first place you look. this was the last place you would look for an intrusion. so now companies are compromised by that implant. stage one was the compromise of
solarwinds and then it went to the folks that downloaded it. that means the attackers had a menu of 17,000 different companies. stage two of the attack was the companies that these adarks intended to do additional action on. and i want to talk about what they did during stage two victims. and i want to say stage one, the attacker has not done anything more than crack open the window in to a company. but they have not gone in to the house to rob anything yet. stage two, they go in to the house to rob. when we look at the stage two threat actor, or stage two victims. this is where microsoft's top down viewpoint from their cloud, where there was a lot of activity, comes up with approximately 60 victim organizations and we read that the government's aware of about 100 organizations. for us, being a stage two, we had first hand account of what they do. they come in through the solarwinds i am plant, they went for your keys and tokens and
stole your identity architecture so they can access your networks the same way your people did. and that's why the attack was hard to find. because the attackers from day one, they had a back door, imagine almost a secret door in to your house and the first thing that happens is all of your keys are right there, they just grab them and now they can get in to any locks you have in your house, the same way people do. in a pandemic where everyone is working from home, it's harder to detect an attack like this, where the only indicator of compromise was somebody logging in as one of your employees and there's nothing else far-fetched about that. right after they got ourer valid credentials and our two factor authentication by-pass. whether it was something else, i have had enough experience to know they target specific
people, like they have collection requirements. so, there they targeted e-mails and documents. so, stage two, it was get credentials, so you could log in, get the keys to the safety deposit boxes, and stage, the next step, step two of that was access e-mail, access documents with said keys and then, the third thing was dependent on who you were and what you did and industry you were in as a victim, it's primarily what i put in the other category, steal software and source code and fire eye, take our red teaming tools that we use to assess people's security programs. bottom line, hard to detect and when i got my first briefing on this, and reviewed the facts on day one, everything aligned to a threat actor who it is my opinion was more concerned about operational security than mission accomplished and the minute you could detect the
folks, and stop them breaking through the door, they evaporated like ghosts. i thank you for setting the stage for the other witnesses. i'm excited to work with all of you and to myfellow witnesses and others in the private sector as well as the public sector, to advance our nation in defending ourselves in cyber space. now, i look forward to taking your questions. >> thank you. i think you need to get your mic on or bring it closer. >> members of the committee, on behalf of solarwinds' employees, partners and customers, in the u.s., and around the world, i would first like to say thank you for inviting us to this hearing. by way of background, i joined
solarwinds on january 4th of this year. prior to solarwinds i was with a company for over five years and previously held executive roles at other technology companies. in my roles, i have been involved with cyber incidents and have seen firsthand the challenges they present as well as the opportunities they create for learnings and improvements. while our products and customers were the subject of this unfortunate and wreckless operation, we take our obligation seriously to work tirelessly to understand it better, to help our customers, and to be transparent with our learnings with our industry colleagues and the government. solarwinds started in 1999 in oklahoma as a provider of network tools. and to this date, we have
remained true to our mission of helping i.t. professionals solve their problems and manage their networks now through more than 90 products. today, we remain a u.s. head quartered company. with over 3,000 employees working extremely hard to deliver customer success. when we learned of of the attacks. our very first priority and that reare mains true today. was the safety and protection of our customers. our teams worked incredibly hard and tirelessly to provide remediation within about 72 hours of knowing about these attacks. we also acted quickly to disclose the events to the authorities while providing remediations and starting our investigations of what do we learn about this, who may have done it, and what exactly
happened in the process of insertion in to our orion platform. we believe the orion platform was specifically targeted in this nation state operation to create a backdoor, in to the i.t. environments of select customers as my colleague kevin noted as well. the threat actor did this by adding malicious code which we call sandburst, between march and june of 2020. in other words, a three-month window was when the code with the malicious code was deployed. i will note this code has been removed and no longer an ongoing threat to the orion platform. additionally, after extensive investigations, we have not found it in our more
than 70 nonorion products. perhaps, the most significant finding to date in our investigation is what the threat actor used to inject sun burst in to our orion plat for. this injector tool we call sun burst was designed to work behind the scenes. sun spot which we discovered poses a grave risk of automated supply chain attacks through many software development companies since the software processes that solarwinds uses is common across the industry. as part of our commitment to transparency, collaboration and timely communications, we immediately informed our government partners and published our findings with an
intention that other software companies in the industry could potentially use the tool to detect possible current and future supply chain attacks within their software build processes. we understand the gravity of the situation and are applying our learnings of sun spot and sun burst and sharing this work more broadly. internally, we call these initiators secure by design. and it's premised on zero transprincipals and developing a best in class secure software development model to ensure our customers can have the utmost confidence in our solutions. we have published the details regarding this in various blog posts. but in summary, they are poexed three primary areas. the first, is further securing our internal infrastructure. the second ensuring and
expanding the security of our build environments and third, ensuring the security and integrity of the products we deliver. given our unique experience, we are committed to not only leading the way, with the respect to secure software development, but to share our learnings with the industry. while numerous experts have commented on the difficulties that these nation state operations present to any company, we are embracing our responsibility to being an active participant in helping prevent these attacks. everyone in at solarwinds is committed to doing so, and we value the trust and confidence that our customers place in us. thank you again for your leadership in this very important matter. and we appreciate the opportunity to share our experiences and our learnings. and i look forward to your
questions. >> thank you, and for the members who have not yet voted, i guess everyone is voted because everyone is almost gone here. so, mr. smith, thank you for being here, we appreciate it. >> well, thank you, vice chairman rubio and a huge thank you for bringing us together to discuss an important topic for the country and the world. and i want though say thank you for kevin and -- it took courage to step forward and share information, and it is only through this kind of sharing of information that we will get stronger to address this. i think kevin and sudaker the did an excellent job explaining what happened so i don't want to retrace their steps. first, what does it mean, and second, what should we do? well, roughly 90 days or so
since we have first heard about it from kevin's firm, fire eye, we found first, we are dealing with a very sophisticated adversary. and vice chairman rubio. i think your caution of labels is well put. at this point we have seen substantial evidence that points to the russian intelligence agency and no evidence that leads us anywhere else. so we will wait for the rest of the formal steps to be taken by the government and others but there's not a lot of suspense at this moments in terms of what we are talking about. it's very, very clear that this agency is very, very sophisticated and as kevin noted that has been true for a long time. that is not new. but i think two other things are new. the first is the scale of this
attack. or hack. or penetration. or whatever we should call it. at microsoft, as we worked with customers that had been impacted by this, we stepped back and just analyzed all of the engineering steps that we had seen and we asked ourselves how many engineers did we believe had worked on this collective effort? and the answer we came to was, at least a thousand. i should say, at least a thousand very skilled, capable engineers. so, we have not seen this kind of sophistication matched with this kind of scale. but there's one other factor that i do believe puts this in a different category from what we have seen and even with the thoughtful consideration, it's appropriate to conclude now this was an act of wrecklessness in my opinion, why? because well, in part because chairman warner put it very well. the world relies on the patching
and updating of software. we rely on it for everything. we rely on it for the safety and health of our computers and we rely on it for our physical infrastructure, hospitals and roads and airports because they all run on software. to disrupt, to damage, to tamper with that kind of software updating process is in my opinion to tamper with what is in affect the digital equivalent of our public health service. it puts the entire world at greater risk. and it was done, i think, one must acknowledge in an indiscriminate way. to seek to plant malware and distribute it to 18,000 organizations around the world, it's an act without clear analogy or precedents. we have seen it done in ukraine, but not quite like this. it's a bit like a burglar who
wants to break in to a single apartment and manages to turn off the alarm system for every home and every building in the city. everyone's safety is put at risk and that's what we are grappling with here. so what do we do? i think we have to start by acknowledging and recognizing that we need to do a lot. we all need to do a lot. we need to do a lot ourselves and we need to do a lot together. certainly as sudaker was mentioning, we need to focus on the integrity and the software build systems the international data corporation estimates there will be half a billion, software apps created in the next three or four years. it's not just software companies, it's banks, it's hospitals. it's government, it's everyone that creates software. there's new steps that we will need to take to better secure and protect against the kind of attack that we saw here. second. i think we have a lot of work
still to do. certainly across the united states when it comes to the modernization of our i.t. infrastructure. and on to the application of i.t. best practices. at microsoft, we can only see this attack among our customers when it got to their use of their cloud services and all of the attacks that took place, took place on premise. meaning a server that was in a serving room or a closet somewhere and it points to the fact that until we modernize and move more people to the cloud, we are going to be operating with less visibility than we should. third, we do need to enhance the sharing of threat intelligence. now, that's the term in the signinger security community for information about attacks that people are seeing. and ourer basic challenge today is that that information too often exists in silos, it exists in silos in the government. exists in different companies. it doesn't come owing.
fourth, i think because of that it is time. not only to talk about, but to find a way to take action to impose an appropriate matter some kind of notification obligation on entities in the private sector. and so, of course, you know, it's not a typical step when somebody comes and says, place a new law on me. put it on ourselves. put it on our customers. but i think it's the only way we are -- it's the only way we are going to protect the country and the world. and i do believe it's time, maybe even overdue time, for us to look at the rules of the road. the norms and laws that if not every government is prepared to follow at least the united states and our like-minded allies are prepared to step up and defend. and among other things the say that this kind of tampering, indiscriminately and disproportionately with a software supply chain needs to be off limits and there needs to
be attribution and i will close by addressing a question that vice chairman rubio, i think you posed, who knows the entirety of what happened here? one entity knows. it was the attacker. the attacker knows everything that they did. and right now the attacker is the only one that knows everything that they did. we have pieces. we have pieces at microsoft. solarwinds, fire eye. we all have slices, people in the u.s. government. but we need to bring the slices together and until we do, we will be living and working and defending on an uneven playing field. that is not a recipe for success. but, let's also acknowledge one other thing. we know more than we did 100 day s ago.
we are better informed and we can turn the knowledge in to a resolve and action. that's what we need to do. that's what i hope the congress can do. that's what i think the country and our allies need to do, if we uses what we have learned, we can better protect our future. thank you. thank you. and finally mr. -- i believe he is on virtual? >> yes, thank you. >> good afternoon members of the committee. during my three decade career in cybersecurity, i have seen first hand the evolution of adversary techniques and have been at the forefront for solutions to thwart them, by the time i co-authored the number one book in security. it was clear that agencies
failed to defend themselves. when i co-found cloud strike in 2011, it was based on the conviction that the then dominant approaches for security were no match for adaptive and well d adversaries, i haveprotected thousands of organizations across the globe. in mid december, solarwinds engaged our professional services team to perform incident response. although we had not worked with them prior to this engagement, nor had they used our software in the past, our teams collaborated effectively to investigate the breach, enhance the security posture and ensure actionable intelligence with the security community. with their encouragement we
shared findings with customers, industry partners and federal agencies as appropriate. today, i would like to highlight a few significant capabilities this particular threat actor exhibited. notably the threat actor took advantage of systemic weaknesses in windows authentication architecture and created false credentials and impersonated users. it modified code in the development pipeline prior to the software build, the final stage before source code is software. the threat actor leveraged unique ip addresses for commanding and control infrastructure for each of its victims, complicating investigations in to the scope of the campaign but using common encryption methods and scrubbing techniques to avoid leaving behind unique indicators. the threat indicator was
selective in activating the back doors, selecting the victims of the wider universe of those that were vulnerable. cloud strike refers to this activity cluster behind the events using the name stellar particle. we are aware that this u.s. government has stated that this is likely a actor of russian origin. we have no information to suggest that is incorrect. regardless of attribution, there's a number of take aways from the event. this campaign in particular stressed the need to improve two security disciplines, those involving supply chains and those involving security development. stellar particle is just a latest demonstration of supply chain attacks as a threat fact vector. this the follows a number of previous high impact campaigns are the origins of attack are at the vendor level. with respect to software development in addition to
securing secure coding practices and adequate code review, they must protect them as well as their enterprise environment. next, i would like to do extend our consideration for the campaign. the first is we know that the adversaries periodically breech well defended enterprises. properly trained and resource die fende -- resource defender stop their goals. everything stops the bad actors from achieving their goals. and the ability to defeat novel threats, machine learning and artificial intelligence is essential. and the need to enhance identity protection and authentication.
the work from anywhere models, enterprise boundaries have continued to erode, this trend increases the risk of relying on traditional authentication methods and further weakens legacy is security technologies. one of the most sophisticated aspects of the campaign was how skillful the threat actor took advantage of the federation service. the golden attack allowed them to jump from customer on premise environments and on to cloud and cloud applications, by-passing multi-z factor authentication. it operates in a cloud scale version of similar attacks that i original wrote about in 1999. moving to the fifth concept. let's touch on principals of zero trust.
instead of authenticate, they must do this for each access. finally, i will touch on something known as xdr, which stands for extend the -- extended detection and response. this committee will appreciate it is guarded against information overload. the last point is critical. often adversaries specifically target smaller organizations as a means to a greater end. this is part of the supply chain problem. we are proud that a number of security companies including cloud strike are committed to offering comprehensive, easy to
use solutions for organizations of all size with varied budgets. we appreciate the need for improvements to government cybersecurity. some of the most talented people in the field currently work in government organizations, unfortunately in many instances our colleagues are hobbled by legacy programs, complex procurement processes and it detracts from the security work. i have described a set of enormous challenges today, but i would like to close on a positive note. with the trillions of events across thousands of customers globally i'm encouraged by the silent victories that the cyber community sees every day. i remain optimistic that working together we can prevail. i hope my testimony today is offered guidance on how we can accomplish that shared goal.
cloud strike has its sleeves rolled up and ready to continue to work with this committee and the greater security community to achieve success. i would like to thank the committee for inviting me to testify today and for its leadership and i look forward to answering your questions, thank you. >> thank you, let me just begin, by saying, you have shown tremendous operational security behavior. that backdrop you have in the video, you could be anywhere in the world, no way we can tell where you are. i will get that backdrop, that is an awesome one. let me ask you and the others the same question. let me say, everyone is familiar. the general public is familiar with cyber attacks and hacks and the general guidance everyone is given is, you know, don't put some simple password like 1, 2, 3, 4, they are easy to guess because we have seen, you know, they can guess it, there's all kinds of things out there to crack them. then, there's the infamous, the
well known phishing e-mail, you get an e-mail and click on it and it's in your system. for folks at home, or who may watch this later or trying to understand what the big deal is. this is involves the other thing that we are told we need to do, constantly upgrade the software. every time you is a software update, put it in, it has new security features. these guys get in that software update and you are basically, and it's like bringing them in to your system under the guise of protecting you. that's what we are dealing with here today. it's been a known vulnerability that people knew was a possibility. it's my understanding it's the first time we have seen it at this scale and scope. and you will correct me in your answer if i'm wrong. the question for all of you, this is a sophisticated technique, it's not something that is done on out of the basement of a home or could we
see it be widespread. what level do you need to embed yourself in a system upgrade that winds up in somebody's system? >> well, you know -- i will jump on that first. and this, this was a planned attack. this is not something done in somebody's basement. there's somebody that thought about this. my gut is it started somewhere that said, if we want to compromise the entities where's the supply chain. they had a list of 5-10 companies. when they got in, they didn't just rush right to the implant. they wanted to make sure to inject code first in the build process. that was in october of '19. and then four to five months later, they have an implant in the four to five months they designed an implant that looked like solarwinds traffic, it was hard to pick up in the network. and it had malware, you hear that word and you shut down.
it's what it did. it slept for the first 11 days after it was installed. so, that if somebody did detect it's beacon going on out, they would not be able to associate it to the update they did randomly 11 days sooner. another thing it did, it looked for nearly 50 different products and shut them down when it ran. so, people are like, why didn't anybody detect the implant, because when it executed, it looked to see if cloud strike's agent was on the end point and it shut it off and you do not make a back door as a bad guy, as a regular user, make one as the root user. a system level back door. senator rubio, no doubt in my mind, this was planned. it was an operation. there's a lot of people involved. and the question really is, where's the next one? when are we going to find it? >> i'm guessing you probably agree with that assessment.
so this is all without little doubt a nation state actor. it would take that level of sophistication, is that right? do you both agree with that? >> i do. >> yes. >> who? who was that nation state actor, have you seen indications that tell you, this is who we believe it is? >> george, you want to go first on that one? >> well, when we look at the adversaries across various nation state actors, there's a level of sophistication and trade craft as i pointed out in my testimony, the trade craft and operational security was sperb. one of the things that we typically look for are things like markings within tool chains. what we saw in particular with the backdoor and the build process was something we call code washing. that was actually removing the tool chains to, these fingerprints that kevin
indicated that our company and his company keep on file, right? so we know who the bad guys are and how they operate. in this case, these tool chains andthe infrastructure is very unique. what it means, they took particular care to actually conceal their identity and at the highest level, we have attributed in my written and verbal testimony to a particular cluster of activity, i know the government has talked about russia as being one of the threat actors. you know, from our perspective, we have nothing further to add to either confirm or deny that. but what i can tell you, it is absolutely a sophisticated nation state actor and as kevin said, it took a lot of work. a lot of planning went in to this. when you think of how difficult software is to build. each of my panelists are in the software business, we know how hard it is to build software and get it working and the idea to
inject something and have it all work without errors and without anyone actually seeing it is, again, superb trade craft and something you have to look at. i will turn it back to kevin and brad, they have further thoughts on the attribution piece, as i mentioned, a sophisticated actor that we continue to track. >> and one thing unique to this case is, when you do the evidence on a thousand cases a year, and something doesn't fall in to a grouping, that is odd, that is peculiar. and then when you go back 17 years in the cases and digital fingerprints and it still does not fall in to it, you start doing a process of elimination. when we found the ip addresses used to attack fire eye, we went to partners like microsoft, we went to the u.s. government and you go to the intel agencies. and nobody had seen them in use before. i will just summit up.
-- it's not consistent with china, north korea and iran. and it's most consistent with the behaviors out of russia. >> appreciate those answers. i do think that we have had the previous administration acknowledge likely russian. we have had testimony of the people in front of us we have had the current administration acknowledge this source as well. i think the sooner we make even more full attribution, the better. we need to call out the adversary, we know who did it and plan an appropriate response. while this incident and i agree with senator rubio, we don't even have the language down entirely, sometimes we know, you know what denial of service and espionage, where it fits is an
ongoing question. i think we have often times talked about it as the solarwinds hack, but there's other vectors and the "wall street journal" repored 30% of the victims were accessed by other means, not solarwinds. maybe it's best for fire eye and cloud strike and microsoft would have a view as well. why aren't we getting more details about the other vehicle tors that the adversary has entered? the other platforms that may have been utilized? again, i think it's reflective of the point that since we are totally waiting on willing participants, we could still be uninformed, because other marriage enterprises could be victims as well and not chosen to come forward. how can we get a better handle on, the nonsolar winds component of the attack?
yeah, i can tell you, this is, we are doing stage 2 investigations right now for customers. and the number one other way we are seeing these attackers break in is what is called password spraying. they are popping past phrases that they got from a breech over here and if you think of us, we all have amazon and microsoft accounts and whatever we are using. we have an e-mail account and pass phrase that we may use to access a bunch of applications. some of those third party breeches make our user i.d. and pass phrase aware to the threat actor and then they try it on your corporate networks. is so these are not, when i say password spraying. i feel like they know the passphrases by the time they show up and knock on your door. so we have 3300 employees at fire eye. i have to believe that some of them used their fire eye.com e-mail to access dozens if not more of the apps on the internet. if any of the vendors debt
compromised and they use the same passphrase for an amazon.com as fire eye.com, we may have a problem. so, that's another tactic that they use a. here's the reality. they have zero day capability most likely. how they get initial foot hold on a network will continue to change, the way you know it's them, when they come back in, they target the same things, the same e-mails, the similar documents. >> my question is i want to make sure, brad and george and if you want to add to this. we have talked about it as a solar winds hack. there's other vectors that they entered. and butfor the fact that you came forward there may be other very large enterprises that have not been as forward leaning that may have the vulnerable that still exists. >> i will say a couple of things. absolutely there's more attack vectors and we may never know
exactly what the right number is. i think the first question you are in affect asking is well, why? and i would say this, you know, this is like finding somebody in the building, and now, you have to figure out how they got in. and you know, in our case at microsoft, we identified 60 customers where we figured out that they have obtained the a password to somebody, an i.t. administrator that could get them in to say, office 365. in each instance, they got in on premise. so is, it was not in our server. or our service. and so, we need to work with somebody else to get to the bottom. >> doesn't that mean though that, this is not demonstrating a unique vulnerability that is in microsoft enterprise? >> oh, absolutely. >> or microsoft cloud, but there's brand name players that may have been with penetrated that have not been as forthcoming or leaving policy
makers and potentially customers in the dark, is that true or not true? >> it is absolutely true. it means two things, yes, there's a variety of services and there's a lot of ways in. i also would just pick up on one of the things that kevin said, because he used a phrase that is familiar to all of us in the cybersecurity community but probably not to say somebody who is say, watching the hearing from home. this notion of a password spray, i think in recent years we have all sort of learned that people may try to figure out our own individual password, a password spray is when you use a single password and you apply it to a lot of accounts. for example, if i were to go back to where i grew up, near green bey, wisconsin and have a thousand e-mail addresses from people in green bay and i just applied the password go pack go, i bet dollars to doughnuts there's a green bay packers fan that is using that password. in fact, i will bet there's more than one and if i find 10 of
those thousand, i am in. it points to a variety of tactics, from the most sophisticated, when you talk about disrupting a supply chain to the broad that points to a lot of factors that we need to keep learning about to secure our own e-mail and other accounts. >> i will move, but it does beg the question that senator rubio and i both asked about when a large enterprise like amazon is invited they should be participating. there's other brand name known i.t. and software and cloud services that may have been vulnerable to this kind of incident as well. and their public inactive participation, we will make sure it takes place. >> thank you, mr. chairman and thanks to each of you for testifying here today. i'm, i share the concern that has been expressed at amazon web services declined to participate. i think that's a big mistake. it denies us a more complete
picture that we might otherwise have i and hope they will reconsider and cooperate with the committee going forward. i thank you for talking with me yesterday, and since you are head quartered in austin, texas, i took note and appreciate that conversation. i think one of the things we discussed is something that chairman warner brought up and that is, even though solarwinds is the focus of what we are discussing here today, this is not unique to solarwinds, correct? >> thank you for that question. you are right. i will elaborate on the question that senator warner has asked, and tied the two comments together here. supply chain attacks are happening as we speak today, independent of solar winds. there was a report just two days ago about a french company being
hacked and it was dubbed as a supply chain attack. as we discovered what we call sun spot, the code that injected too, and as we evaluated it, it is blindingly obvious that, that can be applied to any software development process. which is the reason why we believe that dubbing it simply as a hack is doing injustice to the broader software community and giving us a false sense of security possibly. which is the reason that we are taking correct i have stcorrect an active participant in the endeavor to make us all save and secure by promptly outlining our findings and communicating them with both our government authorities as well as the industry. >> our time is limited today and i hope at some point we can talk
about the attribution and the putting of the russian intelligence services or whoever is responsible here at risk. because right now, it seems to me that we are doing a very bad job generally speaking of punishing the people who are perpetrating the attacks. but let me just ask you at different times. i know there's been legislation offered, senator collins and i discussed something that she introduced previously. with joe lieberman, our friend, the former senator. it seems to me that there should be an obligation of some sort on the part of a victim of a cyber attack like this to share what they know, what they have learned with the appropriate authorities. and i can only imagine the chills that run up and down people's backs when i say that, if we are going to get our arms
around it at all, seems to me that we need to know more than we know under the current practices. and in terms of the obligation of the victims to step forward. before i ask you about that, and what that the would look like, perhaps with some sort of liability protection associatesed where it. i will tell you that, i'm a member of the judiciary committee as senator feinstein is, we have designated seats like the judiciary committee, and mr. smith from your experience testifying there, usually when we are talking about data breeches. people want to talk about the company that allowed the data breech, how can we sue them? and which is entirely different perspective than i think we need to have a more complete approach for this. the offender.
we are about a some sort of what is coupled. the intelligence and the past phone companies and cooperated with a certain collection that got liability protection as part of that. mr. smith do you have a view on that? >> i think the time has come to go in that direction. i think senator collins was ahead of her time or the rest of us were behind our time. but either way, you i think we can find a way to move forward this year. i would perhaps use the word notification rather than disclosure. we should notify someone. we should notify, i think, a part of the u.s. government that would be responsible for aggregating threat intelligence and making sure that it is put to good use to protect the country and for that matter people outside the country. i think we need to decide upon
whom that duty should fall. and some kind of -- it was information fast to the right place. i agree with it, and coming down to another level. preparing for the liabilities, and so, we like the idea of you can notify with threat intelligence that is actionable. you get speed from it if it's confidential. you can have threat data today and your arms around the incident three months from now. this is too big of a gap to have
a disclosure law and we are getting intel three months to five months too late. i like the idea of confidential threat intelligence sharing for whenever has the means to push it out and the legal requirement to inform those impacted and you don't know that day one. and fire eyes case, we were sharing intel really fast, and we did not know what we had lost in our breech yet, but we knew there was something different about it. that's an extra detail, get the intel, out there quickly if it's confidential. my time is expired, i will yield back. >> this is a subject that we will come back to, and there are models out there. i don't think our traditional reporting mechanisms necessarily work. so the national transportation safety board, senator widen is up next. >> thank you, mr. chairman. the impression that the american people may get from this hearing is that the hackers are such
formidable adversaries that there was nothing that the american government or our biggest tech companies could have done to protect themselves. my view is that message leads to privacy violating laws and billions of more taxpayer funds for cybersecurity. now, it may be embarrassing but the first order of business has to be identifying where well known cybersecurity measures could have mitigated the damage caused by the breech. for example, there's concrete ways for the government to improve its ability to identify hackers without resorting to warrantless monitoring of the domestic internet. my first question is about properly configured firewalls. the initial malware in solarwinds software was harmless. it was only after that malware
called home that the hackers took control. this is consistent with what the internal revenue service told me. their server was not connected to the internet to the malware could not communicate with the hackers. so this raises the -- i indicated to your folks, i would ask this, you stated that the back door only work if orion, had access to operate. in your view. they installed it on servers that were either completely disconnected from the internet or were behind fire walls that blocked access to the outside world. >> thanks for the question.
it is true that the orion platform, software, does not need connectivity to the internet for it to perform its regular duties. which could be network monitoring, system monitoring, application monitoring. on premise of our customers. >> yeah -- it just seems to me what i'm asking about is network security 1 on 01, and any responsible organization would not have software with this level of access to internal systems to connect to the outside world then you basically set said almost the same thing. my question thenar for all of you is, the idea that organizations should use firewalls to control what parts of their networks are connected to the outside world is not exactly brand new. nsa recommends that organizations only recommend traffic that is required for operational tasks all other traffic ought to be denied and
nest, the standards and technology group recommends that firewall policy should be based on blocking all inbound and out bound traffic with exceptions -- i would like to go down the row and ask each one of you for a yes or no answer. whether you agree with the firewall advice that would really offer a measure of protection from the nsa, and just yes or no, and if i don't have my glasses on. maybe i cannot see the name tags. let's go down the row. >>. >> i'm going to give you the depends. bottom line, we do 600 red teams a year, fire wall never stopped any. it's like having a gate guard outside the new york city apartment building, some attackers are perfectly disguised as someone who lives in the building and walks
through the gate guard. in theory, it's a sound thing and it's academic in practice it's operationally -- >> i don't want to use all of my time. we will see that your response to nsa and the national institute of standards is it depends. let's go down the row. >> my answer, senator is yes, to standards that are listed on 800-53 and others that define specific guidelines and rules. >> very good. >> i am squarely in the it depends camp. >> okay for the same reasons. >> i think we have one other person don't we? >> yes. >> and i would say firewalls help and are insufficient. as kevin said, and i would agree with him, there's a breech that we have investigated that the company did not have a firewall or even legacy anti-virus. when you look at the capabilities of the firewall, they are needed but they are not the be all and end all and they are a speed bump on the information super highway for
the bad guys. >> the bottom line for me is that multiple agencies were still breeched under your watch by hackers exploiting techniques that experts had warned about for years. so in the days ahead, it's going to be critical that you give this committee assurances that spending billions of dollars more after there were not steps to prevent disasterous attacks that experts had been warning about, was a good investment. so that discussion is something that we will have to continue, thank you, mr. chairman. >> senator cotton. >> yes, i'm here. so, thank you mr. chairman. gentlemen, thank you for appearing today. i want to start with you mr. smith. microsoft said some of the source code was stolen, does
that present future security risks and if so, what are you doing to mitigate it at microsoft? >> the short story is power security system does not depend on the secretsy of our source code, there's more course code by companies publish will -- publiced by source form code. it's not considered to be a particular secret. and our entire threat and security model is based on the premises that there will be times when people will have access to source code. do we like the fact that this actor saw it? absolutely not. but we do not believe it under mines or threatens our ability to keep our customers or ourselves secure. >> we will by the way, as we do, to answer the rest of your question, senator, we will ask ourselves what do we change? it's not apparent to me that i
need to have access to our source code, it's not apparent to me that our senate lobbyists have to have access to the source code. we may have those who have access to source code in the future. it's not at all the center of what we are focused on here. >> okay, approximately 30% of the victims of the attack were not using solarwinds software, what do you think it tells us of the nature of the attack, and how the victims were targeted. >> senator cotton, thanks nor -- thanks for the question. this is referring to the "wall street journal" report, 30% is approximate. there's many different types of attacks and threat vectors. we are not a security company per say, so we would not have detailed information about those types of threat vectors. what i can share is that discoveries that we have made with sun spot, can apply to any
supply chain out there, and it's quite possible the their active supply chain attacks on going right now. some of which we know about. >> would you like to respond? >> george, go ahead. >> when you look at the supply chain attacks here, it's difficult, obviously to identify these things and when we look at the advesaries capabilities and we look at what was done as we talked about earlier. it's not an easy problem to solve. and you know, from my perspective, it's one that we have to come together. we have on continue to share intelligence and information and we have to realize that there are many other techniques and actors that are out there, and when you look at the overall landscape, you know, 30% more from solarwinds this is not a
surprise. over the last year, we stopped 75,000 breeches that were in process and a quarter of them were nation states. so it happens every day from every nation state actor. every e-crime actor, and there's a variety of tools and tasking orders that are out there. it's an ongoing effort, i wish there was a silver bullet, there isn't, the big part is exposing the techniques and how prevalent the attacks are to the american people so we can do something about it and we can come together as a group both in the technology field as well as in government. >> and senator cotton, to me, the attacker did the solarwinds implant they have moved on to whatever is next. we have to go find it, this attacker, maybe their pencil is down for a few months. they will come back and be an ever present offense that we have to play defense against and how who they break in will always evolve and all we can do
is close the window and close the security gap better next time. okay and one final question, i think i put this towards -- to what extent do we think that this was designed for what we might call collection in the intelligence world, simply trying to collect information to learn more about america's intentions, plans, capabilities, or what you might call covert action in the intelligence world, sabotage of public utilities or say an sabotage of public utilities or military actions or could it be both.an sabotage of public utilities or military actions or could it be both.n sabotage of public utilities or military actions or could it be both. sabotage of public utilities or military actions or could it be both. >> we got to see what they did firsthand when they broke into us. they were very focused. they had specific individuals that they targeted. they had keyword searches that they did when they broke in. so this was not a group that operated like a tank through a cornfield. they had a plan, they had
collection requirements. and to some extent i would say they were disciplined and focused on those collection requirements. not a fishing expedition to just grab whatever they could grab. >> and just to add, i think it is important to realize as technology companies, we all leverage big data. the adversary does as well. and while they are collecting this information, they are also storing it, indexing it and they have the ability to go back to it. so if a new specific order comes into target a company, target a government organization, they can look for that access, they can leverage that. the second piece of this is, i mean, early days it was network exploration. and then it turned into data exfiltration. and then it turned into data destruction and impact. so certainly when you have this level of access, you can collect data. if you start impacting systems, it is a pretty good way to get caught. so could it be turned into that
absolutely. but in general what we've seen is collection and that simply goes into the big apparatus to be used again for further missions. >> thank you all for being here. thank you, mr. chairman, for holding this hearing. i wanted to get some clarification along the same lines as senator cotton actually. mr. mandia, maybe i'll start with you just for people at home who don't understand what they have read is this is a solarwinds investigation, that is what they imagine beer dealing with here. that is clearly not the case based on what we saw in the "wall street journal" reportwbe dealing with here. that is clearly not the case based on what we saw in the "wall street journal" reporteer dealing with here. that is clearly not the case based on what we saw in the "wall street journal" reportr dg with here. that is clearly not the case based on what we saw in the "wall street journal" report'rdg with here. that is clearly not the case based on what we saw in the "wall street journal" reporteeag with here. that is clearly not the case based on what we saw in the "wall street journal" reportali with here. that is clearly not the case based on what we saw in the "wall street journal" report alg with here. that is clearly not the case based on what we saw in the "wall street journal" reportdal.
that is clearly not the case based on what we saw in the "wall street journal" reporteal. that is clearly not the case based on what we saw in the "wall street journal" report. help us understand what that means in terms of the ongoing nature of this -- you know, when you say they put their pencils down, have they really put their pencils down or are they working the pencils and we can't see it because we don't know. you started out at the beginning saying maybe they went to a list 6 like five to ten vendors and said these are the likely ways in and we'll pick this one. but clearly they picked other ways in as well. so i'm just trying to get a sense of the full scope. >> and when i said pencils down, they were so successful that they probably got a few days off because they collected so much information. >> so they are waving the flag. >> right now there is such vigilance in the security community, they won't spoil the latest technique right now. we are all looking for it so the pencils down for the next great implant. i would be if you were there. how an attacker gets the initial intrusion varies. solarwinds was the initial campaign. but this group has been around
for a decade or more. we're probably respond togts kids of the people that i responded to in the '90s when this group was active. so the bottom line, how they get a foot hold, solarwinds was a way. they will always have other ways. they hack for a living. and what they do after they break in really doesn't change that much. they target people at least in our case that did work with the government, they target government projects, things that are responsive to key words oig. we respond to a lot of threat groups that you can tell they broke into make money or they broke this and there is a manual review where somebody is literally going through every file alphabetically on a desk stop. these folks have economy of movement. if they broke into your machine. these folks have economy of movement. if they broke into your machine. these folks have economy of movement. if they broke into your machine, they find responsive documents and get out of dodge. they have an economy that shows they professional. and that doesn't change.
so if they broke in yesterday via solarwinds and we patched that, fixed it like we have, tomorrow they will have something else. and they will try to come back through whatever doorway they can find. >> and tomorrow they might be looking for something else too. >> the good news is usually they aren't. but you're right, collection requirements can could change. we've identified this group because they would break in to it a company and then we could get them out and if they got back in, they are after the same sort of things. so tools and tactics can change, but a lot of what they target does not. >> and i'm happy for anybody to jump in. but with the rest of my time, there was some discussion earlier, and sorry we were in and out going to votes and things, about reasons that they might not want to actually destroy data or destroy systems because they might get detected if they do that, whereas if they
stay in there and they don't mess around with stuff, but if they wanted to really do mayhem in our systems, what would that look like? what is our worst nightmare look like, mr. smith? >> i have a few quick thoughts. building on answering the prior question and then this one. i would just add that in addition to targets in the united states, we have identified targets in mexico, canada, the uk, belgium, spain, israel and the uae. so it was broader and international in scope. second, yeah, 82% of the 60 victims were outside government. so i think that there is an aspect to your question, well, who else were they targeting and why. and i would say that there are at least two other reasons that we would surmise, two motives, if you will. sometimes if you are going after a government agency that has very good security practices in place, you might look for a
third party that might have an individual who was given password and network access to say the government's network. and you might hope that that third party organization, maybe it was a computer service provider, maybe it was in accounting or a consulting firm, maybe it was a think tank that was working on a contract. you would hope that maybe they had lesser security in place and that is why you would start there. it is a vehicle to get somewhere else. and then i do think at times they target tech companies in part to understand how technology works, but frankly, programs in the category of counterintelligence, every day we are looking. you heard the reference to threat hunting. we are looking for evidence of this organization engaged in attacks. i think that they want to know what we know about them and what their methods are. but then i do think that your other question is so important because at the end of the day, what do you do once you are
inside. do you just collect information or do you wreak havoc. well, this agency typically collects information. but we know exactly what havoc looks like. all you have to do is look at a day in june in 2017 when another part of the russian government used exactly the same technique, a supply chain disruption with a ukranian accounting software program. that too was an update. it turned off/damaged 10% of that country's computers. atms stopped working. grocery stores stopped the capacity to take credit cards. television news stations went off the air. that is what havoc looks like. and that is what we need to be prepared to defend against as well. >> and what mr. smith just referenced is we refer to as -- that potential existed even in this attack. senator heimrich.
>> thank you. so if i have this right, a nation state actor that is in all likelihood the russians used u.s. software and then command and control servers in u.s. data centers to conduct this attack. and i think the fact that this attack was launched from within the u.s. is potentially a really important part of the story. advance persistent threat actors know that the nsa is prohibited from surveilling domestic computer networks. so it makes sense for them to circumvent u.s. surveillance whenever possible. for any of you, do you believe that the adversary launched the attack from u.s. servers in a deliberate effort to avoid surveillance? >> i think it was sort of an iq test. we can't know exactly what they thought, but it looks like they passed the iq test. they figured out that it would be more effective and less likely to be detected if it was launched from a u.s. data center. >> anyone else want to add to
that? >> i would agree. >> i would agree with those statements. >> for mr. smith, while the focus continues to be on how the private sector shares information with the government, we also want to ensure that the government is doing enough to share information with the private sector. mr. smith, you expressed concerns in a blog following the solarwinds attack about the federal government's insistence on redistricting through its contracting our ability to let even one part of the federal government know what the other part has been attacked. can you elaborate a little bit about this comment and in what ways could the cybersecurity information sharing act of 2015 be improved to ensure that that is possible? >> yeah, i have to admit one of the things that i found surprising and a bit frustrating for us. because the first thing we do when we identify a customer who
has been attacked is we let them know. we notify each and every customer. it was immediately apparent to us that it was important not just to let an individual department or agency of the u.s. government know, but to make sure that there was some central part of the government that would have this information about the government as a whole. and what we found was that our contracts prohibited us from telling any other part of the u.s. government. so we would basically go to each agency and say can you please tell so and so in this other place. and the good news is people did, they acted quickly. but it does not strike me as the type of practice that makes a lot of sense for the future. so there is an opportunity for reform. >> probably not the most efficient way to make sure information travels quickly. >> it doesn't seem like it is consistent with the year 2021 and technology. >> mr. mandia, in your statement for the record, you said victims of crime are the first to know when they have been violated.
but in a case like this, only a few government agencies and a handful of security or other private companies are in a position to be the first to know. i agree that doesn't seem right. you suggested that small group of cyber first responders could prevent or mitigate the impact of cyber incidence through sharing information quickly and confidentially. that is a very intriguing idea. how -- can you describe how you think that would work? >> you bet. there has got to be a way for folks who are responding to breaches to share data quickly to protect the nation, protect industries, and that would require, a, defining what is a first responder. and i think it is pretty simple. if you are trying to figure out what happened to unauthorized or unlawful access to a network, you are a first responder. and if you do that for our companies besides yourself, you are a first responder. and first responders should have an obligation to share a threat of intelligence to some
government agency without worrying about liabilities and disclosures so we're getting intel into people's hands to figure out what to do about it. right now the unfortunate that the reality when you share threat intel, it is just a public disclosure. and it makes people wary to do so and we slow down the process. so that is what i mean by that. i could articulate it more. but first responders know who they are and i think it is easy to define. we have many laws that define certain categories like internet provider. if you are a first responder, you are obligated to get threat intel into the bucket so we can protect the nation. >> i think that is very helpful. when you detected this activity, were are obligated to tell the u.s. government, why or why not and was that obligation legal or moral? >> we notified the government customers we had before we went public with the breach. and we found out later who we had to notify or not. but the minute we had a breach i was talking to the intelligence
community, law enforcement. you know, you don't want to get email when you don't know if your email is secure. so i would say on the record i think that we told every government customer we had that we had a problem, period, before we even went public. >> i think both the point that this was launched from xles domestic servers and the lack of information sharing were important points. and now one of on you new members joining us remotely, senator casey, your first intelligence questions. >> mr. chairman, thanks very much. thanks for the welcome to the committee. and i appreciate the testimony of our witnesses. i wanted to start -- >> you can get a little bit closer to your mic? you're not coming through that well. >> i'll turn that up. you can hear? okay. i wanted to start with the role
of the federal government here. and maybe we'll just go down the panel starting with mr. mandia to give us an assessment of the federal government's response to date. and then i'll move to a second question regarding what we do going forward. so mr. mandia, why don't we start with you. >> without a doubt the number one thing the federal government can do that the private sector cannot do is impose risk and repercussions to the adversaries, period. so we have to have some kind of public doctrine to mr. smith's idea of rules of the road, we have to communicate where is the red line. i know we admire the problem, but we have to come up with tolerable and communicate it and impose risk and repercussions is the purview of the government and the second biggest thing is attribution. the government is in the best place to get attribution the most right. so those two things.
and by the way, there is no risk and repercussions if you don't know who did it. so those are the two things that i would firm place to the government who is best suited to do that. i'll leave to the other witnesses on the government's role and how to work with the private sector. >> i'll keep it quick. and the suggestion that i would make is to leverage some of the recommendations in the som solarwinds report and have a single position that all entities can communicate with and to and have the responsibility of that agency to then disseminate to every relevant party. to date we feel like we have to communicate with multiple agencies and sometimes that doesn't help us from a speed and ag agility perspective.
>> let me point to two successes. i think that it is notable that the nsa published a circular that described in technical detail the nature of the attack, how people could identify whether they were victimized by it and how they could protect themselves from it. and i think that it was extremely well done from a technical and cybersecurity perspective and it was published to the world and i think that the nsa and the u.s. government did the world a great service. and that is the kind of thing that we should aspire to have our government do in the future. last week i thought ann newburger took a similar critical step. she shared for all of us information that frankly none of us had, namely that the government had identified roughly 100 private companies and nine federal agencies that had been impacted by this incident. and that tells me that there is now at work real efforts to consolidate this information
across the different parts of the government. so that is encouraging. she's also indicated that her work is far from done. they are focused on next steps that need to be taken in a variety of ways. but i do think this is a very important moment. the government can authoritatively about the nature of attention and how to protect ourselves. and the government can speak authoritatively about the scope that has happened. >> i would also just to jump on this, i would also say that nist has done a lot of great work, put out some interesting information, some scripts that helped the public. and while we're talking about the government, we're talking about corporations, there is a whole host of smaller entities that are out there that have no real way to protect themselves. so i think to kevin's point as a first responder, which we are, which he is and others, it is important that we have a single source that we can go to.
we're doing incident response not only for big companies and governments, but many small companies and we need to be able to share the information as quickly as we can without impacting the customer themselves. >> mr. kurtz, one followup. when you go through what i think were six proposals, or recommendations, what do you think is the most urgent at least as it relates to the federal government? >> i think probably a couple things, but certainly threat hunting is one of the biggest areas. as we've talked about before, it is a sophisticated actor with enough time and effort, they will get into shr. somewhere. and we make the dids tinks between an incident and a breach. there isn't a major company or government that hasn't had an incident. but you want to identify those very quickly so that they don't turn into breaches. and these are like centuries that are looking for the bad guys. looking for these indicators,
back doors. i pointed out things like artificial learning and intelligence. all of my fellow witnesses are working on these sort of techniques as well as us and that is a big part of a go-forward strategy. figure out what is there, use the technology to our advantage. >> thanks, mr. chairman. >> let's me thank all of our panelists today for your willingness to be here and more importantly for your knowledge in this. i've got to reflect for just a minute and i'll do it even though senator wyden left because i strongly disagree with whahe implied. he implied because nsa and nist said that proper hygiene is a firewall that that should be something that should be mandated and everybody should use it and that would solve our problem. and the three of you that deal specifically in searching out intrusions said no, no, it
doesn't solve it. and so suggest that in the day of covid that you've got a choice between washing your hands, hand sanitizer and masks, but if you choose just to wash your hands and not do the other two you'll never get covid, it is ludicrous and i want the record to show that what the response from those who track these was listen, this is sophisticated. they are way past this. so yeah, that is a good thing for companies to adhere to. but don't think that that is going to solve it with the adversaries we're up against right now. i want to turn to george just real quick. in the solarwinds attack, amazon web services hosted most of the secondary command and control modes. and all of aws's infrastructure was inside the united states. now, i feel like having a cyberattack deja vu here. whether it is the russian hack of dnc in 2016, north korea and sony hack or current supply
chain hacks, we constantly see foreign actors good x. flighting domestic infrastructure for the command and control to hide the nefarious traffic in legitimate traffic. the problem is we don't have the ability to surveil the domestic infrastructure. so what should the u.s. government role be in identifying these types of attacks? >> well, i think that it is working with providers like aws, microsoft and others, crowdstrike and fireeye, and others. because when you look at this particular attack, why did they use u.s. infrastructure, because they just wanted to blend in. and i can tell you there is a ton of attacks that we look at that use foreign infrastructure, that use bulletproof hosting which is the ability to pay for hosting an infrastructure and we know who they are and we tend to look for those bad actors.
so if you can use infrastructure that looks legitimate no matter whose infrastructure it is, i will blend in and make it harder. and this particular attack was insidious just the way it communicated and the protocols it used, it looks like legitimate traffic using infrastructure that is normal. but that is why it is important when you think about these attacks to have visibility. you talked about threat hunting. to have visibility on the end points. because that is the tip of the speer. and these network access devices are just speed bumps as i talked about earlier. what is actually happening is on the end point and beaconing out and you have to have visibility and you have to collaboratively work with the private sector and the public sector together. and i think that is the only way that we'll solve it. >> kevin, i want to ask for a little more specific statement. you alluded to the fact that this won't stop without government dictate that says
here's what we're going to do. let me ask this way. will it stop if they pay no price for what they do? >> no, i think if you don't impose risks or repercussions -- i've used this analogy for so long, you'll get how long i've used it, we're all playing goalie and we're taking slap shots from wayne get gretzky. the puck will get in the net eventually and there is no risk or reproceed kugs. so we're all fighting a losing battle over time. >> so as it relates to solarwinds, can you build software today without the risk of what happened? >> thanks for the question, senator. we've done extensive analysis with our partners at crowdstrike and kpmg of our entire build environment and entire infrastructure. and we see no evidence of the
threat actor in our environment or in our build systems and our products. we've also learned from this experience and applied them to what i've been describing as secure by design. one of the key tenants of that is to evolve to secure development lifecycles.nts of ts to evolve to secure development lifecycles.ents of that is to evolve to secure development lifecycles. so we've come up with a methodology where we use build systems with different people accessing them with different access types. and we correlate the output of them across those three to significantly reduce the potential for the threat being a to are to consistently compromise every one of our build systems at the same time. to are to consistently compromise every one of our build systems at the same time. that is the level our teams are going through to build safe and secure solutions which i hope will be a model for others. >> are these practices that you are sharing with others in the industry?
>> we are completely committed to doing it and we're doing it as we do it. >> thank you, mr. chairman. >> a quick comment that i agree with senator burr's comment that firewall alone cannot keep out a sophisticated actor. but it doesn't mean the corollary. and i had conversations with the solarwinds on this, that just because it is a sophisticated actor doesn't mean that you shouldn't do the good cyber hygiene. >> absolutely. >> it is not an either/or. >> i agree with you totally. i think what we're hearing and maybe we're just not saying it right, is that even with the best cyber hygiene, even with the best protocols in place, because of how good and persistent and how much money the nation state has like russia, we're susceptible. you know, the puck is going to get in the goal as kevin said. and if we've missed anything and you've got something that assures us the puck won't get in the goal, then here or privately
share what it is so that we can begin to pursue and flush out that type of policy. >> but the problem is we may not know the puck was even in the goal but if you got the good cyber hygiene, chances are you will sdoofr discover the puck at some point. we'll continue the hockey analogy as we move to our next new committee member senator gillibrand. welcome to the committee. >> thank you. i want to follow up on knowing whether you've had the puck go into the goal. one of you said that the hack that shut down crowdstrike and other defense software, it affected them before they could start working. so why was there no alarm and how were they shut down? and related, why were there no alarms in the solarwinds and anti-virus software logs which should have shown the unusual behavior access on or other traces of unauthorized access?
>> this is george. maybe i can take that. there were probably multiple dozen software technologies that were targeted to actually be shut down. in our particular case, you can think about the camera. someone came up to a camera and smashed the camera, you'd actually see what they did. and sour particular software has a level of monitoring where if someone tries to tamper with it, we would be able to see that and you'd actually have to reboot the system. as kevin mentioned, pretty persistent where it waited and kind of did things over a nuchlg number of days. >> but no alarm even after the 11 days? >> well, once you have admin access on a particular system if you are shutting it down, you can pretty much do anything you want on it. and what we focus on is -- and i talked about it in my testimony, no silent failure.
and we've designed our system that even if there is a failure somewhere along with what we call the kill chain, this attack sequence, we'll still attack something down the road. and i think this is something really important when i talked about threat hunting. you may not catch the initial stage of the attack, but you are looking to catch it along the way and you are looking to do that with speed. if someone is going to rob a bank, there is only so many ways to rob a bank. you have to get there, you have to get the money and get out. what car they drive, what weapon they use, how they do it, it doesn't really matter. so as long as you can identify the chain of activity which is really important, you can stop these breaches. and that is why we stopped over 75,000 breaches just last year. so it is really about risk mitigation using multiple technologies and haves visibility across your network. >> and mr. smith, i think you said on 60 minutes oig that there were more than 1,000 developers working on writing
this malicious code. why do you know that or how do you know that, and with a group that big, if it is based in russia, how do we didn't detect it or see it before? >> well, there was a lot more than a single piece of malicious code that was written. and so one of the things that we analyze is what was done from an engineering perspective on each of these second stage attacks that kevin was talking about before. and in essence what we saw is a very elaborate and patient and persistent set of work. they entered, as they were in through that back door, they in effect opened a window, they then swept up behind themselves, they closed the back door, they used that window, they identified accounts, they were able for the most part to really rely on stealing passwords and accessing credentials especially
where credentials were not well secured, meaning they weren't stored on a hardware dongle or they weren't in the cloud but they were able to get people's passwords. and they were persistent in using what we call elevated network privilege to work across a network. and we just were able to look at our estimate of how much work went into each of these individual attacks, how many attacks there appeared to be in total. and we asked our engineering teams, these threat hunters that you were hearing about before, what do you think is on the other side of this. and that was their estimate. and we have asked around with others does this estimate seem off base and no one has suggested it is. >> let me ask mr. krishna a final question. so the "wall street journal" reported that there was as many as a third of the victims were
accessed by means other than solarwinds. however those access vectors including ttps and infrastructure have not been made public. why is that and do you expect to release the full details of the other access vectors and what other ways did the cyber actors use to gain access to victims? >> senator, that is a very good question. we as a manufacturer or producer of i.t. management tools do not haves security capabilities to be able to investigate other threat vectors and that is where the colleagues that the witness table with me can help us and the broader industry identify those threat vectors. on our part what we have committed to doing and continue to do is sharing everything that we are finding. and the significant discovery that i mentioned aboutsun spot is one key.