After a massive service outage on Friday, software-as-a-service giant Salesforce restored partial access to its affected customers over the weekend, while admins continued with cleanup into Monday.
The outage was brought on by a scripting error that affected all Pardot marketing automation software clients; a database script that Salesforce pushed out accidentally gave users broader access to data than their permissions levels should allow.
In response, Salesforce on Friday cut off all access to all Salesforce software clients, not just Pardot clients, while it triaged the situation – leading to a bit of a meltdown among users. Twitter hashtags #salesforcedown and #permissiongeddon began trending as users took to social media to complain.
“my salesforce rollout was scheduled at 2pm today. 300 folks on a call to do training with me. oops @salesforce #salesforcedown,” tweeted one user.
“#Salesforce #outage means that I can’t access any meaningful records, or properly do my job Now that most tabs have disappeared like 1/2 the universe in @Avengers, please bring them back, Tony Stark of @salesforce!” tweeted another.
“To all our @salesforce customers, please be aware that we are experiencing a major issue with our service and apologize for the impact it is having on you,” Salesforce co-founder and CTO Parker Harris tweeted on Friday. “Please know that we have all hands on this issue and are resolving as quickly as possible.”
Some users saw an upside to the situation:
All the Sales teams in #America headed to the bar once they heard @salesforce was down. #salesforce #salesforcedown pic.twitter.com/TANrHg2GVg
— Tommy Hayes (@huskysize) May 17, 2019
Over the weekend, the cloud app provider said that access was restored to everyone not affected by the database script, so regular Salesforce.com users were back in business. However, for companies using the affected Pardot software, only system administrators were given access to their accounts – so they could help rebuild user profiles and restore user permissions. According to the incident status page, some regular users remained incapable of logging into the system as of Monday morning as administrators continued the restoration process.
That process could be onerous for many: Salesforce said that if there’s a valid backup of their profiles and user permission data in the service’s sandbox, admins can simply deploy that. However, if there’s no valid backup, admins will need to manually update the profile and permission settings. Salesforce noted in an update Monday that it has deployed automated provisioning to restore permissions where possible.
Balaji Parimi, CEO at CloudKnox, told Threatpost that admins should take care when restoring the settings.
“Enterprises need to understand that their biggest security risk is not from the attackers targeting them or even malicious insiders – it’s identities with over-provisioned privileges,” he said via email. “Security teams need to make sure that privileges with massive powers are restricted to a small number of properly trained personnel. Until companies better understand which identities have the privileges that can lead to these types of accidents and proactively manage those privileges to minimize their risk exposure, they’ll be vulnerable to devastating incidents like the one we’re seeing with Salesforce right now.”
Want to know more about Identity Management and navigating the shift beyond passwords? Don’t miss our Threatpost webinar on May 29 at 2 p.m. ET. Join Threatpost editor Tom Spring and a panel of experts as they discuss how cloud, mobility and digital transformation are accelerating the adoption of new Identity Management solutions. Experts discuss the impact of millions of new digital devices (and things) requesting access to managed networks and the challenges that follow.
