ReadyXRM

A few months ago we pushed a enhancement update to one of our Dynamics CRM customers.  The enhancements were a collaborative effort between our team and our client’s team (whom have been become CRM experts).  The package included number of configuration changes, workflows, dialogs, JavaScript and some security changes.  The enhancements had already been deployed to the TEST system from DEV, tested and confirmed before deployment to production.

The next morning, the users logged in as usual, and began the days work.  Hours later, a few users noticed that particular fields on some of the forms were locked, others were not locked, and some values that typically would get prepopulated on new records were not prepopulating.  It was quickly determined that the “Business Rules” feature of Dynamics CRM were not triggering.

After some various troubleshooting, the effective entities were re-published and the business rules started working.  Had there been a step missed when publishing the new solutions, despite our meticulous check lists?  Regardless, case closed and we moved on.

A few hours later, reports of business rules not working again!  What?  Now we went into deeper troubleshooting mode, why would business rules work for a while and then stop?  Was this happening for specific users?

Very few patterns emerged.  The issue would present itself for for users with the System Administrator role as well.  The problem was intermittent.  The log files did not turn up any information.  Eventually we looked at the servers, routers, firewalls, load balancers, phase of the moon, time of the day, wind direction, etc, etc.

Working with Microsoft support, we analyzed the database, and the issue could not be replicated on TEST or even when the production database was redeployed elsewhere.

Doing some Google searching, very little details appeared.  A few very vague posts on the Microsoft community site, but not really related.  Eventually, there were a couple of other reports of the same issue happening at other sites on the CRMUG forums.  The great thing about the Dynamics community is that everyone is always keen to help each other out, so we compared notes which gave us a few other things to try (press CTRL – F5, recreate the Business Rules on Prod directly, remove JavaScript).  None of these potential fixes resolved the issue.

We took another shot at determining patterns.  There were some perceptions emerging (it was thought that the issue only occurred on IFD clients, only on certain browsers, etc) so I added some modifications to a key entity (Optionset -> Business Rules Not Working, Working) and a field to capture what URL was used.

The results showed that the issue was not 100%, that it happened both on IFD and non-IFD clients and from different browsers, it didn’t show any particular pattern.

We ended up to consider a plan B, replacing the Business Rules functionality with either JavaScript or other tools.  While not ideal, we needed to move ahead.  However, we still wanted to resolve the issue and continued to work with MS Support.

We were looking at trying to find patterns and were about to analyze network traffic.  We are also wanted to pinpoint the exact time after a reboot and see if that would shed some light on the issue.

During this time some of the analysts were working on another unrelated issue with Business Process Flows on the Lead entity.  At this time they updated the security role for the general users to have greater access to the Process entity.  This change was applied to the production system.

A few days later, while compiling the data from the recent reboot and the particular users from the team, we saw that the Business Rules had been working 100%.  Double-checking, we pinpointed that the Business Rules started to work when the recent security role change was made for the Business Process issue.  Whaaaat?  If the issue was security related, wouldn’t the issue present itself immediately?  Also, this would not affect users with the System Administrator role!  This is not a typical permissions issue!

So what happened?

When the issue first presented itself, the “all users” security role was modified to reduce access to the Business Process Flows so the security was changed from “organization” to “user”.  This change was part of the package of the enhancement updates.

A while later, the security role privilege was changed back to “organization”, and the Business Rules started working for everyone.

Apparently, there is a server side caching issue.  As the user loads forms, etc. any business logic (JavaScript and Business Rules) are compiled and cached at the server in order to make the process more efficient.  If a particular user doesn’t have full privileges for the Process entity, the compilation of Business Rules can get corrupted, and the cached form will only run regular JavaScript and Business Rules will not fire, not only regular users, but users that have full access to the Process entity (including system administrators).  If a more privileged user accesses the form first (after a reboot or IISReset) then the cache is fine for all users, until the next time the form gets re-cached, typically a few hours later.  The issue never presented itself on TEST as the testing scripts were done pretty quickly before the cache got refreshed.

Microsoft is apparently investigating a fix so hopefully this issue will be addressed in an upcoming service pack or update.

In the meantime, the workaround is to ensure that all users have at “organization” read access to the Process entity.

While a small issue, it caused a lot of grief.  Hopefully this post will save some of that hassle if anyone Googles “Dynamics CRM/365 Business Process Issue”.

Cheers!
Nick

Nick Doelman is a Microsoft Business Solutions MVP and is pretty pumped that the Ottawa Senators are up 2 games over the Rangers in the NHL Playoffs.