Tools for automatic performance tests in web client and Autopilot

[antoine_mpo]

Hi list,

I know it's an old question, and i read many posts on that topic, but we're trying to have some objective response time measurements of our OneWorld system (web clients).

Of course, we'd like to do that without buying big and expansive solution. We don't need a software for stress tests, for several applications ...
We just want to be able to regulary and automatically run a script (for instance a sales order revision), and know how long it took from start to end.


So, first, we are trying to use Autopilot, because it's bundled with our Xe.
But i meet some difficulties :
- Run an autopilot script with the web client requires to have a JAS set up with low interactivity and multiple browser disabled (which is not what end users have
-> so it would require to create a new dedicated jas instance, only for tests)
- There are some bugs (a script can sometimes stop in the middle, ...)
- Sometimes i don't manage to do exactly what i want (and get the exact same thing i would have if doing it manually)
- There's nothing to schedule the run of a script. If you want to do so, you have to create some external scripts, for instance in VBS, to launch autopilot programm,
open the script, simulate a "F5" key pressed, ...


I know that many people thinks autopilot sucks, and rather use some specialized tools.
I read several time the name of RAPIDFIRE. But i cannot find their website anymore.
Does someone know if it still exists ?

Does anyone know some tools that can be used with OneWorld Xe to run automatically a scenario, with web client, and get the running time, but that is not expensive ?


Thanks for you help.

 

[altquark]

First of all, forget about using Autopilot. Its completely useless, has little to no support from JDE and is not suitable for end-user use. If you look back at some articles on the list, you'll see that it was initially developed by my old group at JDE - Worldwide Advanced Technologies - as a method of stress testing JDE - but it proved to be too difficult to manage and use even for JDE ! JDE don't even use autopilot internally any more.

The reasons for this is because any slight change to an application - whether its a service pack change or an update (ESU) results in the script becoming useless. As such, you would then have to spend months rebuilding your scipts over and over again !

Secondly - just because Autopilot is supplied as part of the code - soes not necessarily mean you actually own it. You have to look into your contract and see if it specifically entitles you to run autopilot for stress testing. If not, then the product costs hundreds of thousands of dollars.

The iConsortium has "Rapidfire" which is an automated testing solution - along with "Webfire", both of these products simulate keystrokes against the screens - and both provide excellent response.

I'm not sure of the current status of Rapidfire or Webfire (I've sent The iConsortium an email requesting information) - but you need to identify exactly WHAT you're trying to test first.

There are several different testing scenarios.

1. Unit testing of custom code
2. Regression testing of custom code against standard JDE code
3. Stress testing and benchmarking

Unit testing against custom code really only works with a specific user since the code is altered often, or is new. It takes too long and places an undue amount of time on the development cycle if a testing tool is utilized to attempt standardized unit testing.

Regression testing is an interesting scenario - and I have set this up for several companies in the past. This requires a new pathcode that sits between PY and PD (I usually call this the "QA" pathcode) - and the data is refreshed from a single point in time each and every time the scripts are run to ensure that the results are always the same. This testing solution should only be utilized in companies that have a large amount of development, and an ongoing development requirement outside of just reporting. If you are making frequent modifications to existing business functions - then this testing might help your organization.

Lastly is a stress test. If you are, for example, loading on an additional plant or rolling out a large number of users and wish to test the system to ensure that the architecture can handle the number of users - then this situation is necessary. I've done this many times in the past for companies, where I use tools such as rapidfire/webfire to automate scripts taken from the business - usually data entry or update scripts - and run a small number of these scripts against the system to simulate user load. Based on the results of the test, the system can then be altered to reduce bottlenecks. This type of testing usually is based on a consultant (like myself) coming in and running the test scenarios for about 1-2 weeks. The results are then forwarded to the customer and highlights recommendations on how to manage the architecture. It is always recommended that a technical audit accompany this type of stress test beforehand to highlight any obvious flaws in the CNC architecture before testing starts.

I hope that helps !

 

[JEMILLER]

For stress testing I have used various expensive web testing tools such as Mercury (now owned by HP). When budget is an issue I use OpenSTA (free). While it doesn't have all the script creation functionality of the more expensive products it does a pretty good job.

For response time monitoring I code monitoring programs in Java using the Apache HTTP Client library. It is part of the Apache Commons Java libraries. You can create programs that emulate an end-user web client session and pretty much do anything a real user can do. Once you have the monitoring program created you can then plug it into a system monitoring tool -- HP OpenView, Nagios, BigBrother, etc.

As an aside, this same library is used to handle the communications between the Change Assistant application and JDE/ORACLE.

 

[Joel]

We use Mercury for load testing with 50-100 virtual users and Quick test pro ( another product from HP ) for getting details on screen display.
With Mercury products cost is a big factor because licences are not cheap.

 

[wainwrr]

We use OpenSta too. In particular at present we have our next DB server and replacement web and app servers set up in a test rig. I'm using OPenSta to replay scripts at the test rig as we tweak various parameters to get some baseline tuning. Obviously that's not a real world load but it does give good reproducibility.

The ability to record a web session and replay it n times for m clients with semi-random delays on the start time allows for rapid, if somewhat crude, test scenarios.

 

[antoine_mpo]

Hi,

Thanks for your answers.

First, let me be more precise about what we are trying to do :
We don't want to do stress test (we don't want to simulate big load on our servers), we don't want to do non-regression tests after objects modification.
What we would like to do is to run some scenarios, several time per day, to get some statistics on how long it takes to run (the goal is to check if we meet some performance issue, as time goes by and database gets bigger, or if there are some particular period of the day when the system gets on its knees).
For instance, we would like to have a scenario where we would go to sales orders entry (P4210), select a test sales order, enter the configuration, and validate it.


I understand from everything i read that Autopilot has big inconvenient (no real support from JDE, scenario gets useless when you change a service pack or install an ESU), but, at the same time, the advantage i found were that :
- It's specific to JDE, and uses objects i know. I don't have to learn some specific langage.
- We don't need to buy some expansive software which have much more functionalities than what we need.


So something like Mercury is too expansive, and not what we need.
I downloaded OpenSta, that some of you mentionned.
But so far, i don't manage to use it the right way.
I don't even manage to create a script to login and logout.
If some of you used OpenSta to create JDE test scenario, i'd be very interested about how you set it up.


Thanks a lot Jon for all the information you provided, and thanks for sending an email to iConsortium about RapidFire.

 

[wainwrr]

One quick thing to check on OpenSta (at least the version I use) is that it really does not like proxy (or auto detection) in your browser settings. Other than that, can you describe what you are getting when trying to create a script?

 

[sputnik]

Hi Antoine,

we faced the same issue as we tried to test our new E812 instance. I performed some tests on the HTML servers with: Microsoft Web Application Stress Tool. It is a free tool, easy to use and you are able to simulate user load/E1 apps on the HTML server.

cheers,

Adrian

 

[altquark]

Rapidfire is very inexpensive, and very, very easy to use - but I'm unsure of its status at the moment.

Alternatives to Rapidfire and Opensta :

WAPT - I really like this tool, it is cheap (free download for testing) and works really well. I've used this at a couple of customers in the past : http://www.loadtestingtool.com/

WebArt - I used this in the past with some customers and it works well, though it IS a little difficult to create the scripts (and has limited support since it seems to be a discontinued development product) : http://sharesite.oclc.org/_ZABLE%5B0%5D_/tab/15/index.htm

Radview - this company has a web stress testing tool that seems to be reasonably priced (compared to Mercury) - but it's still on the higher end of cost.

I'm sure there are others (OpenSTA is new to me, although I'll be trying it out myself soon) - but remember you need to identify a tool that can perform the tasks you desire - and can deliver back the results you need. If you run the tool, and no transactions get created in the JDE database - then its not doing the job correctly (there are a number of tools that work like this, they only really test the "website" - not the entire architecture). Steer clear of the Microsoft freebie webtesting tool or junk similar to those because all they do is test the website, not the JAS server.

Lastly, remember that many of the performance metrics can be obtained behind the scenes - for example in Perfmon on Windows boxes. Of course, you do need to know what you're looking at though - and thats not easy to supply for free !

 

[antoine_mpo]

</font><blockquote><font class="small">En réponse à:</font><hr />
One quick thing to check on OpenSta (at least the version I use) is that it really does not like proxy (or auto detection) in your browser settings. Other than that, can you describe what you are getting when trying to create a script?

[/ QUOTE ]

Hi,
Here is what i tried with OpenSta (I did that from a fat client where i installed OpenSta) :
I create a new script. In Options i select the browser (IE6), in Gateway i select to capture Local (i don't know it port numbers are required ?), and then i record. It opens a IE window. Then i just enter the Jde url, i log in, go to a menu, open a interactive application, close it, and logoout.
then i stop recording.
I installed a plugin on OpenSta, to be able to see the playback in a browser window (otherwise you only see some code result of the url submitted), called BView.

When i playback the script, in BView, i see the login page, but after that, i get error pages (JSP error, like "unable to find /taskexplorer/timeoutrouter.jsp"), or "Session Expired", ... I never see menus pages, nor application.

What's wrong ?

Thks.

 

[antoine_mpo]

Hi Jon,

Could you let me know if you get an answer to the email you sent to iconsortium about RapidFire ?
Because, as far as i remember, when the website was still existing, there was a demo of use of RapidFire with Oneworld ...

I downloaded WAPT. But, from my first tests, it doesn't seems to be what i'm looking for. When i record a scenario, i'd like to really see what it's doing when i play it back. With the tools i can only see some statistics about the different pages or url requested.

I downloaded WebArt and i'm gonna try it too.

It seems that these kind of software only tests the website, and indeed it's not what i'm looking for.
I really want to check how long it takes to run a whole process (including the time to get sql request result, bsfn time). I don't need the detail, just the global time of a scenario.

Thanks a lot Jon for all the information you're providing. I really appreciate your help.
And thanks to all of you for your answers and experience sharing.

Cheers,

 

[peterbruce]

It would appear (from the iconsortium website) that Rapidfire has been superceded by Webfire. The Rapidfire link from there homepage (Solution Center -> Proprietry Solutions -> RapidFIRE) goes nowhere. Here is a link to their WebFire page:

WebFIRE

 

[altquark]

[ QUOTE ]

It seems that these kind of software only tests the website, and indeed it's not what i'm looking for.
I really want to check how long it takes to run a whole process (including the time to get sql request result, bsfn time). I don't need the detail, just the global time of a scenario.


[/ QUOTE ]

You should be able to test the entire process flow - not just the web parts. Both WAPT and Webart will provide timing information between pages - as for the BSFN times, I find that SAW is always best for getting that information.

I think its important to run for a number of users - not just a single user - otherwise you'd not be able to gauge the effect of multiple users on the system. What I usually do is identify a common data entry process (data entry is usually the highest cost from a database perspective) and run that process a large number of times for a large number of users. It really doesn't matter if its the same data - the way that the JDE logic works the only side that would be really affected by similar data is the database - but the next number table would be hit just as much as if it were real users.

So, for example, if you were wanting to test Sales Order Entry - identify how many lines an average order takes, then create a script that logs in, performs the sales order entry, say, 10 times in PY and logs out. Then run this script across the number of sessions you believe would be a fair representation of your system for as long as you want. I usually stop and start JDE services just beforehand - then capture the SAW times for each business function as well as perfmon information for the client, JAS server and APP server as well as all the database PERFMON information.

With that information, you will be armed with information to identify how well your implementation works.

I've run several performance benchmarks for customers in the past - there are some white papers on my website that show some example results.

On the other hand, if all you're trying to do is to capture the average amount of time it takes for end-users to perform transactions - then why bother with the scripts ?! Just look at the SAW business function call information - capture that and you'll soon identify an issue if there is a drop in performance !

 

[wainwrr]

I hadn't seen Bview before and I have to say that having downloaded it, I get similar results. If you don't invoke bview does your script/test complete?

 

[tbrackett]

My apologies for such a belated reply, but I am the "daddy" of RapidFIRE! at the iConsortium and wanted to get you the followup.
1. RapidFIRE! is alive and well, the site is down because of the wonderful dark lords of the web hacked it sufficiently to require a complete rebuild in a new content management.
2. RapidFIRE! was previously licensed as a product, but has since evolved into an outsourced automation service. The reasons for this are:
a) We realized that customers loved the product, but had great difficulties in allocating resources (despite the ease of use) to the initiatives, thereby making license sales fail because "nobody to use it"
b) Many customers simply wanted to automate one or two things, and we could compile automations for them much more affordably as well as embed those automations into documents such as Excel test procedures, etc.
c) The resource load for our company for product development, support, training, and implementations was more than the load to "just do it for the customer"

We are also using RapidFIRE! to build out standalone automation tools to deliver generically to customers - an example of this is iBROADCAST, a simple application that allows IT to define report variables (program/version) and iBROADCAST will monitor the print queue, grab the report, rename it, archive it, and distribute it to a defined email distribution list.
Well, you get the idea...

Anyway! To answer your question, RapidFIRE! COULD be used for your web instances but its true power is with WIN32 deployments. It takes a little work to set it up, based on browser readiness, visual cues, etc., but it does mimic an actual user. Again why we outsource this scripting rather than make customers pull their hair out (we are bald already).

If you get to a release supporting the parameterized URLs then you would be golden. A word of caution though - the drawback of ANY URL based automation tools is that tools releases regularly change the URL correlations and MAF design, so your scripts really can't be "update-proof". With our WebFIRE! service, we have to utilize custom c# routines to update our web scripts once Denver informs us of the changes. (and yes, WebFIRE! toolset is spendy like all the others)

If you would like to talk about ideas, I always like a challenge, so you can reach me at 866-426-6701 Ext. 710.

But your plan is the right one! Proactive performance monitoring at the end user level is so important!

 

[Dennis Remmer]

Hi there - I'm experiencing similar behaviour when trying to use the Apache JMeter project tool - which is similar to those mentioned above - nice tool, have used it successfully for many non-JDE web testing situations.

I'm recording all the HTTP traffic between the JDE client and the server, and upon replay it seems to log in and out OK, but gets lost in the menu operations, form operations, etc. In fact I'm not seeing enough HTTP/HTML responses coming back to warrant the detailed UI that JDE presents in the browser. Not knowing enough about JDE client/server protocols, I sense that there is a lot going on in the Javascript, and that the testing tool does not adequately 'simulate' what needs to occur.

Did you get any further with your OpenSta attempts?

Can anyone shed further light on this discussion?

Thanks
Dennis Remmer
Brisbane, Australia

[ QUOTE ]
Here is what i tried with OpenSta (I did that from a fat client where i installed OpenSta) :
I create a new script. In Options i select the browser (IE6), in Gateway i select to capture Local (i don't know it port numbers are required ?), and then i record. It opens a IE window. Then i just enter the Jde url, i log in, go to a menu, open a interactive application, close it, and logoout.
then i stop recording.
I installed a plugin on OpenSta, to be able to see the playback in a browser window (otherwise you only see some code result of the url submitted), called BView.

When i playback the script, in BView, i see the login page, but after that, i get error pages (JSP error, like "unable to find /taskexplorer/timeoutrouter.jsp"), or "Session Expired", ... I never see menus pages, nor application.

What's wrong ?

Thks.

[/ QUOTE ]

 

[Sebastien Leclerc]

Antoine,

I am responding a bit late to your request, but I ended up reading this post while searching for something completely different.

There is a very powerful open source (Free) scripting/automation tool that may help you : AutoHotKey

I did not use it specifically for testing EnterpriseOne applications, but I have no doubt that it is powerful and yet simple enough for that purpose.

 

[Jaise James]

All,

I am using a very affordable tool to perform some web automation called IMacro. We run some readonly transactions every 15 minutes from different part of the country to gauge the performance and using SQL script we import them into our SQL DB. Then use Reporting services to provide a performance dashboard to Mamangement. We are not only using it for E1 but about 5 other Web applications that our company uses. Some of them are located in our data Center and some are ASP model

I found this tools to me very simple to use and it did accomplish what I needed to accomplish. I can record transaction and play back. I can put counterto measure etc.
I hope this helps