Get help with testing, discuss unit testing strategies etc.


Post by mats »

Welcome to the Siesta forums. We're very interested in hearing your feedback on our product. Things we want to know:
[*] Is the UI intuitive and fun to work with?
[*] Did you find it easy to run test 'headlessly' using PhantomJS?
[*] Which features are you missing?
[*] How could we improve the tool to allow you to write tests more easily and faster?
[*] Bugs found?

Post by Mrfr0g »

First, allow me to say that this is an excellent tool. I have already created over 100 tests for our framework using this tool. I really like the simulation classes, watching the mouse move and expand my grid component is really neat. A couple of thoughts, but first a little bit about our framework.

We use a custom built PHP framework. We have developed PHP classes to render the appropriate JSON config for Ext objects. Further, we wrap each of the Ext objects we use with a custom javascript delegate class provided by prototypejs. On our development servers, and our local machines, we use Ext's loader class to dynamically load the components we use on the page. On production servers we compile those files in to 'packages' and minify them. Needless to say, it's a little complicated.

+ Using the packaged version, I can run the tests an infinite number of times and experience no issues with running the tests.

- Using the Ext Loader class, I am sometimes able to run the tests once, but am not able to run the tests again without reloading the page. This is true even with caching disabled. From what I can tell the Ext.onReady event is only firing once when the classes are loaded dynamically. I am unsure how to resolve this issue, so to test, I am forced to use the packaged file.

- When testing the grid component with mouse simulation ( to resize the grid ), I experienced errors when the mouse moved over the grid headers. It seems that the grid header components call Ext.getCmp when the mouse is moved over the header to determine which header should apply the 'highlight' effect. Ext.getCmp could not find the component, which lead me to manually register the grid components during the StartTest callback.

+ Performing a single mouse simulation is easy. I like the ability to select by element, or coordinate, and the ability to specify a delta.

- Performing complex mouse simulations is difficult. My test case for resizing the grid component, involved just 6 mouse simulations. I could either have a long nested list of callbacks, or try to group up the separate functions in to objects. Either way is cumbersome, because it forces me to manage a callback. The test case went something like this,
// Move to 100, 100
t.drag([0,0], [100, 100], null, function () {
  t.is(element.position.x, 100, 'Is x at 100?');
  t.is(element.position.y, 100, 'is y at 100?');

  // Move to 50, 50
  t.drag([100,100], [50, 50], null, function () {
     t.is(element.position.x, 50, 'Is x at 50?');
     t.is(element.position.y, 50, 'is y at 50?');

     // Move to 0, 0
     t.drag([50,50], [0, 0], null, function () {
        t.is(element.position.x, 0, 'Is x at 0?');
        t.is(element.position.y, 0, 'is y at 0?');
      });
  });
});
If possible, I would like to just be able to define drag tests, and test the result in the callback. I feel this would be a cleaner approach to simulating events.
var simulating = t.beginSimulateMouse(); // Possibly make this call to let the test know you are simulating until the clear is called
// Move to 100, 100
t.drag([0,0], [100, 100], null, function () {
  t.is(element.position.x, 100, 'Is x at 100?');
  t.is(element.position.y, 100, 'is y at 100?');
});
// Move to 50, 50
t.drag([100,100], [50, 50], null, function () {
   t.is(element.position.x, 50, 'Is x at 50?');
   t.is(element.position.y, 50, 'is y at 50?');
});

// Move to 0, 0
t.drag([50,50], [0, 0], null, function () {
   t.is(element.position.x, 0, 'Is x at 0?');
   t.is(element.position.y, 0, 'is y at 0?');
 });

t.endSimulateMouse(simulating);

Post by nickolay »

Mrfr0g wrote:First, allow me to say that this is an excellent tool. I have already created over 100 tests for our framework using this tool. I really like the simulation classes, watching the mouse move and expand my grid component is really neat.
Thanks, very glad to hear! Do you feel yourself more confident in your codebase now? :)
Mrfr0g wrote: - Using the Ext Loader class, I am sometimes able to run the tests once, but am not able to run the tests again without reloading the page. This is true even with caching disabled. From what I can tell the Ext.onReady event is only firing once when the classes are loaded dynamically. I am unsure how to resolve this issue, so to test, I am forced to use the packaged file.
Is it IE specific problem? And do you use the inline code in your preload config? like:
preload : [ { text : 'some javascript' } ]
? It may lead to some issues in IE. Basically ExtJS does not fire the `Ext.onReady()` if it was loaded *after* onload event of the document.
Try to add the following to the end of your preload:
preload : [ 
    ...,
    { text : 'Ext.isReady = true' } 
]
Might be also a fixed bug, we'll update the beta package tomorrow.
Mrfr0g wrote: - When testing the grid component with mouse simulation ( to resize the grid ), I experienced errors when the mouse moved over the grid headers. It seems that the grid header components call Ext.getCmp when the mouse is moved over the header to determine which header should apply the 'highlight' effect. Ext.getCmp could not find the component, which lead me to manually register the grid components during the StartTest callback.
Might be a fixed issue as well, if not - please create a small test case to demo the problem.
Mrfr0g wrote: + Performing a single mouse simulation is easy. I like the ability to select by element, or coordinate, and the ability to specify a delta.

- Performing complex mouse simulations is difficult. My test case for resizing the grid component, involved just 6 mouse simulations. I could either have a long nested list of callbacks, or try to group up the separate functions in to objects. Either way is cumbersome, because it forces me to manage a callback. The test case went something like this,
We are currently thinking about how to improve the asynchronous testing in general. Your suggestion looks good - I'd change it a little though:
    var dragging = t.beginDragging(); // Possibly make this call to let the test know you are simulating until the clear is called
    // Move to 100, 100
    dragging.drag([0,0], [100, 100], null, function () {
      t.is(element.position.x, 100, 'Is x at 100?');
      t.is(element.position.y, 100, 'is y at 100?');
    });
    // Move to 50, 50
    dragging.drag([100,100], [50, 50], null, function () {
       t.is(element.position.x, 50, 'Is x at 50?');
       t.is(element.position.y, 50, 'is y at 50?');
    });

    // Move to 0, 0
    dragging.drag([50,50], [0, 0], null, function () {
       t.is(element.position.x, 0, 'Is x at 0?');
       t.is(element.position.y, 0, 'is y at 0?');
    });

    dragging.done();

Post by Mrfr0g »

support_team wrote: Thanks, very glad to hear! Do you feel yourself more confident in your codebase now? :)
I feel a lot more confident in my codebase. One of the reasons we desire a testing framework is to automate our tests using Selenium. A large portion of our site is already manually tested in Selenium, and there is a large drive to make it automated with every commit. Siesta provides the tool we lacked to test our Ext components.
support_team wrote: Is it IE specific problem? And do you use the inline code in your preload config? like:
preload : [ { text : 'some javascript' } ]
? It may lead to some issues in IE. Basically ExtJS does not fire the `Ext.onReady()` if it was loaded *after* onload event of the document.
Actually this has been occurring on Chrome. I'll try your suggestion by setting the Ext.isReady bit manually.

I like your suggestion with dragging. I'm looking forward to the next release.

Post by mats »

mrfrog: can you post your test which does the resizing of the grid? How are you resizing it, is it the panel itself or the columns in it?

Post by Mrfr0g »

Here is the resize test. It will move the mouse to the resize handles, and drag them so the panel resizes.

I've inserted some additional comments which explain some of my choices in the test. These comments will begin with //#
StartTest(function (oTest) {
//# While running the tests I've found that there is a slight difference in time between when the test runs, and when the grid actually renders on the page. 
	var iDelay = 3000,
		aAsync = oTest.beginAsync();
	
	// Here, I wrap the entire test within prototypes bind/defer methods. This allows me to maintain the scope while delaying the execution of the test by the predefined delay.
	(function () {
		// Is the grid sized appropriately?
		var aSize = GridPanelInstance_GridPanel_1000.Grid.getSize();
		oTest.is(aSize.width, 500, 'Is the grid 500px wide?');		
		oTest.is(aSize.height, 400, 'Is the grid 400px tall?');
		
		// Do the resize handles exist?
		var aResizeHandles = $$('div.x-resizable-handle');		//# I use Prototype's DOM selection to detect the handle elements and store them
		oTest.is(aResizeHandles.length, 3, 'Did we detect all the resize handles ?');
		
		// Due to Siesta scoping bug, register any mouseover components with the component manager
		//# This bug still occurs in Siesta-0.21
		var aComponents = [
			// Resize Handles //
			GridPanelInstance_GridPanel_1000.Grid.resizer.east,
			GridPanelInstance_GridPanel_1000.Grid.resizer.south,
			GridPanelInstance_GridPanel_1000.Grid.resizer.southeast
		];
		
		aComponents.each(function (oComponent) {
			Ext.ComponentManager.register(oComponent);
		});
		
		// Grid Headers
		GridPanelInstance_GridPanel_1000.Grid.headerCt.items.items.each(function (oComponent) {
			Ext.ComponentManager.register(oComponent);
		});
		
		//# My suite of horizontal tests are defined within this object. They will essentially chain call the functions
		var oHorizontalTests = {
			onComplete_Resize550 : function () {
				// Did the grid resize?
				aSize = GridPanelInstance_GridPanel_1000.Grid.getSize();
				oTest.is(aSize.width, 550, 'Is the grid 550px wide?');
				
				// Can we resize back?
				oTest.drag(
					aResizeHandles[2],
					null,
					[-50, 0],
					oHorizontalTests.onComplete_Resize500
				);

			},
			onComplete_Resize500 : function () {
				// Did the grid resize?
				aSize = GridPanelInstance_GridPanel_1000.Grid.getSize();
				oTest.is(aSize.width, 500, 'Is the grid 500px wide?');
				
				// Can we resize to less than the set width?
				oTest.drag(
					aResizeHandles[2],
					null,
					[-50, 0],
					oHorizontalTests.onComplete_Resize450
				);
			},
			onComplete_Resize450 : function () {
				// Did the grid resize?
				aSize = GridPanelInstance_GridPanel_1000.Grid.getSize();
				oTest.is(aSize.width, 450, 'Is the grid 450px wide?');
				
				// Start Vertical resize tests
				oTest.drag(
					aResizeHandles[0],
					null,
					[0, 50],
					oVerticalTests.onComplete_Resize450
				);
			}
		};
		
                //# Here is the suite of vertical tests, they perform exactly like the horizontal tests
		var oVerticalTests = {
			onComplete_Resize450 : function () {
				// Did the grid resize?
				aSize = GridPanelInstance_GridPanel_1000.Grid.getSize();
				oTest.is(aSize.height, 450, 'Is the grid 450px in height?');

				// Can we resize back?
				oTest.drag(
					aResizeHandles[0],
					null,
					[0, -50],
					oVerticalTests.onComplete_Resize400
				);
			},
			onComplete_Resize400 : function () {
				// Did the grid resize?
				aSize = GridPanelInstance_GridPanel_1000.Grid.getSize();
				oTest.is(aSize.height, 400, 'Is the grid 400px in height?');

				// Can we resize to less than the set height?
				oTest.drag(
					aResizeHandles[0],
					null,
					[0, -50],
					oVerticalTests.onComplete_Resize350
				);
			},
			onComplete_Resize350 : function () {
				// Did the grid resize?
				aSize = GridPanelInstance_GridPanel_1000.Grid.getSize();
				oTest.is(aSize.height, 350, 'Is the grid 350px in height?');

				oTest.endAsync(aAsync);
				oTest.done();
			}
		}
		
		// Start by dragging to 550
                //# This starts the drag test suite
		oTest.drag(
			aResizeHandles[2],
			null,
			[50, 0],
			oHorizontalTests.onComplete_Resize550
		);
			
	}).bind(this).defer(iDelay);
});

Post by mats »

You testing Ext 3 or 4? Here's something I would use, and seems to work fine. Don't use defer's or arbitrary timeout - tests will be fragile. Instead use a waitFor statement.
StartTest(function (oTest) {
    var Grid = new Ext.grid.Panel({
        width: 500,
        height: 400,
        resizable: true,
        renderTo: Ext.getBody(),
        resizeHandles: 'w e s',
        columns: [{ text: 'foo', flex : 1}],
        store: new Ext.data.Store({
            model: Ext.define('moo', { extend: "Ext.data.Model" })
        })
    });

    function getHandle(dir) {
        return Grid.el.down('.x-resizable-handle-' + dir);
    }

    // Is the grid sized appropriately?
    oTest.isDeeply(Grid.getSize(), { width : 500, height : 400 }, 'Is the grid 500x400px?');

    oTest.is(Ext.select('div.x-resizable-handle').getCount(), 3, '3 resize handles found');

    var dragSteps = [
        { handle : 'east', deltaX : 50, deltaY : 0, expectedSize : { width : 550, height : 400 } },
        { handle : 'east', deltaX : -50, deltaY : 0, expectedSize : { width : 500, height : 400 } },
        { handle : 'east', deltaX : -50, deltaY : 0, expectedSize : { width : 450, height : 400 } },
        { handle : 'south', deltaX : 0, deltaY : 50, expectedSize : { width : 450, height : 450 } },
        { handle : 'south', deltaX : 0, deltaY : -50, expectedSize : { width : 450, height : 400 } },
        { handle : 'south', deltaX : 0, deltaY : -50, expectedSize : { width : 450, height : 350 } }
    ];

    function getDragFn() {
        
        var cfg = dragSteps.shift();
        if (cfg) {
            return function() {
                oTest.drag(getHandle(cfg.handle), null, [cfg.deltaX, cfg.deltaY], function() {
                    oTest.isDeeply(Grid.getSize(), cfg.expectedSize, 'Grid has correct width and height.');
                    getDragFn(cfg)();
                });
            };
        } else {
            return function() {
                oTest.done();
            }
        }
    }

    oTest.waitForComponentVisible(Grid, getDragFn());
});


Post by Mrfr0g »

We are testing Ext 4.05a. I had considered using waitForComponentVisible, but I am unsure on how to implement it with our testing scheme. Each one of our tests uses the hostPageUrl property to point to a PHP which generates the javascript configuration for our Ext components as well as our framework classes. When the javascript code is ran, our framework class is first initialized, then the framework initializes the Ext component. A typical example would look like;
Delegate Class {
initialize : function (config) {
 // Perform some data manipulation
 // Create a grid
 this.Grid = Ext.create('Ext.grid.Panel', {config});
}
}
When I have issues with the test is when the delegate has loaded, but the Grid (in this case) hasn't been created. Without having the Grid object, I would be unable to use waitForComponentVisible. Is there another way of creating a test for our environment?

Post by mats »

Can't you also include the init code in your test? Maybe we could jump on Skype to try to figure out how to make Siesta work well for you? My skype is mats.bryntse.

Post by Mrfr0g »

Including the init code in my tests would be difficult. The tests are written only in javascript, I guess I could create a PHP js/test object, but I feel that wouldn't be a clean test. I don't have a microphone or webcam installed on my computer at work, so I would have to wait until this afternoon when I got home to attempt a Skype call.

After reviewing the documentation, I may have a solution. This appears to work well for me, but let me know if it's not the correct way to use Siesta.
StartTest(function (oTest) {

oTest.waitFor(function () {
   return DelegateInstance && DelegateInstance.Grid;
}, RunTest);

function RunTest() {
   oTest.ok(DelegateInstance, "Does the delegate instance exist?");
   oTest.ok(DelegateInstance.Grid, "Does the grid component exist?");
}

});

Post Reply