DSC API is developed by the Computer Systems Department at Jozef Stefan Institute , Ljubljana, Slovenia.

This API can be used for statistical comparison of stochastic optimization algorithms for multiple-problem analysis.

Please attach your data in csv format. Use this template to organize your data.

Choose the appropriate statistical test that you want to use.

Please choose the statistical significance.


Forgot password?

Sign up

Download the ranked matrix


Registration form

Please fill out the form in order to use the DSC appraoch.

Once you created your username and password, you are able to use the DSC approach.

The evaluation of the DSC approach is performed by using the results from the Black-Box Benchmarking 2015, which is a competition that provides single-objective functions for benchmarking. The statistical comparisons are performed using algorithms that are selected from the 15 algorithms used in the BBOX 2015. The experimental results show that DSC gives more consistent results compared to the common approach in the cases when the results form the common approach are affected by outliers or the ranking scheme used by some standard statistical tests.

This work was supported by the project ISO-FOOD, which received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 621329 (2014-2019).

DSC API is developed by the Computer Systems Department at Jozef Stefan Institute , Ljubljana, Slovenia.

We highly appreciate all feedback and want to know your thoughts and experience using the tool.

Please contact us if you need assitance.

Main Contact

Assoc. Prof. Peter Korošec, Ph. D.

Computer Systems Department, Jožef Stefan Institute

E-mail: peter.korosec@ijs.si


Team

Tome Eftimov, Ph. D. Student , Statistician, Developer, Handball enthusiast

Computer Systems Department, Jožef Stefan Institute

E-mail: tome.eftimov@ijs.si


Assoc. Prof. Peter Korošec, Ph. D. , Project lead, Team lead, Lead developer, Superhero, Mravljus

Computer Systems Department, Jožef Stefan Institute

E-mail:peter.korosec@ijs.si


Assoc. Prof. Barbara Koroušić Seljak, Ph. D. , Team lead

Computer Systems Department, Jožef Stefan Institute

E-mail: barbara.korousic@ijs.si

Interface

DSC API introduces two web services through an HTTP REST API. The both web services accept input parameters in JSON and return results in JSON.

The base URL for all API endpoints is https://ws.ijs.si:8443/DSC-1.0 and the only allowed method is POST.

All requests expect a content type header of Content-Type: application/json and accept header of Accept: application/json .

API Clients

You can use the DSC API as commnad-line tool or programmaticaly from R through the api-R client. Check out the source code. It's open source, and contirbutions are welcome!

If you develop an open source DSC API client in any other programming language and want to make it freely available, please tell us about it and we will publish it here.

To use the DSC appraoch through the web interface or command line, first you need to register by filling out the registration form. Once you provide your e-mail and password you can use them to sign in to the system and to use the DSC approach.

Input parameters

Apart from the header and the authentication parameters, all other parameters should be provided in a JSON encoded POST payload.

For the first web service you need to specify the algorithms name, and to provde data obtained for each problem (function) by using each of the algorithms. Also, you need to specify the significance level that will be used by the two-sample Kolmogorov-Smirnov test.

An example of the input parameter for the first web service:

            {
            	"alpha": 0.5,
            	"data":[
            		{
            			"name":"algorithmName1",
            			"problems":[
            				{
            					"name":"problemName1",
            					"data":[11.0,2.0,13.0,4.0,15.0]
            				},
            				{
            					"name":"problemName2",
            					"data":[1.0,7.0,8.0,9.0,4.0]
            				}
            			]
            		},
            		{
            			"name":"algorithmName2",
            			"problems":[
            				{
            					"name":"problemName1",
            					"data":[1.0,2.0,3.0,4.0,5.0]
            				},
            				{
            					"name":"problemName2",
            					"data":[6.0,7.0,7.0,9.0,11.0]
            				}
            			]
            		}
            	]
            }
          

For the second web service the result from the first web service combined with the name of the appropriate statistical test and the significnace level is used as input parameter.

An example of the input parameter for the second web service:

            {
            	"ranked_matrix":[
            		{
            			"problemName":"problemName1",
            			"result":[
            				{
            					"algorithmName":"algorithmName1",
            					"rank":1.5
            				},
            				{
            					"algorithmName":"algorithmName2",
            					"rank":1.5
            				}
            			]
            		},
            		{
            			"problemName":"problemName2",
            			"result":[
            				{
            					"algorithmName":"algorithmName1",
            					"rank":1.5
            				},
            				{
            					"algorithmName":"algorithmName2",
            					"rank":1.5
            				}
            			]
            		}
            	],
            	"number_algorithms":2,
            	"parametric_tests":0,
            	"method":{
            		"name":"methodName",
            		"alpha":0.05
            	}
            }
          

The ranked_matrix, number_algorithms, and parametric_test are the outputs obtained by the first web service. Using them, you need only to specify the name of the method that will be used for analyze, together with the statistical significance level.

The Statistical API exposes two methods. The first one is for ranking your data and provides the data needed for statistical comparison. The second one uses the data returned by the first mathod and performs statistical comparisons using it.

An example of the result from the first web service:

            {
            	"ranked_matrix":[
            		{
            			"problemName":"problemName1",
            			"result":[
            				{
            					"algorithmName":"algorithmName1",
            					"rank":1.5
            				},
            				{
            					"algorithmName":"algorithmName2",
            					"rank":1.5
            				}
            			]
            		},
            		{
            			"problemName":"problemName2",
            			"result":[
            				{
            					"algorithmName":"algorithmName1",
            					"rank":1.5
            				},
            				{
            					"algorithmName":"algorithmName2",
            					"rank":1.5
            				}
            			]
            		}
            	],
            	"number_algorithms":2,
            	"parametric_tests":0,
            	"method":{
            		"name":"methodName",
            		"alpha":0.05
            	}
            }
          

An example of the result from the second web service:

            {
            	"message":"The null hypothesis is not rejected.",
            	"p_value":1.0,
            	"t":0.0,
            	"method":{
            		"name":"Iman Davenport's correction of Friedman's rank sum test",
            		"alpha":0.05
            		}
            }
          

curl is an open source command line tool and library for transferring data with URL syntax.

A curl example for the first web service:

curl -i --user <user:password> -H "Accept: application/json" -X POST https://ws.ijs.si:8443/DSC-1.0/rank -H "Content-Type: application/json" -d '{"alpha":0.05,"data":[{"name":"a1","problems":[{"name":"f1","data":[5.32,4.32]},{"name":"f2","data":[1.23,2.23]}]},{"name":"a2","problems":[{"name":"f1","data":[1.22,5.1]},{"name":"f2","data":[2.12,1.0]}]},{"name":"a3","problems":[{"name":"f1","data":[1.23,4.65]},{"name":"f2","data":[1.22,5.1]}]}]}'

A curl example for the second web service:

curl -i --user <user:password> -H "Accept: application/json" -X POST https://ws.ijs.si:8443/DSC-1.0/evaluate -H "Content-Type: application/json" -d '{"ranked_matrix":[{"problemName":"f1","result":[{"algorithmName":"a1","rank":1.5},{"algorithmName":"a2","rank":1.5}]},{"problemName":"f2","result":[{"algorithmName":"a1","rank":1.5},{"algorithmName":"a2","rank":1.5}]}],"number_algorithms":2,"parametric_tests":0,"method":{"name":"friedman","alpha":0.05}}'

If you need help using the DSC API or are having issues with DSC approach, please contact us.