Reimplementing the DES Carmen code in python
  • Python 34.5%
  • C 25.4%
  • Perl 17.2%
  • R 11.5%
  • Shell 7.4%
  • Other 4%
Find a file
Giuliano Antoniol d384431b7a simple rephrase
2022-04-04 18:55:22 -04:00
carmen-files added merged fronts 2022-04-04 15:16:53 -04:00
docs updated 2021-04-16 15:24:37 -04:00
obsolete cleanup 2021-06-21 11:51:44 -04:00
policier-replication simple rephrase 2022-04-04 18:55:22 -04:00
random added merged fronts 2022-04-04 15:16:53 -04:00
test-data update 2021-01-17 14:16:02 -05:00
testing-code cleanup 2021-06-21 12:03:40 -04:00
tools baseline 2021-04-18 10:08:28 -04:00
utility updated 2021-04-19 12:14:56 -04:00
.gitignore baseline 2021-07-07 08:46:34 -04:00
config.py reformatting 2020-12-21 11:40:52 -05:00
cpu.py cleanup 2021-06-21 11:44:33 -04:00
daemon-cleanup.sh baseline 2021-07-01 15:21:07 -04:00
daemon-spanner.pl baseline 2021-07-01 15:21:07 -04:00
daemon-stop.sh baseline 2021-07-01 15:21:07 -04:00
doBaseline.sh baseline 2021-08-30 13:44:23 -04:00
error.py test 2021-01-12 08:43:39 -05:00
estimationNoise.py added sorting algorithm for bin packing 2021-03-29 11:10:23 -04:00
files4greedy-evaluation.pl to run greedy evaluation 2021-07-22 11:18:43 -04:00
files4pessimistic-evaluation.pl uddated to simpliofy passing error 2021-08-08 10:11:28 -04:00
gitlab-ci.yml cleanup for now 2021-01-12 08:56:22 -05:00
greedy-pareto.sh updated for greedy pareto evolving fronts 2022-01-20 18:17:02 -05:00
greedy-test.sh updated 2021-07-22 11:32:57 -04:00
greedy_pareto.py update 2022-01-21 17:24:18 -05:00
greedyEvolution.py to run greedy evaluation 2021-07-22 11:18:43 -04:00
hv_eval.py added merged fronts 2022-04-04 15:16:53 -04:00
memo.txt baseline 2021-07-01 15:23:56 -04:00
MoeaOptimize.py added merged fronts 2022-04-04 15:16:53 -04:00
MyDuplicates.py updated for greedy pareto evolving fronts 2022-01-20 18:17:02 -05:00
MyProblem.py ported to pymoo 0.5 and added MOEA/D 2022-01-09 09:15:20 -05:00
MyServerCrossover.py ported to pymoo 0.5 and added MOEA/D 2022-01-09 09:15:20 -05:00
MyServerMutation.py ported to pymoo 0.5 and added MOEA/D 2022-01-09 09:15:20 -05:00
MyServerSampling.py updated for greedy pareto evolving fronts 2022-01-20 18:17:02 -05:00
OnlyOneObject.py updated time unit cost and solution print 2021-03-01 11:48:48 -05:00
Optimize.py added merged fronts 2022-04-04 15:16:53 -04:00
pareto-plot.r added merged fronts 2022-04-04 15:16:53 -04:00
pessimistic-tests.sh uddated to simpliofy passing error 2021-08-08 10:13:42 -04:00
pessimisticEstimate.py updated with a kludge 2021-08-08 10:07:16 -04:00
plot-compare-rq3.r simple comparison graphs 2021-08-11 09:41:52 -04:00
plot-compare.r simple comparison graphs 2021-08-11 09:41:52 -04:00
random-baseline.sh to get a baseline 2021-08-30 13:43:52 -04:00
random_baseline.py fixed loop 2021-08-30 13:59:51 -04:00
randomBaseline.sh to compute all figire for random init 2021-08-30 14:21:23 -04:00
README.TXT updated with creen and shell running 2021-03-03 09:16:01 -05:00
requirements.txt updated requirements 2020-10-07 06:15:47 -04:00
rq3_5.csv uddated to simpliofy passing error 2021-08-08 10:11:28 -04:00
run1experiment.sh cleanup 2021-06-21 11:57:38 -04:00
run1GreedyParetoRQ2experiment.sh baseline 2022-01-21 13:55:43 -05:00
run1MoeaRQ2experiment.sh baseline 2022-01-11 17:47:18 -05:00
run1RQ2experiment.sh baseline 2021-07-01 15:29:23 -04:00
screen-status.sh baseline 2021-07-01 15:21:30 -04:00
ScreenDesDaemon.pl baseline 2021-07-01 15:29:23 -04:00
ScreenRQ2DesDaemon.pl update 2022-01-17 14:15:43 -05:00
ScreenRQ2GreedyDaemon.pl to run a greedy search 2022-01-21 13:51:00 -05:00
ScreenRQ3DesDaemon.pl tp simplify and make fatser RQ3 2021-08-09 08:27:43 -04:00
server.py cleanup 2021-06-21 11:44:33 -04:00
server_farm.py updated fitness to cache hits 2022-01-21 10:23:44 -05:00
server_pool.py focded update for greedy pareto 2022-01-20 18:24:14 -05:00
simple-baseline.sh to get a baseline 2021-08-30 13:43:52 -04:00
simple-test.sh added merged fronts 2022-04-04 15:16:53 -04:00
simulation_engine.py cleanup 2021-06-21 11:44:33 -04:00
skill.sh baseline to cleanup screen sockets 2021-07-05 07:11:42 -04:00
solution.py updated fitness to cache hits 2022-01-21 10:23:44 -05:00
span.sh baseline 2021-07-01 15:32:38 -04:00
spanRQ2.sh baseline 2021-07-01 15:32:38 -04:00
spanRQ3.pl simplify rq3 run 2021-08-08 10:20:42 -04:00
tc.py cleanup 2021-06-21 11:50:57 -04:00
test_cpu.py test 2021-02-14 17:16:03 -05:00
test_set.py tp simplify and make fatser RQ3 2021-08-09 08:27:43 -04:00
test_sim_engine.py to get a baseline 2021-08-30 13:43:52 -04:00
test_solution.py forced update 2021-02-13 18:48:36 -05:00
tmoea.py added merged fronts 2022-04-04 15:16:53 -04:00
tmoea.sh update 2022-01-17 14:15:43 -05:00
vcpu.py cleanup 2021-06-21 11:48:23 -04:00

This code is to simulate a set of servers running tasks. The idea is to partition tasks between servers and
find close to optimal compromise cost/time. In order to do so the plan is to use pymoo NSGAII. The project is a follow up of Carmen Coviello project and Java implementation.

Notice you need python 3.5 or higher, you need flake8 and yapf. flake8 is a syntax/style checker while yapf is a source code beautifier.  As environment consider PyCharm community edition.

There are a few directory. The code source in the current directory plus:

docs: store documents  here design ideas etc
carmen-files: Carmen test sets and amazon data
test-data: store simple test cases to manually validate here
testing-code: baseline use of various structures to make sure at least they run
tools: wrappers for flake8 and yapf

Files and roles

config.py: we should store here various project specific config parameters, almost empty for now

error.py: error  and warning procedures to simplify error handling

cpu.py: the basic unit to model a processor it is the base class or vCPU and server

vCPU.py: a server may have multiple vCPUs matter of fact it must have at least one , perhaps this may not be needed
	   	  keeping for now

server.py: it is the server model, it has an id and a reference to the amazon server config file. This is to say the server id
	      is the line number where the server was found in the server pool configuration  file

server_pool.py: when we read servers configuration we create a server pool; the pool is of immutable object and is
		used to instantiate server farms. Each server farm is actually a configuration used to compute a solution.

server_farm.py: a list of servers; it is created via the pool instantiate_farm(nservers, max_per_type) method calls.
		       This takes 2 params: how many servers and how  many server per type. Loosely, for now we create
		       an array of nservers randomly choosing servers from the pool. For each server type we create at most
		       max_per_type instances. Notice that since the servers have (possibly) multiple cpus, there is no simple
		       relation server to cpu; we need to count them!
		       
solution.py:  it is made our model: a farm and a test set,. plus the evaluate function, the fitness calculation

tc.py: simple model of a test  case

test_set.py: this is an array  of test cases. Notice when we instantiate a solution we do a deep copy of a test
	     set and we can permutate them in this way we have variability but we do not screw up the original test set




EXECUTION
-------------

You can run the pythin code or use the wrapper


./runExsample.sh


see the parameter passes  or the example below running it into a detached screen:


screen -dmS DES  ./runExsample.sh  -t carmen-files/dataset/datasetFastJFreeChart.csv -s carmen-files/systems/serverRealiAmazon.csv