Archiv der Kategorie: cloud

cloud ist das projekt, welches daten und streams aufnehmen und verteilen soll.
cloud is the project to take and deliver data and streams.

summer solstice 2017

the „fete de la musique“ took place in 2017, in berlin and other cities. regularily, i had moved to schillingbrücke, near yaam, where a public audience stage was prepared, like any other year, before, but from the „ritter butzke“ crew….

Stream videos with Ustream

a minute of ustream clip, that ive collected during that event with a 360° roundview, that maybe has its gaps, here on the playback. data was streamed via vodafone lte, but within a overloaded cell, so the transfering bandwidth was maybe not accessible. below a clip, that i had made on the weekend before, in prenzlauer berg  ( new swabia ) which runs slightly smoother. i was using a freemium account, so no guarantee, that it will last forever.

Stream videos with Ustream

on ustream, one can broadcast from the mobile device as well as from stationary devices. with the network in the backhand, some users could be served with that, with no effort, but with dashboarded reports.

nodemcu and arduino

having such small device, with wifi and many analog and digital i/o ports by hand, is somehow nice. that general purpose input output connectors are known as GPIO….. the loveley mr. wang from hongkong, tower 4, building c had send me that esp8266 v3 (340) board within the second week, directly from china, by airmail.

nodemcu/esp8266
device ports with addresses

that device ( nowerdays called „iot device“ ) can be used as serial bridge, for incomming / outgoing signals. a little more sophisticated is the idea, to send all the data by http with tcp/ip or as udp paket. a lot of traffic can be estimated, beside failure correction , missing paket and other crapy errors, that are common.

pushing a json string that way, triggered, when the value changes, could be more gentle to the chip. the unknown state of the receiver is the pitfall there. using websockets on the nodemcu will work with some message client, like mqtt, is the next climbing. the client subscribes to a master ( tomcat / rabbit mq or pure servlet with extras )  and  delivers the information straight there. the server will handshake the  request with a response, directly to a open port, on the client ip.

should be all, will see, when the code is working as expected. 20 sensors with 50 samples/second could mean 1000 requests per second – commonly. if nodemcu/raspberrypi3 is able to feature all that, is the question for the comming days. the arduino is something one have to learn, how to use  that efficiently is the first step. the next is, to have some java ide, like eclipse ready with java, then tomcat and a dbms or storage engine, like mariadb installed within a virtual machine, with the same debian, that is used for raspberrypi, that is called raspian. then the code delivery or the binaries should work on both enviroments, even if the architecture will differ between amd and arm.

a related story was some years ago, the teensy to get into the mac

ustream broadcasting

while we spoke, to get a hardcore radio back in business, the infrastructure as a service is there. at ustream, one can do exactly whats needed, taking care for content, audience and growth. the prices starts at 99$ for 100 slots/user. 5 user is free, more then the entry plan is possible, for not named numbers.

the embeddable player, with activated social  and archive function, can be seen and used, like below. sharing that link is then a click away.

budget.node.-.examples

while we went warm with theoretical package dense of up to 8x nodes in 1he server case, the reality is comming by and tell us, what is really up to do. the given hardware is:

  • 1x cubieboard1 | 1 hdd 500 gb
  • 1x cubieboard3 | 1 hdd 2 tb
cubieboard 1 + 3
cubieboard
2x + 1x

the price tag is 90 + 50 = 140 € for the boards alone, without any harddrive nor router nor accessoires.  the money is allready spent to the development god of tiny devices with giant duties. it could handle 100 mbit/s to the outside world, or with some usb-ethernetadapter possible the 1000 mbit/s from the bigger one, but on some point, the conversion/transversion from gigabit ethernet to 100 mbit/s must be done. the best way would be a board with dual-network-connector, but its not available yet.

 

 


  • 1x cubieboard3
  • 1x 2 core
  • 1x 1 gb ram
  • 1x 100 mbit/s / 1000 mbit/s nic
  • 1x 500 gb / 2 tb hdd
  • replication factor 1
  • price: 1 board + 1 hdd = 90 € + 90 € = 180 €
node config 2X
cubieboard 3
node config 2X

thats the current state. as the network extension cable is not working well, as we tried to wire the two leds for connection and traffic to the outside world. so the 100 mbit/s / 1000 mbit/s version is ready to gamble, but just with a replication factor of 1. thats poor, that less then raid 1 to compare it a bit. the harddrive should never fail or the backup for 500 gb is able to sync. within 24 hours on dsl-speed, at home. two fans should be ok for that amount of power consumption.

 

 

 

 


  • 1x cubieboard1 + 1x cubieboard3
  • 1x 1 core + 1x 2 core = 3 core
  • 1x 2gb ram + 1x 1 gb ram = 3 gb ram
  • 1x 100 mbit/s / 1000 mbit/s + 1x 100 mbit/s nic = 200 mbit/s
  • 1x 2 tb hdd + 1x 500 gb hdd = 500 + 1500 gb hdd
  • replication factor 2
  • price: 2 board + 2 hdd = 90 €  + 50 € + 2x 90 € = 320 €
node config 2X + 1X
cubieboard 1 + 3
node config 2X + 1X

here is the configuration with the available boards that i have for now.  line speed is like the same, as above, the logical bandwith would be 200 mbit/s if duplex is working without a loss. nothing needs to be shoped, all visible parts are on stack. screwing that into the case is another step, but then the system should run, even if on harddrive fails. a internal router is needed, 3 ports would be fine for that.  another way is, to connect the smaller load-balancer bord with a usb-ethernet-converter, but thats mystery from here on, as not every library is compiled on armhf. the power conversion needs to be doubled 4 ampere should be available for both boards.

 

 


  • 1x cubieboard1 + 2x cubieboard3
  • 1x 1 core + 2x 2 core = 5 core
  • 1x 1gb ram + 2x 2 gb ram = 5 gb ram
  • 1x 100 mbit/s + 2x 100 mbit/s / 1000 mbit/s nic
    = 300 mbit/s + 1800 mbit/s
  • 3x 2 tb hdd
  • replication factor 3
  • price: 3 board + 2 hdd = 2x 90 € + 50 € + 3x 90 € = 500 €
node config 2 x 2X + 1X
cubieboard 1 + 3
node config 2 x 2X + 1X

here is a version with a configuration of smooth compaction. logical bandwith would be 300 mbit/s or 2 x 1000 mbit/s + 100 mbit/s if the adapter is willing to switch it that way. another chance is a router, with a uplink, which handle that multiplex shifting. replication would be the minimum, as suggested. a internal router is needed, 4 or 5 ports would be fine for that. the power conversion needs more attention, estimated 6 ampere should be available therefore. three fans will blow the heat of the discs away, beside the little temperature fom the tiny boards.

 

 


its hard to find a conclusion on that. one the one hand 320 € are allready spent, with harddrives and that two boards.  the actual configuration costs around 180 €, but upgrading to the goal of replication factor 3, is another 180 € away. were in the middle of that price tag.

okay, the cheapest entry class server on dells online shop starts around 600 € for 2 core, 4 gb ram. for 745 € we could get 4 core and 8 gb ram, just to have more blast. to get replication 3, we need to spend another 400 € for additional sas-drives, makes it to 999 €, with the 2c / 4gb configuration. having more cpu and three drives, the price would be 1145 €. thats twice of our diy cloud-node and the example case to show up with.

we could have replication factor 3, but a quarter of cpu power available, compared to a pentium-class or xeon-based server with replication factor 1. having replication factor 3 there, with the bottleneck of a single board.


the independent boards could be configured via software as needed. thats called software-defined-network ( SDN ) nowadays. the single server board will be rougher as a developer board, but that would just be one node, with a unstriped disc-array. the cubie´s would present 3 nodes, for half the price, with a estimated four to six times slower arm v7 cpu. using the gpu with software will take another 1 or 2 year, from now on – if happen anywhere.

so, bringing it online the described upgrade way, would save the money to the point, when replication of 3 is urgently needed or when another chasis/case is ready to be deployed. then, the 2 + 3 drives would make the boom, the system is going to present for that issues.


serving web content is one case for that system. handling large files as no-sql engine, filebased with some glue, is the main goal within the logical network layer. synchronizing a database-cluster on one master and two slaves, would lower the bandwith for the need of replication. that will just scale between 2-5 or so. the distributed system would level that above 3 boards, in one server-chasis.

the network behaviour is not counted in this mind construct, a layer-3-switch would be very gentle to the paket-flow, but is outside the budget nor tested at all. switching the 5 – 8 available ports with 10 gbps would be fair enough, then the machine would have 1.5 gbyte/s inner bandwith, makes it half a iscsi device, targeted with single connect on 4 gbps.

cubie.in.da.houze

im dezember wurde die olle dell-möhre ausgeschlachtet, nun hatte ich ein paar wochen/enden zeit, um die boards auf herz und nieren zu testen.

cubieboard mit einem kern, einem gigabyte ram und 100mbit/s würden als loadbalancer oder datanode vermutlich ausreichen, um ein paar sachen laufen lassen zu könne. die notwendigen dienste jedoch, brauchen aber in der mindest-konfiguration schon mehr als diese eine gigabyte, daher hab ich jetzt den cubietruck in das 19 zoll servergehäuse geflanscht.

cubietruck nearly installed
cubietruck inside the 19 inch standard server chasis, with 1 he

die pestflatte hat nun einen sata-anschluss, statt ultra-scsi und ist am steckverbinder mit den gehäuse verklebt. heißkleber hat sich in letzter zeit als durchaus brauchbar dargestellt, die mechanische belastung sollte aber dennoch nicht zu hoch sein. der taster zum einschalten und zwei leds zum netzteil bieten ein wenig einfluss und information von vorn.

da die spannung mit 12v aus dem netzteil kommt, das board aber 5v haben will, gibt es einen regler, der diese spannung mit  etwa 85% wirkungsgrad verkleinert. von lm 317 hab ich noch jede menge hier rumfliegen, aber in zeiten von längsreglern und schaltnetzteilen, nicht mehr ganz zeitgemäß. wie dem auch sei, der eine regler dient auch der versorgung der hdd, 2 A sollten da wohl in summe ausreichen. da im gehäuse eigentlich 4 kaskadierte lüfter ( also 8 insgesamt ) drehen und im netzteil auch solche lärmdrüsen eingebaut sind, hab ich 2 von denen ans ende versetzt, damit festplatte und board eine gewisse zwangslüftung erfahren. man weiß ja nie, wer unter einem so kocht ( welcher server direkt darunter hitze produziert ).

cubietruck on 19 inch
cubie from the rear side of the case

die zuleitung für die festplatte reicht knapp aus, dafür muss die netzwerkdose noch aus dem gehäuse geführt werden. die meisten rj-45 döschen haben eine anzeige, wenn elektrischer kontakt besteht und wenn traffic auf der leitung ist. diesen luxus bietet das cubieboard 3, doch die übliche cat5 oder cat6 leitung hat nur 4 doppel-päärchen, also 8 leitungen. für 2 leds brauchts aber auch nochmal 4 drähte, also muss eine zweite leitung her. als ersatzteilspender bietet sich das alte dual-sockel-mainboard an, dort sind zwei dosen aufgelötet. eine fürs netz und eine für recovery, wie dem auch sei, mit einem 160w mörder-lötkolben ist sie schnell getrennt vom board. eine heißluftpistole wäre auch schick gewesen, die hat aber vor kurzem ihren geist aufgegeben.

jedenfalls macht das netzwerk noch zicken, daher mal ich mal die nächste version schonmal auf. 2 cubietrucks + 2 x 4 tb hdd und 1 cubieboard + 1 x 4 tbb + 4/5 x switch gb …. to be continued….

teensy-3.1.-.256k.rom.-.64k.ram.-.arm.v4

what to to, when a dude smashes a harddrive from a mac? the new drive dosnt have a clue about the ingredients of the machine, the firmware/bios is like a sissy to not let the owner accessing the system. several ways to get into the mac osx, but over the years, the rules went harder and one should explain the story to the support at some point.

pcb
top view

 

apple puts a lot of energy into their politics, to seal the borders of that system and to save the mac from other influences, except theirselfs. icloud is nowadays a default package, the most services, like imessage, is backed up in the cloud. no way to use such devices without such id for that storage and control of puzzlepiece. the mac is locked until the code of consumption is entered or penalty is payed to apples money-machine via customer care. fucking with assemblercode, to tweak the bios with u/efi and all that shit, is not even a plan. the firmware to restart the system is hidden within some hashes from the mainboard and the password itself, reverse engineering was not my plan for today.

windows programmer
ready or not, depends on the configuration of buttons and lights. this picture indicates a working chip.

if the mac is hanging on the four number code, to get back to live, one could call the customer care to get a scbo file or try typing the nearly 9999 combinations by hand. others are lazy to do so, a small processor, based on arm with 64kb ram, can do the same, much more gently. the tiny pcb ( printed circuit board ) with a size of 35 x 17mm holds a usb-connector, the cpu with 256kb rom and a lot of connectors to participate with others.

the firmware tells its a v.1.03, flashed with the customized arduino ide   ( integrated desktop enviroment { development tool } ) within seconds on the freescale MK20DX256, but with some difficulties. arduino is not just a misswritten word, they could be seen as major pusher behind that uprising developer-board-scene. that teensy is a clon in some way, but in fact, arm or atmel had a product line over the past years, so one can find several configurations, sources and dealerprices, to get into the rush of diy, for a fistful flies. another eclipse plugin should be named, but i had so-so results on the download site, all files were incomplete,  i guess the traffic limit is taken, on the end of the month. the project should move to the next level then.

chip is ready to reboot
update cycle finished

as i hadnt much experience with avr/atmel chips at all, i decided, to put everything in a virtual box, on windows 7 for some reasons. but suprise suprise, the winavr-toolkit wants to lay inside the default folder ( ${winDrive/winavr } ) instead of being sorted with others in the standard folder for program files (x86). anyhow, the masterpiece of microsoft is, that the usb port and parameter is buffered/saved within the registry and due to other lacks of their understanding of software design, the programer wont flash the hex.code from the arduino software into the chip. excellent, couple of hours later i gave up on this, as i wanted to use eclipse for larger developments.

so, burning the working code to the chip was only possible, with a direct connect to the host-system.  some years ago, i played with a atmega ( think was …32 ) for a laser-tester, that one had started with basic-code. the arduino enviroment has its own syntax and file-extension.ino and with gnu, c/c++ software the architecture could be cascaded.

happy hour for the chip
upgrade sucessfullly, the new code is flashed to the soc

so, the chip is now ready to stupidly check each number, from 0000->9999 until the mac agrees to work again, when the correct passcode is clearly analyzed, by try& error. oh yeah, the soc could do much more then this, but the case was just there, to check the stuff with a goal to reach. some dozen projects and use-cases for that little machine is visible on the web: dj-controller, measurement boxes with lcd displays or game-engines are out there and many other ideas are just eyeblinks away.

bigger or larger projects will clearly need some more attention and notes to be realized, but with a sd-card-slot in dollar-distance, the bugs for a unique or genuine solution is definitly no high steak or far away. things can be made and grown with developer kits, analog or digital interfaces, up to the samplerate of a oscilloscope or analyzer by mean.

windows programmer
ready or not, depends on the usb connection and windows is a sissy in that area, after 15 years of existance, a unhappy programmer with no chance to pump new bytes into the chip.

crunching on arm

as far as it looks like, iam not the only one, crunching on arm. dzone got a article from thailand, where some guys connected 22 cubieboards, runinning apache spark, for in-memory-computation.

but ive seen the bottleneck of a 100 mbits network connector on cubieboard pretty much earlier. so ive decided to buy the cubietruck, which includes a 1 gbits adapter, which brings theoretically 128 mbyte/s a sata drive for saturation.

a cluster from france was presented before on cubieboard.org with the same idea, to fit as many cubieboards within a standard rack server enclosure, as possible.

i calculated 8 cubieboards in a 1 he server rack as maximum amount of boards and hard discs last year. the price is then something like 1600-2000 € for the hardware. for that price tag, the memory of 16 gb ram and 16 tb of hdd is not that much, but the theroretical bandwith between the nodes ( within the cluster ) is something like 8 gbits 🙂

8 nodes
8 boards in a 1 he standard rack server

the picture above shows the former choosed boards from hardkernel, called odroid, but the price for that board is above the budget, so i sized it down to a dual-core, from the quad-core.

cloudAnbieter

da festplatten zwar immer günstiger und größer werden, die dateien und formate aber auch, bleibt das ewige spiel mit dembackup. wie möchte man eine 2-3-4 terabyte platte sichern? wie kann man 10 oder 20TB sinnvoll wegschaufeln? was tun, wenn das haus abgebrannt oder abgesoffen ist – was ist dann mit den eigenen daten. klar, man kann sie brennen oder auf usb kopieren, aber ab einer bestimmten größe macht das alles keinen spass mehr. nun hab ich heut beim g+ ein paar interessante beispiel für verfügbare clouds aufgeschnappt:

tarsnap fällt durch schlichte leistung und geringe kosten auf. 0.30us$/gb-monat für den space und 0.30us$/gb beim traffic scheinen günstig zu sein. der wesentliche punkt aber ist, dass nur veränderte daten hochgeladen werden, ein inkrementelles backup sinngemäß. aes-256 wird angeboten und kann, da linux-basierter-client, durch eigene maßnahmen ergänzt werden. als infrastruktur wird amazon ec2 verwendet, also nichts aufregendes.

crashplan sieht wesentlich ausgereifter und professioneller aus. die verschlüsselung beträgt 448-bit für die dateien und 128bit-aes für die verbindung. 4us$ für einen rechner, total-backup, klingt ok – eine konkrete größe vermisse ich aber dann doch. für firmenkunden werden auch pakete mit 200 oder 1000 und mehr angeboten. die infrasturktur ist nicht auf den ersten blick ausgeschildert.

der vollständigkeithalber wären noch die üblichen  verdächtigen zu nennen:

google-drive
apple-icloud
µ$-onedrive
strato-hidrive

klar kann man sich auch ein nas hinstellen, doch es geht ja um ein offsite-backup, also eine sicherungskopie ausserhalb der eigenen vier wände. sicher, im bunker, tief im rechenzentrum, dupliziert, verteilt, gespiegelt – einfach bombensicher *hust*. was die einzelnen betreiber jedoch im kleingedruckten für regeln und leistungen haben, muss man dann doch genauer evaluieren.