Rendering on cloud

    We all probably have heard about the clouds. Know that there are the most global and the heavy computing. The clouds are the most difficult sites, and even I had tried out the taste of the clouds and moved back to your site. Naturally I immediately felt the difference, the site began to work immediately, and I was able to customize it very subtly under your taste and needs, which cannot be done on the normal sharing hosting and the price per unit of resource has declined dozens of times. And now I came up with the idea, and that if the move to the cloud rendering. I’m not going to describe in detail what is the cloud, there’s a good article on Wikipedia. I only say that any cloud 99.999999% uptime guarantees and payment as resource use, i.e. do not use do not pay.

    And now what is the difference between cloud and conventional services. Let’s say, for if you want to place the site before it was necessary to draw to sharing hosting provider that gives you Daddy on your server where you upload your site, then your site’s limited ability to host. Or in the case of the render farm, in order to render the scene you have to collect and pass on this scene at someone else’s render farm, where you will find, once again, such services are limited to the functionality of the render farm.

    In the cloud, you are given is not a resource, and an entire operating system in a virtual environment like. Each such system is up to 100% of the resources of the machine on which it is performed, i.e., if the system is running on a computer with 2 XEON Processors and 44 GB of RAM then it may need to get all these resources. But if a real computer running your task falls, the task automatically is shifted to another working computer, the same applies to disk space, and the guarantee of the health% 99.999999.

    As for the description of the whole process needed for dozens or even hundreds of pages, and I don’t want you to bind to a specific cloud, all the information is aggregated and very dense form. And you are already on the basis of this article can understand how the cloud works and what it is for computer graphics. For a more detailed understanding of all the settings you have to read the instructions on your cloud. Also in the article are not exactly household already has operations, as what the program is and how is network rendering, and implies a certain experience of the reader.

    How to create a cloud

    There are a large number of cloud providers, description and distinction which I present below in the next section. For example, I decided to choose the most tricked out of them with the maximum convenience and functionality of Amazon EC2. And then based on the listed and each opportunity will afford a cloud hoster.

    Stage one. Familiarity with cloud and the creation of a single machine.

    The first step is to register. Registration is no big deal, and the registration of any other Internet service, except that the password will tell you the robot calling American numbers on your phone, in English, of course. During the registration process you will also need to bind your card Visa or MasterCard, or any other supported systems to your account, which as you use the clouds will be written off. During registration you cancel 2 dollar, this is a test, a week off will return.

    After successful registration, you will need to log in to your account and find there click AWS Management Console, click on it and get into the cloud.

    Don’t be put off by the word console, this is actually very simple and informative management system, write anything manually here. Each tab is a separate service. What does each we consider will not, for this is the official guide to the cloud that you choose. And to work we need total less, not more than three.

    Find bookmark Amazon EC2 and click it. Go to the service creation and management of virtual machines.

    Before you can create virtual machines, you must select the region in which they reside. Depending on the region of both the cost and the available configuration of the machines. Me for personal needs more suited United States Virginia, why does explain a bit later.

    Now we can start to create the first virtual machine. Click on the buttons to create an instance. And follow not tricky, but note that some plug-ins, services are paid separately, read the following instructions carefully before connecting. First time settings may eyes run, but over time you get used to and understand that it’s Wizard calculator. When all settings are done, you can press the button to Start Instance.

    Started creating an instance. It takes up to 15 minutes, but usually much less. After creating an instance we need to access it. To do this, we need to know the password of it. To do this click on the instance of the RMB and select Get Windows Password, feed the master password pair you got on the stage, create an instance, and you answer everything you need to connect remotely to a machine address, machine name, user name, and password.

    As you can see there’s nothing complicated, just touched. The whole procedure resembles work with conventional virtual machines only in the browser.

    Some of the important elements in the settings: 1. Security Group is aws firewall for your machines, you enable or disable the ports. 2. Elatic IPs-here you can get your external IP for your machines, one for free, others for money. 3. Unfortunately Amazon (like most of the clouds), you cannot create common instansov repository, so this problem you might decide you any available means, the easiest way is to use the machine to store the configuration in a 10Gbps network. 4. the Block Store-Elastik flexible storage, you can create additional storage and connect it to your machines as standard storage device (hard drive).

    Stage two. Install the required software.

    Now when we have the virtual machine we can supply the necessary us program to your taste and color. In General, you must create three mashines. The first will be the render Manager and license server on a second program to render and will perform the role of instansa reproduction of the original render of nod, the third will keep work files.

    To facilitate the work with the remote machine as soon as you put the required programs, such as for remote file transfer, excellent teamviewer. Then we can put all the necessary for rendering program. I have for example it will be 2013 3DS MAX + VRay 2.30, but you can put any you need.

    Next put the render Manager, again any one you like more, I for example Deadline. On the machine with the software to render the set Deadline Client.

    Note: I originally tried to put the Backburner, but the automatic reproduction stage render nod he refused to work correctly. If the render node create individually created machines, it works fine.

    Next, we will need a new machine which will stand the repository deadline. According to the same principle as in the first paragraph, we create a new machine, share the folder and put back the Deadline Reposytories.

    Do final configuration, the Deadline set by the Client to the network path Reposytories Deadline. Try to start the server and monitor.

    It is time to create a total for the whole of the render farm store. For this, another instance we create and share the folder. Further to this folder will dump scene and applied to the files. Carefully choose the type instanced, lax will pull up to 5-10 render nodes, stronger up to 10-20, clustered up to 100-300, for more need to independently build a distributed file system.

    Only debug startup process render nodes after starting the computer. Check whether the automatic deadline client, whether the computer can connect to the repository without asking for a password and other settings. When you have, you can make a test run of the render.

    If done correctly, you can begin to multiply render nodes.

    Stage three. Reproduction of render nodes.

    And now the most interesting. As the same from one node to make tens, hundreds, thousands of render nodes. But again very simple. For a start it from one finished, fully customized render nodes to create a sample. To do this we click RMB on the instanced render node and click Create image (EBS AMI). To begin creating an image, it takes about 15 minutes.

    Now we have the image of a render node from which you can run many clones. You can continue to operate on two variants:

    Variant 1-Manual.

    Go to bookmark AMIs find our image render nodes, click RMB and press start the instance. And fall into the already familiar instansa Wizard, this time only on the second tab, because the image already selected.

    In paragraph Number of Instance, specify the required number of render us nod, and paragraph Instance Type specify the equipment on which our render nodes will run, necessarily all render farms, otherwise they will not see each other. Click the Continue several times and look like began creating specified number of render nodes.

    When we look at the render Manager and see how it gradually begin to appear the render node. Here’s what all of our render farm is ready, you can start render jobs.

    And then, as necessary, you can adjust the number of machines in render farm, raise additional machine in case of need, from the image of AMI, or destroy those extra machines if they are temporarily not needed. The most important thing to keep track of the number, because the money is removed for any included machinery, whether they are or not loading.

    Note: this method of cloning machines they have the same network name, which is the Deadline, they are defined as a render node. That does not prevent him from rendering to render nodes, but makes it difficult to manage render nodes through the render Manager. If you want to divide the machines that would each appear as a render node, simply set render node unique name. That would automate this process can start render nodes place a .bat file that automatically when you first run the rename. In the Backburner, there is the same problem, but much more difficult, each car with the same name periodically selects a connection manager, which leads to the collapse of the render. Renaming machine problem does not solve, but if raising each individual node render problem did not manifest.

    Variant 2 – automatically, as needed.

    ДFor this purpose there are special services CloudWatch, used to monitor clouds and create alerts and Elastik MapReduce, is used to automatically launch certain kinds of work, but because they have no clue what is rendering and render Manager, for us they are difficult to apply.

    A much more practical solution would be to use cloud development tools. Tools are quite simple and all kept to simple commands, turning these tools to your render Manager, you can automatically adjust the number of machines. For example if you render more than 10 Manager tasks, runs 10 render nodes, if 20 is 20, and vice versa, if there are any idle machines, they are automatically deleted.

    Conclusion: that’s just going to create and manage their own cloud. As you can see, this is no big deal. And work much easier and nicer than with other organizations, whether sharing hosting or render farm or any other task. The price of such a cloud at times below commercial render farms, much more computing resources, and a set of features that you define your needs. At the end of the use of the cloud do not forget to turn off any unnecessary machine that would not pay extra for any idle machines.

    At the end of the billing period you withdraw money in the amount of resources that you used and send an extract which you can familiarize with the information on used resources.

    Adult games or try to up to 2 Tesla alive

    One of the great advantages of the clouds, it is the presence of a large selection of hardware on which to run cloud. Here we can try different generation of server processors and, in practice, see the difference between them. And, in the case of the Amazon, we can try and GPU Computing. In general conversation about the GPU can be put in a separate article, but I still go only to cite the results of the tests. In the pictures below we can see the results of calculations of identical pictures on 3 types of iron and the result of GPU Computing.

       
    Without GI

    Core i7 920 – 40 min

    2 x Intel Xeon X5570 – 18 min

    2 x Intel Xeon E5-2670 – 12 min

    2 * Tesla M2050 – 9 min
    With GI, LC + IR

    2 x Intel Xeon X5570 – 37 min

    2 x Intel Xeon E5-2670 – 23 min

    2 * Tesla M2050 – 18 min

    As you can see the different generation of server processors when an equal frequency of more than 50% of the difference give a performance, so be careful when selecting hardware and its cost, including cloud.

    As for GPU Computing, then everyone will make conclusions for himself. If arhvizer sees much more noisy picture counted during the same time, the CG Artist will see a picture with acceptable noise levels with beautiful global illumination counted almost the same time. However, the quality of textures in GPU Computing is sharply reduced, and the number of supported technologies VRay is limited, which greatly limits the GPU render the animation.

    But with regard to iRay, tests with him could not be held, he simply could not find a Tesla, let’s leave it to developers.

    Experiment with Selectel

    Selectel cloud works only on Linux machines. But as 3DS MAX for Linux is not yet written, operate directly, we can not, but it’s not necessary. Actually not all tasks require mandatory presence of 3DS MAX to render nodes, in most cases only one render, which is usually presented in Linux versions. I for example will show at the VRay Standalone. Details of working with Linux I down, raiding it does not understand, and implied a certain experience with the readers of this topic.

    Register for Selectel and making cloud-based machine. I take my favorite Linux distro OpenSuse.

    Start virtual machine creation, it takes 5 minutes. On completion of the establishment we can connect to it via SSH. Now we need to configure the render node, it is enough to just get there VrayStandalone, you can take out version of Vray for Maya for Linux, and during the installation, select Standalone Only. Description of the installation process and pulling the license server described in detail to help the installer. Also it needs a library in X 11 Xinerama, enough to put through the console YAST packages X 11, you can without a server. That’s basically all the render node is ready.

    Now, let’s run it to render. It is also very simple. From under the car Windows open a scene you want to render in 3DS MAX (Maya, Cinema4D, XSI, Blender or any other) and you press the button export format scene Vray scene and specify the location and name of the file where to save it (the Cyrillic alphabet does not create the work file).

    Get the stage that understands the Vray, if you use other renderers, they usually also have the opportunity to create their own scenes. VrayScene is an open file, you can open and edit in a text editor. Inside is all about the scene geometry, shading, lighting, camera, render settings, and render, generally the same as in the archives of the renderman RIB.

    Before you send the correct way to render textures and objects on the Linux path. To do so, open any powerful text editor can easily edit a file and open it on our stage and simple replacement of the changing patch with what you have on Windows machines to those that you have on Linux machines. Now save the file.

    You can now send the render node. To do this, you must copy the files VRay scenes and textures on our Linux machine, it is easiest to do this by FTP to the folder that we have treatment, patients sometimes when changing lanes. And a simple command to launch the file to render (and path to vray screenshot in the first line). I examine all available options in the help section.

    Here’s what all the rendering went. What to look for: 1. option -scene File = “scene”-specifies which scene to render, 2. option -imageFile = “where and how to save file-path and file format which will be saved and render layers, 3. a very important option -display = 0 -disables the display window rendering, when you try to render with VRay asks for a screen, which naturally is not, and will crash with an error.

    And now the fun part and why Selectel. So goes a very little consumption of RAM, the system consumes up to 360 MB of memory and 30 MB memory consumes VRay, which amount to 1 million. triangles is less than 400 MB, I remind that 43.6 GB left in stock. Turns out that the only consumable resource is processor, eventually rendering per hour came around 6 rubles, for everything. That is the lowest of the known to me the cost of rendering. For example, the cost of rendering on a home computer, with far more modest resources, approximately 0.07$ per hour, if you upgrade a computer once in 4 years. And render farms to commercial at least 8 times cheaper. The only drawback is there are limits to the number of machines.

    Choosing a cloud provider

    1. Giants Amazon EC2 cloud computing and Windows Azure, most powerful and most functional clouds.

    Amazon EC2-by far the most powerful cloud hoster. With the largest selection of services and configuration options. There are also such exotic variants like machines with installed graphics Tesla. But unfortunately has the least cloud prices on hourly work machines. If you take the right option for rendering graphics, 2 CPU Xeon with 23 GB of RAM, then a month of uninterrupted work have to give away $ 1.610 per Hour * 24hours * 30days = 1159,2$. Agree, the numbers are just astronomical. But luckily Amazon very flexible price policy and have the opportunity to make a one-time contribution for 1 or 3 years, after which the cost of one hour of work the machine is reduced at times. Amazon is currently the largest cloud hoster in the Park, of which there are almost 500 thousand vehicles, which is enough for any queries.

    Windows Azure is a cloud-specific, it is not an operating system, and a service in which you can arrange your application, typically ASP or Windows services that already and will run on a certain amount of clouds. Theoretically, it can accommodate and we render the application, but I have this attempt created some difficulty of the work required to be done for this, so leave this cloud the fate of Web applications and services. Fortunately for the price of a much more democratic Amazon EC2, the most powerful configuration costs just 720$ per month and that’s not counting discounts. Of the drawbacks of cloud oriented Web applications and the most powerful package has only 14 GB RAM on the machine.

    Addition: at the time of writing, the cloud Azure strongly expanded its functionality. Now it can be exactly like the other clouds run virtual machines and run on them absolutely any operation.

    2. Less famous cloud providers with more democratic prices, but less functionality.

    ActivCloudSideBarScalaxyISPServerCloud4Y and that only Russians too good cloud providers differ less service and more democratic prices for a similar configuration are asked several times less money. For a machine with 2 XEON and 23 GB of RAM already here asking to give away from 12 to 25 thousand rubles per month, or from 14 to 35 roubles per hour of work, and that’s not counting discounts. There are a large selection of operating systems and hardware configurations. But there are drawbacks to some hosting servers, you cannot make a copy of the existing machines, as a result, each machine will have to raise yourself from scratch. On the other are asked to pay once a month, test service on convenience and quality. Some clouds on each unit is allocated an individual disk space and the problem of sharing common resources have to decide. In addition, domestic data centers where more modest in size and raise 1000 or 100 machines at once, but usually such scale and does not require.

    3. Cloud service providers to payment upon load.

    For example on one of these clouds is my site. The beauty of these clouds is that initially the image selects the entire power of machines on which it is performed, i.e., in my case it Selectel, allocated a 2 XEON processor and  44 GB of RAM. But the payment is only used in, i.e., my site is spinning at 64 bit OpenSuse 12.1 consumes only 0.44% of the CPU and 371 MB of RAM, respectively and the payment I have going on that amount and is only $0.2 per day or 3$ a month. If I want to perform a very difficult operation I can utilize all the power of the machine and then payment will increase by several times. On a cloud Selectel also works great site VKontakte.

    Addition: while writing to become an opportunity to mount a single repository to all running instances, which greatly simplified the creation of a single store. Single images can be created from an image, in much the same way as in the Amazon.

    Clodo Calculator

    Selectel also the same principle works Clodo. Prices from these providers is comparable with prices of other hosters cloud, approximately $ 750 at full load for 2XEON and 24 GB of RAM. The lack of these providers that don’t have Windows machines, and you can only render on linux machines, and Linux almost no suitable for us programs. And dignity that you always have the most powerful machine, but do not pay for something that is not in use. If you have used it for a month only 10% of the time or 10% power, and will pay you 10% of the total $ 750.

    Economic justification

    1. Creating clouds against the purchase of equipment

    And now let’s count how many we do purchase your own render farm for 100 machines in the farm on 1 machine. Server with XEON 2 and 23 GB of RAM costs about $ 2,500. Given that the equipment completely changed every 4 years, the price of the server for 1 year is $ 625, ascribe to the same disk array purchase subtotal $ 1,250 a year. Besides our equipment consumes electricity and in need of high-speed network + $ 70., total $ 1320 a year. Also this equipment should someone serving, turn on the salary system. Admin + $ 200 total $1520 per year. In addition our equipment periodically breaks down or even what is more frightening is happening +20% total $1820 per year or $ 150 a month worth of one machine in the farm of 100 cars, if the car is less than the cost of servicing the machine 1 increases dramatically.

    Cloud in average costs much more 400-$ 750 per month. But if your farm fixed costs per month, then in the cloud you only pay for the amount of the advantage. Let’s say if you used 200 hours a month, and you only have to pay for these 200 hours. That is $ 750 per month * 200 hours we used/720 hours per month = $ 200. At the end of the calculations your cloud collapses and falls asleep while waiting for the new team, meanwhile, computing power has already another person.

    Turns out that the greatest benefits you need to find a balance between the volume of own farm and a cloud farm. Let’s say if you need every month to count a very cool video for 24 hours, as the calculations are much cheaper to use the cloud. And if you have a constant flow of files to render and periodically appears on the task that not enough of your capacities the there is already combined, continuous threads take on their own farm, and at peak loads to the clouds. To compensate for the downtime of both types of farms they can rent.

    Besides cloud data centers have a very high degree of reliability and, in the event of a failure in the equipment automatically fires back. Even in case of power failure the Centre picked up batteries and alternator, and even if there is a spare generator.

    2. Use the clouds against the use of commercial render farms

    There are a large number of commercial render farms. For example, compare with one of the best Rebusfarm. First consider the number, usually in these farms is very little cars, 30, 80, and only in as much as there are as many as 340 Rebusfarm machines. In the cloud, we can raise at least 1000 machines in one click. With regard to prices, Rebusfarm 3.9 cents per requests/hour, Let’s translate this into our cloud 2 XEON PROCESSORS for 8 cores each, 3.9 cents/hour * 2 Ghz CPU * 8 physical cores * 2.5 GHZ each = $ 1.56 per hour. Number is mildly astronomical, and is very expensive version of the very expensive cloud hoster Amazon. And that’s when you can stand in the queue, and the premium account is already twice as expensive. In the cloud, we’re in the queue will never stand. And as the show options considered there are clouds and $0.4 per hour, up to 4 times cheaper.

    In addition to these render farms is a strictly defined set of programs and plugins, if you are using third-party or proprietary plugins to render you fail. In the cloud, you decide which programs to set and how to configure them, and can not be afraid for their intellectual property that your brilliant own plugin anyone drag you, access to the cloud have only you.

    Who needs it

    1. Freelancers.

    Periodically there are very attractive offers to make what may be a very beautiful movie. You understand that from a creative standpoint, there are no problems and everything’s done for two clicks, but with the computing resources the trouble on your super cool computer video will be a couple of months and had to abandon the project. So now we can safely make the cloud and find everything in it. Theoretically, you can even connect to your favorite phone keyboard and mouse set up remote access to the cloud and to work with mobile phone.

    2. Company.

    Companies too often go luck or long-awaited success, and you need to render that technical or financial status does not render. In this case, again, you can hire the services of clouds.

    Welcome to the cloud – Happy END

    PS: during the time of this writing, none of the above, the license has not suffered =)

     If you liked the article, you can help expand it by clicking the like on your social buttons, and if it has brought you a great favor or benefit, and you want to thank the author, or inspire me to new more interesting articles you can make donations. All the collected money will go to improving the site and meet their own needs.

    8th August 2012 Articles, Home

    Leave a Reply