{"id":4020,"date":"2023-11-04T23:14:00","date_gmt":"2023-11-04T23:14:00","guid":{"rendered":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/"},"modified":"2023-11-05T05:48:23","modified_gmt":"2023-11-05T05:48:23","slug":"using-azure-batch-to-run-large-scale-parallel-workloads","status":"publish","type":"post","link":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/","title":{"rendered":"Using Azure Batch to run large scale parallel workloads"},"content":{"rendered":"<h1>Introduction<\/h1>\n<p>Managing large-scale parallel workloads can be challenging, especially when it comes to allocating resources efficiently and cost-effectively. Azure Batch offers a cloud-based solution for running parallel workloads at scale, and provides a scalable, distributed infrastructure that allows you to run your applications across multiple nodes.<\/p>\n<p>This tutorial will walk you through how to use Azure Batch to run large-scale parallel workloads. We\u2019ll cover creating a Batch account and pool, submitting a job, and monitoring progress, as well as best practices for optimizing performance and reducing costs.<\/p>\n<h1>Prerequisites<\/h1>\n<p>To follow this tutorial, you will need:<\/p>\n<ul>\n<li>An Azure account with an active subscription<\/li>\n<li>The Azure CLI installed on your local machine<\/li>\n<li>Basic knowledge of command line interfaces (CLI)<\/li>\n<\/ul>\n<h1>Creating a Batch Account and Pool<\/h1>\n<p>First, we need to create a Batch account and pool to manage our parallel workloads.<\/p>\n<h2>Creating a Batch Account<\/h2>\n<p>To create a Batch account, follow these steps:<\/p>\n<ol>\n<li>Open the Azure portal and log in to your Azure account.<\/li>\n<li>In the search bar, type \u201cBatch accounts\u201d and select it from the results.<\/li>\n<li>Click the \u201cAdd\u201d button.<\/li>\n<li>Fill out the required fields, such as the resource group, account name, and region.<\/li>\n<li>Configure other settings such as the virtual network, subnet, and storage account.<\/li>\n<li>Review and accept the terms of use, then click \u201cCreate\u201d.<\/li>\n<\/ol>\n<h2>Creating a Pool<\/h2>\n<p>Once you have created a Batch account, you need to create a pool to manage your compute resources. A pool consists of one or more virtual machines (VMs) that you can use to run your parallel workloads.<\/p>\n<p>To create a pool, follow these steps:<\/p>\n<ol>\n<li>Navigate to your Batch account in the Azure portal.<\/li>\n<li>In the left-hand menu, click \u201cPools\u201d.<\/li>\n<li>Click the \u201cAdd\u201d button.<\/li>\n<li>Fill out the required fields, such as the pool name, VM size, and number of nodes.<\/li>\n<li>Configure other settings such as the image type and authentication.<\/li>\n<li>Review and accept the terms of use, then click \u201cCreate\u201d.<\/li>\n<\/ol>\n<h1>Submitting a Job<\/h1>\n<p>Once you have created a Batch pool, you are ready to submit a job to run your parallel workload.<\/p>\n<h2>Creating a Job<\/h2>\n<p>To create a job, follow these steps:<\/p>\n<ol>\n<li>Navigate to your Batch account in the Azure portal.<\/li>\n<li>In the left-hand menu, click \u201cJobs\u201d.<\/li>\n<li>Click the \u201cAdd\u201d button.<\/li>\n<li>Fill out the required fields, such as the job ID and pool name.<\/li>\n<li>Configure other settings such as the task command line and resource files.<\/li>\n<li>Review and accept the terms of use, then click \u201cCreate\u201d.<\/li>\n<\/ol>\n<h2>Creating a Task<\/h2>\n<p>Once you have created a job, you need to create a task to perform the parallel workload. A task is a unit of work in Batch that can be run on one or more nodes in a pool.<\/p>\n<p>To create a task, follow these steps:<\/p>\n<ol>\n<li>Navigate to your Batch account in the Azure portal.<\/li>\n<li>In the left-hand menu, click \u201cJobs\u201d.<\/li>\n<li>Click on the job that you created.<\/li>\n<li>Click the \u201cAdd\u201d button.<\/li>\n<li>Fill out the required fields, such as the task ID and command line.<\/li>\n<li>Configure other settings such as resource files and environment variables.<\/li>\n<li>Review and accept the terms of use, then click \u201cCreate\u201d.<\/li>\n<\/ol>\n<h2>Monitoring Progress<\/h2>\n<p>You can monitor the progress of your job and task in the Azure portal, or through the Batch CLI.<\/p>\n<p>To monitor your job and task in the Azure portal, follow these steps:<\/p>\n<ol>\n<li>Navigate to your Batch account in the Azure portal.<\/li>\n<li>In the left-hand menu, click \u201cJobs\u201d.<\/li>\n<li>Click on the job that you created.<\/li>\n<li>Review the status of the job and task in the overview page.<\/li>\n<\/ol>\n<p>To monitor your job and task through the CLI, follow these steps:<\/p>\n<ol>\n<li>Open a command prompt or terminal window.<\/li>\n<li>Log in to your Azure account using the Azure CLI.<\/li>\n<li>Run the command <code>az batch job show<\/code> to view the status of your job.<\/li>\n<li>Run the command <code>az batch task list<\/code> to view the status of your task.<\/li>\n<\/ol>\n<h1>Best Practices<\/h1>\n<p>To optimize performance and reduce costs when using Azure Batch for large-scale parallel workloads, consider the following best practices:<\/p>\n<h2>Containerize your applications<\/h2>\n<p>By containerizing your applications, you can make them more portable and easier to manage. Containers can be stored in container registries and deployed to different environments, and can even be used with Kubernetes for container orchestration.<\/p>\n<h2>Use low-priority VMs<\/h2>\n<p>Low-priority VMs are up to 80% cheaper than regular VMs, but are reclaimed by Azure if demand for resources increases. By using low-priority VMs, you can reduce costs while still getting the compute power that you need.<\/p>\n<h2>Use autoscaling<\/h2>\n<p>Batch provides autoscaling capabilities that allow you to automatically adjust the number of nodes in your pool based on demand. By using autoscaling, you can ensure that you always have enough compute resources to run your workload, while minimizing costs during periods of low demand.<\/p>\n<h2>Optimize network throughput<\/h2>\n<p>Azure Batch provides several options for optimizing network throughput, such as using RDMA-enabled VMs, high-speed interconnects, and optimized network settings. By optimizing network throughput, you can reduce the time it takes to transfer data between nodes, which can significantly improve performance.<\/p>\n<h1>Conclusion<\/h1>\n<p>In this tutorial, we covered how to use Azure Batch to run large-scale parallel workloads. We walked through creating a Batch account and pool, submitting a job and task, and monitoring progress, as well as best practices for optimizing performance and reducing costs.<\/p>\n<p>By following these best practices, you can efficiently and cost-effectively manage your parallel workloads using Azure Batch, and take advantage of the scalability and distributed infrastructure provided by the platform.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction Managing large-scale parallel workloads can be challenging, especially when it comes to allocating resources efficiently and cost-effectively. Azure Batch offers a cloud-based solution for running parallel workloads at scale, and provides a scalable, distributed infrastructure that allows you to run your applications across multiple nodes. This tutorial will walk <a href=\"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/\" class=\"btn btn-link continue-link\">Continue Reading<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[1],"tags":[411,868,30,96,866,867,865,869,864],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Using Azure Batch to run large scale parallel workloads - Pantherax Blogs<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Using Azure Batch to run large scale parallel workloads\" \/>\n<meta property=\"og:description\" content=\"Introduction Managing large-scale parallel workloads can be challenging, especially when it comes to allocating resources efficiently and cost-effectively. Azure Batch offers a cloud-based solution for running parallel workloads at scale, and provides a scalable, distributed infrastructure that allows you to run your applications across multiple nodes. This tutorial will walk Continue Reading\" \/>\n<meta property=\"og:url\" content=\"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/\" \/>\n<meta property=\"og:site_name\" content=\"Pantherax Blogs\" \/>\n<meta property=\"article:published_time\" content=\"2023-11-04T23:14:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-11-05T05:48:23+00:00\" \/>\n<meta name=\"author\" content=\"Panther\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Panther\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\n\t    \"@context\": \"https:\/\/schema.org\",\n\t    \"@graph\": [\n\t        {\n\t            \"@type\": \"Article\",\n\t            \"@id\": \"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/#article\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/\"\n\t            },\n\t            \"author\": {\n\t                \"name\": \"Panther\",\n\t                \"@id\": \"http:\/\/localhost:10003\/#\/schema\/person\/b63d816f4964b163e53cbbcffaa0f3d7\"\n\t            },\n\t            \"headline\": \"Using Azure Batch to run large scale parallel workloads\",\n\t            \"datePublished\": \"2023-11-04T23:14:00+00:00\",\n\t            \"dateModified\": \"2023-11-05T05:48:23+00:00\",\n\t            \"mainEntityOfPage\": {\n\t                \"@id\": \"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/\"\n\t            },\n\t            \"wordCount\": 900,\n\t            \"publisher\": {\n\t                \"@id\": \"http:\/\/localhost:10003\/#organization\"\n\t            },\n\t            \"keywords\": [\n\t                \"\\\"Azure cloud services\\\"\",\n\t                \"\\\"batch processing\\\"\",\n\t                \"\\\"cloud computing\\\"\",\n\t                \"\\\"distributed computing\\\"\",\n\t                \"\\\"high performance computing\\\"\",\n\t                \"\\\"job scheduling\\\"\",\n\t                \"\\\"parallel workloads\\\"\",\n\t                \"\\\"scalable computing\\\"]\",\n\t                \"[\\\"Using Azure Batch\\\"\"\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"WebPage\",\n\t            \"@id\": \"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/\",\n\t            \"url\": \"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/\",\n\t            \"name\": \"Using Azure Batch to run large scale parallel workloads - Pantherax Blogs\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"http:\/\/localhost:10003\/#website\"\n\t            },\n\t            \"datePublished\": \"2023-11-04T23:14:00+00:00\",\n\t            \"dateModified\": \"2023-11-05T05:48:23+00:00\",\n\t            \"breadcrumb\": {\n\t                \"@id\": \"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/#breadcrumb\"\n\t            },\n\t            \"inLanguage\": \"en-US\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"ReadAction\",\n\t                    \"target\": [\n\t                        \"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/\"\n\t                    ]\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"BreadcrumbList\",\n\t            \"@id\": \"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/#breadcrumb\",\n\t            \"itemListElement\": [\n\t                {\n\t                    \"@type\": \"ListItem\",\n\t                    \"position\": 1,\n\t                    \"name\": \"Home\",\n\t                    \"item\": \"http:\/\/localhost:10003\/\"\n\t                },\n\t                {\n\t                    \"@type\": \"ListItem\",\n\t                    \"position\": 2,\n\t                    \"name\": \"Using Azure Batch to run large scale parallel workloads\"\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"WebSite\",\n\t            \"@id\": \"http:\/\/localhost:10003\/#website\",\n\t            \"url\": \"http:\/\/localhost:10003\/\",\n\t            \"name\": \"Pantherax Blogs\",\n\t            \"description\": \"\",\n\t            \"publisher\": {\n\t                \"@id\": \"http:\/\/localhost:10003\/#organization\"\n\t            },\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"SearchAction\",\n\t                    \"target\": {\n\t                        \"@type\": \"EntryPoint\",\n\t                        \"urlTemplate\": \"http:\/\/localhost:10003\/?s={search_term_string}\"\n\t                    },\n\t                    \"query-input\": \"required name=search_term_string\"\n\t                }\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"Organization\",\n\t            \"@id\": \"http:\/\/localhost:10003\/#organization\",\n\t            \"name\": \"Pantherax Blogs\",\n\t            \"url\": \"http:\/\/localhost:10003\/\",\n\t            \"logo\": {\n\t                \"@type\": \"ImageObject\",\n\t                \"inLanguage\": \"en-US\",\n\t                \"@id\": \"http:\/\/localhost:10003\/#\/schema\/logo\/image\/\",\n\t                \"url\": \"http:\/\/localhost:10003\/wp-content\/uploads\/2023\/11\/cropped-9e7721cb-2d62-4f72-ab7f-7d1d8db89226.jpeg\",\n\t                \"contentUrl\": \"http:\/\/localhost:10003\/wp-content\/uploads\/2023\/11\/cropped-9e7721cb-2d62-4f72-ab7f-7d1d8db89226.jpeg\",\n\t                \"width\": 1024,\n\t                \"height\": 1024,\n\t                \"caption\": \"Pantherax Blogs\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"http:\/\/localhost:10003\/#\/schema\/logo\/image\/\"\n\t            }\n\t        },\n\t        {\n\t            \"@type\": \"Person\",\n\t            \"@id\": \"http:\/\/localhost:10003\/#\/schema\/person\/b63d816f4964b163e53cbbcffaa0f3d7\",\n\t            \"name\": \"Panther\",\n\t            \"image\": {\n\t                \"@type\": \"ImageObject\",\n\t                \"inLanguage\": \"en-US\",\n\t                \"@id\": \"http:\/\/localhost:10003\/#\/schema\/person\/image\/\",\n\t                \"url\": \"http:\/\/2.gravatar.com\/avatar\/b8c0eda5a49f8f31ec32d0a0f9d6f838?s=96&d=mm&r=g\",\n\t                \"contentUrl\": \"http:\/\/2.gravatar.com\/avatar\/b8c0eda5a49f8f31ec32d0a0f9d6f838?s=96&d=mm&r=g\",\n\t                \"caption\": \"Panther\"\n\t            },\n\t            \"sameAs\": [\n\t                \"http:\/\/localhost:10003\"\n\t            ],\n\t            \"url\": \"http:\/\/localhost:10003\/author\/pepethefrog\/\"\n\t        }\n\t    ]\n\t}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Using Azure Batch to run large scale parallel workloads - Pantherax Blogs","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/","og_locale":"en_US","og_type":"article","og_title":"Using Azure Batch to run large scale parallel workloads","og_description":"Introduction Managing large-scale parallel workloads can be challenging, especially when it comes to allocating resources efficiently and cost-effectively. Azure Batch offers a cloud-based solution for running parallel workloads at scale, and provides a scalable, distributed infrastructure that allows you to run your applications across multiple nodes. This tutorial will walk Continue Reading","og_url":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/","og_site_name":"Pantherax Blogs","article_published_time":"2023-11-04T23:14:00+00:00","article_modified_time":"2023-11-05T05:48:23+00:00","author":"Panther","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Panther","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/#article","isPartOf":{"@id":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/"},"author":{"name":"Panther","@id":"http:\/\/localhost:10003\/#\/schema\/person\/b63d816f4964b163e53cbbcffaa0f3d7"},"headline":"Using Azure Batch to run large scale parallel workloads","datePublished":"2023-11-04T23:14:00+00:00","dateModified":"2023-11-05T05:48:23+00:00","mainEntityOfPage":{"@id":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/"},"wordCount":900,"publisher":{"@id":"http:\/\/localhost:10003\/#organization"},"keywords":["\"Azure cloud services\"","\"batch processing\"","\"cloud computing\"","\"distributed computing\"","\"high performance computing\"","\"job scheduling\"","\"parallel workloads\"","\"scalable computing\"]","[\"Using Azure Batch\""],"inLanguage":"en-US"},{"@type":"WebPage","@id":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/","url":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/","name":"Using Azure Batch to run large scale parallel workloads - Pantherax Blogs","isPartOf":{"@id":"http:\/\/localhost:10003\/#website"},"datePublished":"2023-11-04T23:14:00+00:00","dateModified":"2023-11-05T05:48:23+00:00","breadcrumb":{"@id":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/localhost:10003\/using-azure-batch-to-run-large-scale-parallel-workloads\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/localhost:10003\/"},{"@type":"ListItem","position":2,"name":"Using Azure Batch to run large scale parallel workloads"}]},{"@type":"WebSite","@id":"http:\/\/localhost:10003\/#website","url":"http:\/\/localhost:10003\/","name":"Pantherax Blogs","description":"","publisher":{"@id":"http:\/\/localhost:10003\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/localhost:10003\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"http:\/\/localhost:10003\/#organization","name":"Pantherax Blogs","url":"http:\/\/localhost:10003\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/localhost:10003\/#\/schema\/logo\/image\/","url":"http:\/\/localhost:10003\/wp-content\/uploads\/2023\/11\/cropped-9e7721cb-2d62-4f72-ab7f-7d1d8db89226.jpeg","contentUrl":"http:\/\/localhost:10003\/wp-content\/uploads\/2023\/11\/cropped-9e7721cb-2d62-4f72-ab7f-7d1d8db89226.jpeg","width":1024,"height":1024,"caption":"Pantherax Blogs"},"image":{"@id":"http:\/\/localhost:10003\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"http:\/\/localhost:10003\/#\/schema\/person\/b63d816f4964b163e53cbbcffaa0f3d7","name":"Panther","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/localhost:10003\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b8c0eda5a49f8f31ec32d0a0f9d6f838?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b8c0eda5a49f8f31ec32d0a0f9d6f838?s=96&d=mm&r=g","caption":"Panther"},"sameAs":["http:\/\/localhost:10003"],"url":"http:\/\/localhost:10003\/author\/pepethefrog\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"http:\/\/localhost:10003\/wp-json\/wp\/v2\/posts\/4020"}],"collection":[{"href":"http:\/\/localhost:10003\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/localhost:10003\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/localhost:10003\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/localhost:10003\/wp-json\/wp\/v2\/comments?post=4020"}],"version-history":[{"count":1,"href":"http:\/\/localhost:10003\/wp-json\/wp\/v2\/posts\/4020\/revisions"}],"predecessor-version":[{"id":4516,"href":"http:\/\/localhost:10003\/wp-json\/wp\/v2\/posts\/4020\/revisions\/4516"}],"wp:attachment":[{"href":"http:\/\/localhost:10003\/wp-json\/wp\/v2\/media?parent=4020"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/localhost:10003\/wp-json\/wp\/v2\/categories?post=4020"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/localhost:10003\/wp-json\/wp\/v2\/tags?post=4020"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}